By pgb


2009-06-28 23:37:19 8 Comments

I want to extend a large C project with some new functionality, but I really want to write it in Python. Basically, I want to call Python code from C code. However, Python->C wrappers like SWIG allow for the OPPOSITE, that is writing C modules and calling C from Python.

I'm considering an approach involving IPC or RPC (I don't mind having multiple processes); that is, having my pure-Python component run in a separate process (on the same machine) and having my C project communicate with it by writing/reading from a socket (or unix pipe). my python component can read/write to socket to communicate. Is that a reasonable approach? Is there something better? Like some special RPC mechanism?

Thanks for the answer so far - however, i'd like to focus on IPC-based approaches since I want to have my Python program in a separate process as my C program. I don't want to embed a Python interpreter. Thanks!

8 comments

@JohnMudd 2015-11-23 15:51:23

I've used the "standard" approach of Embedding Python in Another Application. But it's complicated/tedious. Each new function in Python is painful to implement.

I saw an example of Calling PyPy from C. It uses CFFI to simplify the interface but it requires PyPy, not Python. Read and understand this example first, at least at a high level.

I modified the C/PyPy example to work with Python. Here's how to call Python from C using CFFI.

My example is more complicated because I implemented three functions in Python instead of one. I wanted to cover additional aspects of passing data back and forth.

The complicated part is now isolated to passing the address of api to Python. That only has to be implemented once. After that it's easy to add new functions in Python.

interface.h

// These are the three functions that I implemented in Python.
// Any additional function would be added here.
struct API {
    double (*add_numbers)(double x, double y);
    char* (*dump_buffer)(char *buffer, int buffer_size);
    int (*release_object)(char *obj);
};

test_cffi.c

//
// Calling Python from C.
// Based on Calling PyPy from C:
// http://doc.pypy.org/en/latest/embedding.html#more-complete-example
//

#include <stdio.h>
#include <assert.h>

#include "Python.h"

#include "interface.h"

struct API api;   /* global var */

int main(int argc, char *argv[])
{
    int rc;

    // Start Python interpreter and initialize "api" in interface.py using 
    // old style "Embedding Python in Another Application":
    // https://docs.python.org/2/extending/embedding.html#embedding-python-in-another-application
    PyObject *pName, *pModule, *py_results;
    PyObject *fill_api;
#define PYVERIFY(exp) if ((exp) == 0) { fprintf(stderr, "%s[%d]: ", __FILE__, __LINE__); PyErr_Print(); exit(1); }

    Py_SetProgramName(argv[0]);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString(
            "import sys;"
            "sys.path.insert(0, '.')" );

    PYVERIFY( pName = PyString_FromString("interface") )
    PYVERIFY( pModule = PyImport_Import(pName) )
    Py_DECREF(pName);
    PYVERIFY( fill_api = PyObject_GetAttrString(pModule, "fill_api") )

    // "k" = [unsigned long],
    // see https://docs.python.org/2/c-api/arg.html#c.Py_BuildValue
    PYVERIFY( py_results = PyObject_CallFunction(fill_api, "k", &api) )
    assert(py_results == Py_None);

    // Call Python function from C using cffi.
    printf("sum: %f\n", api.add_numbers(12.3, 45.6));

    // More complex example.
    char buffer[20];
    char * result = api.dump_buffer(buffer, sizeof buffer);
    assert(result != 0);
    printf("buffer: %s\n", result);

    // Let Python perform garbage collection on result now.
    rc = api.release_object(result);
    assert(rc == 0);

    // Close Python interpreter.
    Py_Finalize();

    return 0;
}

interface.py

import cffi
import sys
import traceback

ffi = cffi.FFI()
ffi.cdef(file('interface.h').read())

# Hold references to objects to prevent garbage collection.
noGCDict = {}

# Add two numbers.
# This function was copied from the PyPy example.
@ffi.callback("double (double, double)")
def add_numbers(x, y):
    return x + y

# Convert input buffer to repr(buffer).
@ffi.callback("char *(char*, int)")
def dump_buffer(buffer, buffer_len):
    try:
        # First attempt to access data in buffer.
        # Using the ffi/lib objects:
        # http://cffi.readthedocs.org/en/latest/using.html#using-the-ffi-lib-objects
        # One char at time, Looks inefficient.
        #data = ''.join([buffer[i] for i in xrange(buffer_len)])

        # Second attempt.
        # FFI Interface:
        # http://cffi.readthedocs.org/en/latest/using.html#ffi-interface
        # Works but doc says "str() gives inconsistent results".
        #data = str( ffi.buffer(buffer, buffer_len) )

        # Convert C buffer to Python str.
        # Doc says [:] is recommended instead of str().
        data = ffi.buffer(buffer, buffer_len)[:]

        # The goal is to return repr(data)
        # but it has to be converted to a C buffer.
        result = ffi.new('char []', repr(data))

        # Save reference to data so it's not freed until released by C program.
        noGCDict[ffi.addressof(result)] = result

        return result
    except:
        print >>sys.stderr, traceback.format_exc()
        return ffi.NULL

# Release object so that Python can reclaim the memory.
@ffi.callback("int (char*)")
def release_object(ptr):
    try:
        del noGCDict[ptr]
        return 0
    except:
        print >>sys.stderr, traceback.format_exc()
        return 1

def fill_api(ptr):
    global api
    api = ffi.cast("struct API*", ptr)

    api.add_numbers = add_numbers
    api.dump_buffer = dump_buffer
    api.release_object = release_object

Compile:

gcc -o test_cffi test_cffi.c -I/home/jmudd/pgsql-native/Python-2.7.10.install/include/python2.7 -L/home/jmudd/pgsql-native/Python-2.7.10.install/lib -lpython2.7

Execute:

$ test_cffi
sum: 57.900000
buffer: 'T\x9e\x04\x08\xa8\x93\xff\xbf]\x86\x04\x08\x00\x00\x00\x00\x00\x00\x00\x00'
$ 

@jhegedus 2015-09-12 06:40:06

This seems quite nice http://thrift.apache.org/, there is even a book about it.

Details:

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.

@Eli Courtwright 2009-06-28 23:42:27

I recommend the approaches detailed here. It starts by explaining how to execute strings of Python code, then from there details how to set up a Python environment to interact with your C program, call Python functions from your C code, manipulate Python objects from your C code, etc.

EDIT: If you really want to go the route of IPC, then you'll want to use the struct module or better yet, protlib. Most communication between a Python and C process revolves around passing structs back and forth, either over a socket or through shared memory.

I recommend creating a Command struct with fields and codes to represent commands and their arguments. I can't give much more specific advice without knowing more about what you want to accomplish, but in general I recommend the protlib library, since it's what I use to communicate between C and Python programs (disclaimer: I am the author of protlib).

@bdk 2009-06-29 02:18:06

I've done this before and it worked out well. I had several C processes that communicated by sending structs over a socket and wanted to allow python processes too. Writing the protocol stuff was trivial in python and I was able to write a Python script that ran as part of the build to autocreate the Python code to pack/unpack the C structs by parsing the .H files. Only problem is due to all the string packing/unpacking, performance isn't nearly what you would get with native C acting directly on the binary representation of the structs just memcopy and casting the raw data off the socket.

@Mandrake 2009-10-25 17:16:09

apparently Python need to be able to compile to win32 dll, it will solve the problem

In such a way that converting c# code to win32 dlls will make it usable by any development tool

@Pharaun 2010-06-15 20:53:37

This only applies to win32 platform :-p What about Unix?

@Alex Martelli 2009-06-29 02:44:04

If I had decided to go with IPC, I'd probably splurge with XML-RPC -- cross-platform, lets you easily put the Python server project on a different node later if you want, has many excellent implementations (see here for many, including C and Python ones, and here for the simple XML-RPC server that's part the Python standard library -- not as highly scalable as other approaches but probably fine and convenient for your use case).

It may not be a perfect IPC approach for all cases (or even a perfect RPC one, by all means!), but the convenience, flexibility, robustness, and broad range of implementations outweigh a lot of minor defects, in my opinion.

@D.Shawley 2009-06-29 00:15:01

I haven't used an IPC approach for Python<->C communication but it should work pretty well. I would have the C program do a standard fork-exec and use redirected stdin and stdout in the child process for the communication. A nice text-based communication will make it very easy to develop and test the Python program.

@Craig McQueen 2009-06-29 00:23:47

Is fork-exec possible on Windows? Not sure what the questioner's platform is.

@D.Shawley 2009-06-29 01:03:34

There is a (barely) documented way to do fork-exec under Windows, but I wouldn't do it there. He mentioned unix sockets so I'm assuming that the target isn't Windows. I guess it would be more like spawn and pipe under Windows.

@hhafez 2009-06-29 00:00:42

Have you considered just wrapping your python application in a shell script and invoking it from with in your C application?

Not the most elegant solution, but it is very simple.

@Crashworks 2009-06-29 00:31:21

Indeed if he wants to run Python in a separate process and talk to it over stdin/stdio, that's just about the best solution.

@janneb 2009-06-28 23:48:34

See the relevant chapter in the manual: http://docs.python.org/extending/

Essentially you'll have to embed the python interpreter into your program.

@user13107 2018-05-21 01:40:15

I think the link you gave talks about calling C code from python (which is opposite of what OP wants). Am i right?

Related Questions

Sponsored Content

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

16 Answered Questions

[SOLVED] What are metaclasses in Python?

36 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 3052786 View
  • 2601 Score
  • 36 Answer
  • Tags:   python datetime time

62 Answered Questions

[SOLVED] Calling an external command from Python

23 Answered Questions

[SOLVED] Does Python have a ternary conditional operator?

10 Answered Questions

[SOLVED] Improve INSERT-per-second performance of SQLite?

12 Answered Questions

[SOLVED] How can I make a time delay in Python?

27 Answered Questions

[SOLVED] How do you set, clear, and toggle a single bit?

25 Answered Questions

[SOLVED] How can I safely create a nested directory?

37 Answered Questions

[SOLVED] How do I check whether a file exists without exceptions?

Sponsored Content