Anatomy of a Python C Module

March 13, 2008

Traducción Español

Writing Python modules in C is relatively easy. The main reason to do so is to increase performance over using Python code. I will demonstrate how to implement the following Python function in C. This function was originally found here:

def fib2(n): # return Fibonacci series up to n
    """Return a list containing the Fibonacci series up to n."""
    result = []
    a, b = 0, 1
    while b < n:
        result.append(b)    # see below
        a, b = b, a+b
    return result

Note that this isn’t a particularly slow function—in fact it’s quite fast—it simply has a number of aspects interesting to implementing a C module, like creating and appending a Python list in C. The Python Example for creating a C module isn’t as comprehensive as it could be, so hopefully implementing the Fibonacci sequence in C will explain just a bit more.

To start, we always include Python.h:

#include <Python.h>

Next we’ll create the fib function. First we define the function as a Python Object, which is passed arguments:

static PyObject *
fib(PyObject *self, PyObject *args)
{

Then we get to the body of the fucntion. First, let’s initialize some variables:

int a = 0, b = 1, c, n;

Then we move on to parsing the arguments passed to the function. For this we utilize PyArg_ParseTuple. Read more in the documentation page Parsing arguments and building values, which gives an overview on how to parse different types of arguments. Our example, however, only accepts a single integer. If that doesn’t work, we return NULL.

if (!PyArg_ParseTuple(args, "i", &n))
    return NULL;

Then we instantiate a new Python list, using PyList_New, which accepts an integer as the length of the list. Since we don’t know how long the list will be when we finish, we start with zero.

PyObject *list = PyList_New(0);

Then we get to the guts of the actual calculation. The line we pay attention to is the PyList_Append(list, PyInt_FromLong(b));, as that is where and how we add another item to the list. PyList_Append is analogous to Python’s list.append() method. We use PyInt_FromLong to create a Python object from the integer in the loop.

while(b < n){
    PyList_Append(list, PyInt_FromLong(b));
    c = a+b;
    a = b;
    b = c;
}

And then we return the list:

    return list;
}

That makes up the guts of the function, but how do we integrate this into Python as a module? First we create a PyMethodDef object with the functions we want to build into the module. Since we only have the one function, we only have one definition, like so:

PyMethodDef methods[] = {
    {"fib", fib, METH_VARARGS, "Returns a fibonacci sequence as a list"},
    {NULL, NULL, 0, NULL}
};

The last step is to initialize the module. To understand what’s going on here, read through this page as it has a thorough explanation of everything happening.

PyMODINIT_FUNC 
initfib()
{
    (void) Py_InitModule("fib", methods);   
}

Now that our Python C module is complete, we need to compile it. The easiest way to do so is by using the distutils module. We create a setup.py like so:

from distutils.core import setup, Extension

setup(name = "Fib",
      version = "1.0",
      ext_modules = [Extension("fib", ["fib.c"])])

That tells distutils that our module is located in fib.c. Now we run:

$ python setup.py build
$ python setup.py install

And it’s installed! To use it we import the module and use the function accordingly:

$ python
>>> import fib
>>> fib.fib(123)
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]

A fibonacci sequence is easy to calculate, and doing so in C is an exercise to display how one might implement their own function or module in C. However, despite being easy to calculate, the C version outperforms its Python equivalent by a factor of four. This simple example should show how easy it can be—and useful—it is to implement C extensions.

There’s a lot more to this process, so if you are interested, I highly recommend reading through the Extending and Embedding the Python Interpreter document.

Comments

This is a good concise writeup. Where were you six months ago when I was writing my first extension module? :)

Posted by Ned Batchelder

Thank you for this. Well-written and and (more importantly) correct. Helped to get me from a C file to a Python module in ~10 minutes.

Posted by Andrew Winslow

Would you mind if i do a spanish translation of this post? Is just great.

Posted by Edorka

Edorka: No problem, just make sure to credit me :)

Posted by SuperJared

done, you have it at http://www.geosincrona.com/?p=127

I've done some changes (not 1st or 2nd persons form and will add some details to de install process)

In near future i will try to extend the tutorial to build modules with objects and attributes, i'll tell you.

Thanks a lot for your job :)

Posted by Edorka

When you initialize the new Python list using:

PyObject *list = PyList_New(0);

how would I alter this to declare a new integer named sum? I'm using this tutorial to make another function, in which I need to declare a new integer, but I can't figure out how to do this. If you know of any helpful links, any help would be greatly appreciated. Thanks!

Mike

Posted by Mike Jones

I believe there may be a typo in the setup.py example.

<pre>
from distutils.core import setup, Extension

setup(name = "Fib",
version = "1.0",
ext_modules = [Extension("fib", ["fib.c"])])
</pre>

The first arg to the Extension constructor should be "_fib", not "fib". I'm building this on Windows using MSVC, and if the value is "fib", I get the following error:

<pre>
C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link.exe /DLL /nologo
/INCREMENTAL:NO /LIBPATH:d:\programs\Python24\libs /LIBPATH:d:\programs\Python2
4\PCBuild /EXPORT:initexample build\temp.win32-2.4\Release\example.obj build\tem
p.win32-2.4\Release\example_wrap.obj /OUT:build\lib.win32-2.4\example.pyd /IMPLI
B:build\temp.win32-2.4\Release\example.lib
example_wrap.obj : error LNK2001: unresolved external symbol initexample
build\temp.win32-2.4\Release\example.lib : fatal error LNK1120: 1 unresolved ext
ernals
LINK : fatal error LNK1141: failure during build of exports file
error: command '"C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\link
.exe"' failed with exit status 1141
</pre>

On Linux, it builds, but fails when you try to import the module in Python.

The intro article on SWIG (http://www.swig.org/Doc1.3/Python.html) touches upon this point somewhat.

Posted by Misha

When I try building this example, I get the following errors:

fib.c
fib.c(9) : error C2275: 'PyObject' : illegal use of this type as an expression
c:\python26\include\object.h(108) : see declaration of 'PyObject'
fib.c(9) : warning C4013: 'list' undefined; assuming extern returning int
fib.c(11) : warning C4047: 'function' : 'PyObject *' differs in levels of indire
ction from 'int (__cdecl *)()'
fib.c(11) : warning C4024: 'PyList_Append' : different types for formal and actu
al parameter 1
fib.c(16) : warning C4047: 'return' : 'PyObject *' differs in levels of indirect
ion from 'int (__cdecl *)()'
error: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' fa
iled with exit status 2

Any ideas on why using PyObject causes an error? I'm running Windows XP and MSVC++ 9.0.

Posted by Pete

Sorry, it's actually the following error, although the main problem of PyObject is still there:

fib.c
fib.c(9) : error C2275: 'PyObject' : illegal use of this type as an expression
c:\python26\include\object.h(108) : see declaration of 'PyObject'
fib.c(9) : error C2065: 'list' : undeclared identifier
fib.c(11) : error C2065: 'list' : undeclared identifier
fib.c(11) : warning C4047: 'function' : 'PyObject *' differs in levels of indire
ction from 'int'
fib.c(11) : warning C4024: 'PyList_Append' : different types for formal and actu
al parameter 1
fib.c(16) : error C2065: 'list' : undeclared identifier
fib.c(16) : warning C4047: 'return' : 'PyObject *' differs in levels of indirect
ion from 'int'
error: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe"' fa
iled with exit status 2

Posted by Pete

Did you ever find a solution to Pete's problem? I'm having the same trouble with a different extension module.

Posted by limscoder

Thank you very much for this example: much clearer than the Python example! Now I can call C programs from within the interpreter. Great!
Only one minor problem when I do "python setup.py install" the installer wants to copy the program under the general Python installation where I don't have write permission... is there a way to tell install where to put the compiled file?
Sorry I am rather new to Python. In any case it's not a big deal because one can manually copy fib.so..

Posted by Gio

hey, thanks for this ... but it seems there is a memory leak ...

>>> import fib
>>> while True:
... fib.fib(3212)

Posted by deli

BTW I have just found out how to build your C extension in your current directory:

python setup.py build_ext --inplace

Posted by Gio

Is this the internal or external anatomy of a python

Posted by al

Thank you so much for writing this simple example up.

I copy pasted the source here:
http://github.com/ivanistheone/arXivL...

Posted by ivan

@deli It is true that this example has a memory leak.
Here is the corrected code:

number = PyInt_FromLong(b); // Need to get the pointer (see below)
PyList_Append(list, number);
Py_DECREF(number); // so I can DECREF it if necessary
// more info: http://www.python.org/doc/2.3.5/api/r...

and somewhere above you have to define:

PyObject *number;

now there is no more memory leak ;)

Posted by ivan

Add your comment

No HTML; Only URLs and line breaks are converted.