Solarian Programmer

My programming ramblings

Python - using C and C++ libraries with ctypes

Posted on July 18, 2019 by Paul

In this article, I will show you how to use C or C++ dynamic libraries from Python, by using the ctypes module from the Python standard library. ctypes is a foreign function library for Python that provides C compatible data types. Although it is mostly used to consume C and C++ libraries, you can use ctypes with libraries written in any language that can export a C compatible API, e.g. Fortran, Rust.

The advantage of using ctypes is that it is already included with your Python installation and that, in theory, you can call any C or C++ shared or dynamic libraries. Another advantage of using ctypes is that you don’t need to recompile the library in order to be able to use it from Python.

A disadvantage of ctypes is that you need to manually wrap the data and functions from the foreign library in Python code. Also for C++ libraries you will need to have the exported functions wrapped in an extern C block. If you need to use a heavy OOP C++ library from Python, I recommend that you look into pybind11.

You can find the source files and image examples used here on the GitHub repo for this article.

The article consists of four parts:

Loading the system C library and directly using functions from the C library

Probably the simplest example of using ctypes is to show you how to directly call functions from the system C library. In the next piece of code I will use ctypes to find the system C library and make it available from Python:

 1 """Simple example of loading and using the system C library from Python"""
 2 import sys, platform
 3 import ctypes, ctypes.util
 4 
 5 # Get the path to the system C library
 6 if platform.system() == "Windows":
 7     path_libc = ctypes.util.find_library("msvcrt")
 8 else:
 9     path_libc = ctypes.util.find_library("c")
10 
11 # Get a handle to the sytem C library
12 try:
13     libc = ctypes.CDLL(path_libc)
14 except OSError:
15     print("Unable to load the system C library")
16     sys.exit()
17 
18 print(f'Succesfully loaded the system C library from "{path_libc}"')

Once you have a handle on the system library, you can call any functions from it. Here, I’m showing examples of using the puts and printf C functions:

1 # ...
2 
3 print(f'Succesfully loaded the system C library from "{path_libc}"')
4 
5 libc.puts(b"Hello from Python to C")
6 
7 libc.printf(b"%s\n", b"Using the C printf function from Python ... ")

Please note the use b to convert the Python string to a byte string as expected by C.

This is what I see on a macOS machine when I run the above script:

1 ~ $ python3 t0.py
2 Succesfully loaded the system C library from "/usr/lib/libc.dylib"
3 Hello from Python to C
4 Using the C printf function from Python ...
5 ~ $

An interesting note here is that, since a Python string is immutable, you can’t change it from C. If you want to pass a mutable string to C from Python, you will need to use the create_string_buffer function provided by ctypes. Example:

 1 # ...
 2 
 3 # Create a mutable string
 4 mut_str = ctypes.create_string_buffer(10)
 5 
 6 # Fill the first 5 elements with 'X':
 7 libc.memset(mut_str, ctypes.c_char(b"X"), 5)
 8 
 9 # Print the modified string
10 libc.puts(mut_str)

If you are careful, you can even do pointer arithmetic. Please note that this is not a good idea! Say that we want to add 4 O after the above 5 X in the mut_str. We’ll start by getting the address of the mut_str, add 5 to this and convert it to a char pointer:

1 # Add 4 'O' to the string starting from position 6
2 p = ctypes.cast(ctypes.addressof(mut_str) + 5, ctypes.POINTER(ctypes.c_char))

Now, we can use memset to fill 4 slots with O:

1 libc.memset(p, ctypes.c_char(b"O"), 4)

Finally, use puts to print the string:

1 libc.puts(mut_str)

Here is what I see, if I run the modified code:

1 ~ $ python3 t0.py
2 Succesfully loaded the system C library from "/usr/lib/libc.dylib"
3 Hello from Python to C
4 Using the C printf function from Python ...
5 XXXXX
6 XXXXXOOOO
7 ~ $

Building a shared C library and using the library from Python

In the next example I assume that you have Clang or GCC installed on your machine. The code should work the same on Linux, Windows or macOS. If you need to install GCC on Windows check this article. The MSVC compiler is a bit particular about the way you need to annotate functions in a shared library. I will show you how to use it in the next part of this article.

Let’s assume that you keep your function declarations in a header file named mylib.h:

 1 #pragma once
 2 
 3 #ifdef __cplusplus
 4 extern "C" {
 5 #endif
 6 
 7 void test_empty(void);
 8 float test_add(float x, float y);
 9 void test_passing_array(int *data, int len);
10 
11 #ifdef __cplusplus
12 }
13 #endif

and here is the implementation of the above three functions:

 1 #include <stdio.h>
 2 #include "mylib.h"
 3 
 4 void test_empty(void) {
 5     puts("Hello from C");
 6 }
 7 
 8 float test_add(float x, float y) {
 9     return x + y;
10 }
11 
12 void test_passing_array(int *data, int len) {
13     printf("Data as received from Python\n");
14     for(int i = 0; i < len; ++i) {
15         printf("%d ", data[i]);
16     }
17     puts("");
18 
19     // Modifying the array
20     for(int i = 0; i < len; ++i) {
21         data[i] = -i;
22     }
23 }

First function simply prints a message from C, second function receives as arguments two floats and returns their sum. Last function receives an array of integers and the array length, prints the array as received and than modifies the array.

If you want to build the above code as a shared library, with Clang on macOS, you can use these commands:

1 ~ $ clang -std=c11 -Wall -Wextra -pedantic -c -fPIC mylib.c -o mylib.o
2 ~ $ clang -shared mylib.o -o mylib.dylib

On a Linux system with GCC you can use:

1 ~ $ gcc -std=c11 -Wall -Wextra -pedantic -c -fPIC mylib.c -o mylib.o
2 ~ $ gcc -shared mylib.o -o mylib.so

On Windows, assuming that you have GCC installed:

1 ~ $ gcc -std=c11 -Wall -Wextra -pedantic -c -fPIC mylib.c -o mylib.o
2 ~ $ gcc -shared mylib.o -o mylib.dll

Once we have the shared library, we can write a Python wrapper, mylib.py, that loads the library and exports the functions. Let’s start, like before with the code that loads the library:

 1 """ Python wrapper for the C shared library mylib"""
 2 import sys, platform
 3 import ctypes, ctypes.util
 4 
 5 # Find the library and load it
 6 mylib_path = ctypes.util.find_library("mylib")
 7 if not mylib_path:
 8     print("Unable to find the specified library.")
 9     sys.exit()
10 
11 try:
12     mylib = ctypes.CDLL(mylib_path)
13 except OSError:
14     print("Unable to load the system C library")
15     sys.exit()

Please note that we don’t specify the library extension, find_library will match the system library extension. This makes our code portable on most operating systems. A possible problem is that find_library uses the system library search path and the local folder. If you want to restrict the search path only to the local folder use something like:

1 mylib_path = ctypes.util.find_library("./mylib")

Another potential problem is that, on UNIX like systems, find_library will search for both libmylib and mylib unless you specify the search path. When you search for your own libraries, it is probably a good ideas to prefer:

1 mylib_path = ctypes.util.find_library("./mylib")

instead of the more general:

1 mylib_path = ctypes.util.find_library("mylib")

Next, let’s make the functions available at the module level:

 1 #
 2 # ... Load libray code ...
 3 #
 4 
 5 # Make the function names visible at the module level and add types
 6 test_empty = mylib.test_empty
 7 
 8 test_add = mylib.test_add
 9 test_add.argtypes = [ctypes.c_float, ctypes.c_float]
10 test_add.restype = ctypes.c_float
11 
12 test_passing_array = mylib.test_passing_array
13 test_passing_array.argtypes = [ctypes.POINTER(ctypes.c_int), ctypes.c_int]
14 test_passing_array.restype = None

Please note the way we specify the types of the input, argtypes, and output, restype, for the last two C functions. While this is not always required, it is usually a good idea to be specific and it will give us more meaningful error messages if we try to pass the wrong data type.

OK, let’s use the Python wrapper now. Create a new file named test_mylib.py and add the next code:

1 import mylib
2 import ctypes
3 
4 print("Try test_empty:")
5 mylib.test_empty()
6 
7 print("\nTry test_add:")
8 print(mylib.test_add(34.55, 23))

This is what I see, if I run the above code on my machine:

1 ~ $ python3 test_mylib.py
2 Try test_empty:
3 Hello from C
4 
5 Try test_add:
6 57.54999923706055
7 ~ $

If you pass the wrong data type to test_add, you’ll get an error, e.g.:

1 >>> import mylib
2 >>> mylib.test_add(23, "abba")
3 Traceback (most recent call last):
4   File "<stdin>", line 1, in <module>
5 ctypes.ArgumentError: argument 2: <class 'TypeError'>: wrong type
6 >>>

This is how you define a fixed sized array in Python using ctypes:

1 my_array = (ctypes.c_int * number_of_elements)()

You can also initialize the array from other Python objects, e.g. for a list:

1 my_array = (ctypes.c_int * number_of_elements)([1,2,3,4])

Let’s add an extra test to test_mylib.py:

 1 import mylib
 2 import ctypes
 3 
 4 print("Try test_empty:")
 5 mylib.test_empty()
 6 
 7 print("\nTry test_add:")
 8 print(mylib.test_add(34.55, 23))
 9 
10 # Create a 25 elements array
11 numel = 25
12 data = (ctypes.c_int * numel)(*[x for x in range(numel)])
13 
14 # Pass the above array and the array length to C:
15 print("\nTry passing an array of 25 integers to C:")
16 mylib.test_passing_array(data, numel)
17 
18 print("data from Python after returning from C:")
19 for indx in range(numel):
20     print(data[indx], end=" ")
21 print("")

This is what I see if I run the above code on my machine:

 1 ~ $ python3 test_mylib.py
 2 Try test_empty:
 3 Hello from C
 4 
 5 Try test_add:
 6 57.54999923706055
 7 
 8 Try passing an array of 25 integers to C:
 9 Data as received from Python
10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
11 data from Python after returning from C:
12 0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16 -17 -18 -19 -20 -21 -22 -23 -24
13 ~ $

Building a shared C library with MSVC and using the library from Python

As promised, let’s build the library with MSVC. In order to build a dynamic library using the Microsoft cl compiler we can use this command:

1 cl /LD mylib.c

while you will end up with a mylib.dll binary, you won’t be able to use it because the symbols are not exported. MSVC requires that every function that you want to be exported to be prefixed with:

1 __declspec(dllexport)

So, we need to modify the mylib.h file if we want to be compatible with the MSVC compiler:

 1 #pragma once
 2 
 3 #ifdef _WIN32
 4     #ifdef BUILD_MYLIB
 5         #define EXPORT_SYMBOL __declspec(dllexport)
 6     #else
 7         #define EXPORT_SYMBOL __declspec(dllimport)
 8     #endif
 9 #else
10     #define EXPORT_SYMBOL
11 #endif
12 
13 
14 #ifdef __cplusplus
15 extern "C" {
16 #endif
17 
18 EXPORT_SYMBOL void test_empty(void);
19 EXPORT_SYMBOL float test_add(float x, float y);
20 EXPORT_SYMBOL void test_passing_array(int *data, int len);
21 
22 #undef EXPORT_SYMBOL
23 #ifdef __cplusplus
24 }
25 #endif

The above code will use dllexport when the library is built, if you define the BUILD_MYLIB macro. E.g.:

1 cl /LD /DBUILD_MYLIB mylib.c

Another observation is that EXPORT_SYMBOL is active only on Windows!

Once the DLL was properly generated and the symbols exported, you can use the library from Python without a problem.

The above header is also compatible with Clang and GCC on Windows:

1 gcc -std=c11 -Wall -Wextra -pedantic -c -fPIC -DBUILD_MYLIB mylib.c -o mylib.o
2 gcc -shared mylib.o -o mylib.dll

Optionally, only if you want to be able to use the DLL generated by GCC with a different C compiler, like MSVC, you will need to add an extra parameter to the second line:

1 gcc -std=c11 -Wall -Wextra -pedantic -c -fPIC -DBUILD_MYLIB mylib.c -o mylib.o
2 gcc -shared mylib.o -o mylib.dll -Wl,--out-implib,libmylib.a

Building a shared C++ library and using the library from Python

In general, when you want to interface Python with C++ you are better off with using a dedicated library like pybind11. However, if all you need is to call a couple of C++ functions from Python and the C++ library has a C API, you can use ctypes.

In order to exemplify the approach, I will use a small BMP image file manipulation I wrote a few months ago. You can find the original article here, if you want to read it.

For the C API, I will define functions to read an image from disk, write an image to disk, create an image in memory, get the pixel data as a one dimensional array, fill a rectangular region of the image and finally a function to release the resources used by the library.

Here is the header of the library we want to call from Python:

 1 #pragma once
 2 
 3 #ifdef __cplusplus
 4     #include <cstdint>
 5 #else
 6     #include <stdint.h>
 7     #include <stdbool.h>
 8 #endif
 9 
10 #ifdef _WIN32
11     #ifdef BUILD_CBMP
12         #define EXPORT_SYMBOL __declspec(dllexport)
13     #else
14         #define EXPORT_SYMBOL __declspec(dllimport)
15     #endif
16 #else
17     #define EXPORT_SYMBOL
18 #endif
19 
20 #ifdef __cplusplus
21 extern "C" {
22 #endif
23 
24 typedef struct {
25     void *handle;
26     int32_t width;
27     int32_t height;
28     int32_t channels;
29 } CBMP;
30 
31 EXPORT_SYMBOL CBMP *BMP_read(const char *fname);
32 EXPORT_SYMBOL CBMP *BMP_create(int32_t width, int32_t height, bool has_alpha);
33 EXPORT_SYMBOL void BMP_destroy(CBMP **cbmp);
34 
35 EXPORT_SYMBOL void BMP_write(CBMP *cbmp, const char *fname);
36 EXPORT_SYMBOL uint8_t *BMP_pixels(CBMP *cbmp);
37 EXPORT_SYMBOL void BMP_fill_region(CBMP *cbmp, uint32_t x0, uint32_t y0, uint32_t w, uint32_t h, uint8_t B, uint8_t G, uint8_t R, uint8_t A);
38 
39 #ifdef __cplusplus
40 }
41 #endif

What is different versus our previous examples is that, this time, the header file contains a struct and functions to manipulate it. The handle is an opaque pointer to the underlying C++ class that is not visible from C.

Let’s start by writing a Python wrapper around this library. The file is named cbmp.py. The initial boilerplate code that loads the library is almost identical with what we’ve used before:

 1 """ Python wrapper for the C shared library cbmp"""
 2 import sys, platform
 3 import ctypes, ctypes.util
 4 
 5 # Find the library and load it
 6 cbmp_path = ctypes.util.find_library("./cbmp")
 7 if not cbmp_path:
 8     print("Unable to find the specified library.")
 9     sys.exit()
10 
11 try:
12     cbmp = ctypes.CDLL(cbmp_path)
13 except OSError:
14     print("Unable to load the system C library")
15     sys.exit()

ctypes has a helper Struct class from which we can inherit. There is basically a one to one mapping to the original C struct:

 1 #
 2 # ... Load library ...
 3 #
 4 
 5 class CBMP(ctypes.Structure):
 6     _fields_ = [
 7         ('handle', ctypes.c_void_p),
 8         ('width', ctypes.c_int32),
 9         ('height', ctypes.c_int32),
10         ('channels', ctypes.c_int32)
11     ]

What is interesting in the above code is that we can map the names of the C structure and data types by using a list of pairs name type.

Next, we can write the wrapping code for the six functions declared in cbmp.h:

 1 # ... Prev. code ...
 2 
 3 # Make the function names visible at the module level and add types
 4 BMP_read = cbmp.BMP_read
 5 BMP_read.argtypes = [ctypes.c_char_p]
 6 BMP_read.restype = ctypes.POINTER(CBMP)
 7 
 8 BMP_create = cbmp.BMP_create
 9 BMP_create.argtypes = [ctypes.c_int32, ctypes.c_int32, ctypes.c_bool]
10 BMP_create.restype = ctypes.POINTER(CBMP)
11 
12 BMP_destroy = cbmp.BMP_destroy
13 BMP_destroy.argtypes = [ctypes.POINTER(ctypes.POINTER(CBMP))]
14 BMP_destroy.restype = None
15 
16 BMP_write = cbmp.BMP_write
17 BMP_write.argtypes = [ctypes.POINTER(CBMP), ctypes.c_char_p]
18 BMP_write.restype = None
19 
20 BMP_pixels = cbmp.BMP_pixels
21 BMP_pixels.argtypes = [ctypes.POINTER(CBMP)]
22 BMP_pixels.restype = ctypes.POINTER(ctypes.c_uint8)
23 
24 BMP_fill_region = cbmp.BMP_fill_region
25 BMP_fill_region.argtypes = [ctypes.POINTER(CBMP), ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint32, ctypes.c_uint8, ctypes.c_uint8, ctypes.c_uint8, ctypes.c_uint8]
26 BMP_fill_region.restype = None

The library can read and write BMP images with three or four channels. The fourth channel is the transparency channel. The actual pixel data is stored as a contiguous one dimensional array of unsigned chars and can be directly accessed from Python, by using the BMP_pixels function. A BMP image typically has the channels in BGR or BGRA order.

Here are some examples of using the library from Python:

 1 import cbmp
 2 import ctypes
 3 
 4 # Read a BMP image with transparency, draw two rectangles on it, save it and release the object
 5 img = cbmp.BMP_read(b"cpp-bmp-images/Shapes.bmp")
 6 cbmp.BMP_fill_region(img, 0, 0, 100, 200, 0, 0, 255, 255)
 7 cbmp.BMP_fill_region(img, 150, 0, 209, 203, 0, 255, 0, 255)
 8 cbmp.BMP_write(img, b"Shapes_copy.bmp")
 9 cbmp.BMP_destroy(img)
10 
11 # Read an RGB BMP image, draw two rectangles on it, save it and release the object
12 img = cbmp.BMP_read(b"cpp-bmp-images/Shapes_24.bmp")
13 cbmp.BMP_fill_region(img, 0, 0, 100, 200, 0, 0, 255, 255)
14 cbmp.BMP_fill_region(img, 150, 0, 209, 203, 0, 255, 0, 255)
15 cbmp.BMP_write(img, b"Shapes_24_copy.bmp")
16 cbmp.BMP_destroy(img)
17 
18 # Create a 4 channels image in memory, save it empty, draw one rectangle,
19 # save it agaian, release the object memory
20 img = cbmp.BMP_create(600, 600, True)
21 cbmp.BMP_write(img, b"create_empty_32.bmp")
22 cbmp.BMP_fill_region(img, 100, 100, 200, 300, 99, 75, 180, 200)
23 cbmp.BMP_write(img, b"create_rect_32.bmp")
24 cbmp.BMP_destroy(img)
25 
26 # Create a 3 channels image in memory, get a handle to the raw pixel data
27 # modify the pixels in Python by creating a colored rectangle,
28 # write the image to disk.
29 img = cbmp.BMP_create(500, 500, False)
30 img_pixels = cbmp.BMP_pixels(img)
31 print(img.contents.width, img.contents.height, img.contents.channels)
32 channels = img.contents.channels
33 for y in range(img.contents.height//2):
34     for x in range(img.contents.width//2):
35         indx = channels * (y * img.contents.width + x)
36         img_pixels[indx + 0] = 150  # Blue
37         img_pixels[indx + 1] = 50   # Green
38         img_pixels[indx + 2] = 200  # Red
39         if channels == 4:
40             img_pixels[indx + 3] = 255  # Alpha
41 
42 cbmp.BMP_write(img, b"create_rect_24.bmp")
43 cbmp.BMP_destroy(img)

If you want to learn more about Python, I recommend reading Python Crash Course by Eric Matthes, the book is intended for beginners:

If you want to learn about C99/C11 I would recommend reading 21st Century C: C Tips from the New School by Ben Klemens:

or the classic C Bible, The C Programming Language by B.W. Kernighan, D.M. Ritchie:


Show Comments