pthread with callback to python VM

Let's say I have a python script which loads a shared library (SL) through ctypes.

• The SL sets up a pthread T1
• The python script configures callbacks through the SL i.e. python script calls functions from the SL with references to python callables

alt text http://www.gliffy.com/pubdoc/1993061/L.jpg

Now, let's say T1 calls a "callback" function, are the following assumptions true:

• the callback function on the Python side is executed within the context of T1
• I can use a queue to communicate between T1 and the Python VM
• I would need to poll the said queue on the Python VM side

I understand all the concepts of threading, shared state etc. but I haven't dug very deep on the mul开发者_StackOverflow中文版ti-threading side with Python. Since there is an adaptation layer which I do not know enough at the moment (ctypes), I am afraid I'll be missing some key aspects of the process.

---

Polling the queue isn't normally necessary (you can devote another thread on the Python side to doing blocking .get calls on it), but that's not a big deal. Problem is, with such an arrangement, you might get caught by the GIL -- see the three links from this wikipedia page for ample treatments thereof.

When you interface to/from C with C (or Cython) code using the Python C API, you can release and acquire the GIL pretty simply, at least, hopefully avoiding deadlocks and the like; with ctypes, GIL operations are automated in callback to/from C situations, so if there's any other lock in play a deadlock is a risk (since things are not within your control you can't easily ensure Djikstra's Banker Algorithm is applied).

---

This can be easily implemented with Cython . I have written a sample at https://github.com/sachinpc/cython/tree/master/Demos/mt_callback

The example shows how to wrap python callback in cython, when being called from a pthread. For your specific use-case , you could add to a Queue or dequeue object in the thread code from the callback function . In the sameple , I have added prints for both pthread-id and python-thread-id in the extension .