How to structure my application for several tasks using redis with gevent or threading in Python

I hav开发者_Python百科e a process which sends financial tick data through redis pubsub in realtime. Now I want my Python Application to handle the input data (json) for instance calculations like moving average and so on. the results I want to be send back via redis to other tasks (doing further calculations based on the results of the 1st task). Further I want to trigger some tasks regularly once a day or each second. With this complex and unforseen structure issue I've stumbled upon solutions like gevent, Celery or just Threads.

But what I'm wondering is What are my options to do this the right way? How can I structure my redis pubsub by doing Worker/Task stuff the most efficient way? So, suggestions are welcome, in the lines of libraries (if you've used any of the mentioned please share your experiences), techniques (Python's structure Best Practice), how to make use of redis's pubsub to do the job the best way.

---

If any of those calculations are computationally expensive and you do them in python and you want scalability then celery makes perfect sense.

gevent would just make your code more efficient in specific cases, but won't help you in terms of scalability. That also holds true if you use threads.

Bear in mind that you can configure celery to run the worker pool on gevent (or eventlet)