Solution for distributing MANY simple network tasks? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.

Closed 7 years ago.

Improve this question

I would like to create some sort of a distributed setup for running a ton of small/simple REST web queries in a production environment. For each 5-1开发者_StackOverflow社区0 related queries which are executed from a node, I will generate a very small amount of derived data, which will need to be stored in a standard relational database (such as PostgreSQL).

What platforms are built for this type of problem set? The nature, data sizes, and quantities seem to contradict the mindset of Hadoop. There are also more grid based architectures such as Condor and Sun Grid Engine, which I have seen mentioned. I'm not sure if these platforms have any recovery from errors though (checking if a job succeeds).

What I would really like is a FIFO type queue that I could add jobs to, with the end result of my database getting updated.

Any suggestions on the best tool for the job?

---

Have you looked at Celery?