Enterprise Platform in Python, Design Advice

I am starting the design of a somewhat large enterprise platform in Python, and was wondering if you guys can give me some advice as to how to organize the various components and which packages would help achieve the goals of scalability, maintainability, and reliability.

The system is basically a service that collects data from various outside sources, with each outside source having its own separate application. These applications would poll a central database and get any requests that have been submitted to perform on the external source.

There will be a main website and REST/SOAP API that should also have access to the central data service.

My initial thought was to use Django for the web site, web 开发者_JS百科service and data access layer (using its built-in ORM), and then the outside source applications can use the web service(s) to get the information they need to process the request and save the results. Using this method would allow me to have multiple instances of the service applications running on the same or different machines to balance out the load. Are there more elegant means of accomplishing this? i've heard of messaging systems such as MQ, would something like that be beneficial in this scenario?

My other thought was to use a completely separate data service not based on Django, and use some kind of remoting or remote objects (in they exist in Python) to interact with the data model. The downside here would be with the website which would become much slower if it had to push all of its data requests through a second layer.

I would love to hear what other developers have come up with to achieve these goals in the most flexible way possible.

---

Consider using Celery. It lets your web apps do as little as possible, then fire off other tasks that'll be completed later. It uses AMQP (RabbitMQ) underneath, but it's in Python and plays very well with Django.

http://celeryproject.org/

(If you want to learn more AMQP, I wrote up some slides: http://johntellsall.blogspot.com/2009/11/message-queuing-slides-and-source-code.html )

---

I think that your architecture sounds fine. One comment is that SOAP is usually considered heavy in the python community. Have you considered JSON or something else instead? By now, there are JSON libraries for most languages.

Another possibility for running code remotely is pyro (Python Remote Objects), which works fine on Windows, and is cross-platform.

There are a couple of gotchas writing services in python (like if your service always dies after a while, make sure it's not writing to stdout and filling up its buffer), but if you're considering this route then I assume you're on top of that. :)