开发者

GAE: scheduled import of big gzipped file from a third party site

开发者 https://www.devze.com 2023-02-14 22:32 出处：网络

I\'m working on a Python web app that needs to import big (in terms of GAE limits) gzipped files from a third party site on regular basis. Think of rdf exports DMOZ project is producing on regular int

相关专题：google-app-engine

I'm working on a Python web app that needs to import big (in terms of GAE limits) gzipped files from a third party site on regular basis. Think of rdf exports DMOZ project is producing on regular intervals.

This means daily fetching of a 500+ MB gzip file, gunzipping, parsing, processing and storing the results in GAE's datastore for later use.

What's the proper way to implement this functionali开发者_如何学Pythonty on GAE, having in mind the limits for maximum download, processing time, etc?

The limit on downloaded file size in App Engine is currently 64MB. As a result, you've got two options:

Use HTTP Range headers to download and process the file in chunks.
Use an external service to do the download, split it into pieces, and send the pieces to your App Engine app.

My initial gut reaction (without knowing what's inside the gzipped file) is to do the processing somewhere else (AWS?) and then pushed the processed data to your GAE application in small bits.