How would you write files to s3 in parallel? i.e. not serialized one by one

I have 10K files t开发者_开发知识库o write to S3 every hour, how can I do this in parallel?

Use multiple threads or try EventMachine with a S3 driver. You need to a throughput of about 3 files per second which depending on the file size shouldn't be that hard. Make sure to use HTTP keep-alive to not reconnect in between each file.