Tens of millions of small files, where to host them? (prefer a free one) [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

开发者_如何学Python

We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.

Closed 5 years ago.

Improve this question

I have over 10 million small static html files (10K each). And I am an user of bluehost.com. It have a limitation of 50000 files. It sent an email to warn me that if I didn't delete the files in 30 days, it would disable my account.

So I am looking for a free service to host my files. I considered google app engine, but it has a even more strict limitation: no more than 1000 files.(each should not larger than 1 Mb). And it seems that I could upload the files to code.google.com which provides free project hosting service.

Any good suggestions? I prefer a free one or a cheap one. And it should have a programming interface to upload and download files. Thank you in advance.

---

I would consider converting all the files into a database and coding a small server side script to retrieve the data, then use some rewriting rules to redirect the visitor to the script.

Most web hosts nowadays offer some sort of server side language a database of some sort. Many also allow you to use .htaccess files to put your rewrite rules in.

---

10 million files made by hand ? If they where made by a program try to move the program into a dynamic web language like php.

---

Use zip and a frontend that unpacks the files (if needed).

10 million files is generated code. Don't. Just create on demand.

[edit] Then you don't need to store the pages at all. There are data structures that let you reconstruct the original page while being fast searchable combined with using little extra storage space.

---

You can certainly do this on a Linode (cheapest plan $20/month). But I think that might be overkill. Amazon S3 charges by the gigabyte, so you wouldn't pay very much for that. As far as straight web hosting goes, you've got me - I don't know of a provider that will let you do that to their poor directories.

---

Do you need access to all the files all the time?

You could archive them into zip files, as text tends to compress quite well, and maybe that would give you the required space saving.