Self hosted S3 alternative [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. 开发者_如何学JAVA Closed 10 years ago.

I am looking for an S3 alternative which relies on a RESTful API, so that I can simply insert links such as http://datastore1.example.com/ID and they are directly downloadable. I have looked at RIAK and Bitcache. They both seem very nice: http://bitcache.org/api/rest but they have one problem. I want to be the only one who can upload data. Else anyone could use our datastore by sending a PUT Request.

Is there a way to configure RIAK so that everyone can "GET" but not everyone can PUT or DELETE files except me? Are there other services which you can recommend?

Also adding Bounty :)

Requirements:

• RESTful API
• Guests GET only
• Runs on Debian

Very nice to have:

• auto distributed

EDIT: To clarify I don't want any connection to S3 I have great servers just lying around with harddrives and very good network connection (3Gbps) I don't need S3..

---

The Riak authors recommend to put a HTTP proxy in front of Riak in order to provide access control. You can chose any proxy server you like (such as nginx or Apache), and any access control policy you like (such as authorization based on IP addresses, HTTP basic auth, or cookies, assuming your proxy server can handle it). For example, in nginx, you might specify limit_except (likewise LimitExcept in Apache).

Alternatively, you could also add access control to Riak directly. It's based on Webmachine, so one approach would be to implement is_authorized.

---

Based on the information that you have given, I would suggest Eucalyptus ( http://open.eucalyptus.com/ ). They do have an S3 compatible storage system.

---

The reliable, distributed object store RADOS, which is part of the ceph file system, provides an S3 gateway.

We used the Eucalyptus storage system, Walrus, but we had reliably problems.

---

If you are looking for a distributed file system, why don't you try hadoop hdfs?

http://hadoop.apache.org/common/docs/r0.17.0/hdfs_design.html

There is a Java API available:

http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/fs/FileSystem.html

Currently, security is an issue - at least if you have access to a terminal:

http://developer.yahoo.com/hadoop/tutorial/module2.html#perms

But you could deploy hdfs, put an application server (using the Java API) in front of it (GlassFish) and use Jersey to build the RESTful API:

http://jersey.java.net/

If you're interested in building such a thing, please let me know, for I may be building something similar quite soon.

You can use the Cloudera Hadoop Distribution to make life a bit more easy:

http://www.cloudera.com/hadoop/

Greetz, J.

---

I guess that you should ask your question on serverfault.com , as it's more system related. Anyway, I can suggest you mogileFS which scales very well : http://danga.com/mogilefs/ .

---

WebDAV is about as RESTful as it gets and there are many implementations that scale to various uses. In any case, if it is REST and it is HTTP then whatever authentication scheme that the server supports should allow you to control who can upload.

---

You can develop it yourself as a web app or a part of your existing application. It will consume HTTP requests, retrieve their URI component, convert it to S3 object name and use getObject() to get its content (using one of available S3 SDKs, for example AWS Java SDK).

You can try a hosted solution - s3auth.com (I'm a developer). It's an open source project, and you can see how this mechanism is implemented internally at one of its core classes. HTTP request is processed by the service and then re-translated to Amazon S3 internal authentication scheme.