S3 limit to objects in a bucket_问答_开发者_运维开发者技术经验分享

Does anyone know if there is a li开发者_运维百科mit to the number of objects I can put in an S3 bucket? can I put a million, 10 million etc.. all in a single bucket?

According to Amazon:

Write, read, and delete objects containing from 0 bytes to 5 terabytes of data each. The number of objects you can store is unlimited.

Source: http://aws.amazon.com/s3/details/ as of Sep 3, 2015.

It looks like the limit has changed. You can store 5TB for a single object.

The total volume of data and number of objects you can store are unlimited. Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

http://aws.amazon.com/s3/faqs/#How_much_data_can_I_store

There is no limit on objects per bucket.
There is a limit of 100 buckets per account (you need to request amazon if you need more).
There is no performance drop even if you store millions of objects in a single bucket.

From docs,

There is no limit to the number of objects that can be stored in a bucket and no difference in performance whether you use many buckets or just a few. You can store all of your objects in a single bucket, or you can organize them across several buckets.

as of Aug 2016

While you can store an unlimited number of files/objects in a single bucket, when you go to list a "directory" in a bucket, it will only give you the first 1000 files/objects in that bucket by default. To access all the files in a large "directory" like this, you need to make multiple calls to their API.

There are no limits to the number of objects you can store in your S3 bucket. AWS claims it to have unlimited storage. However, there are some limitations -

By default, customers can provision up to 100 buckets per AWS account. However, you can increase your Amazon S3 bucket limit by visiting AWS Service Limits.
An object can be 0 bytes to 5TB.
The largest object that can be uploaded in a single PUT is 5 gigabytes
For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

That being said if you really have a lot of objects to be stored in S3 bucket consider randomizing your object name prefix to improve performance.

When your workload is a mix of request types, introduce some randomness to key names by adding a hash string as a prefix to the key name. By introducing randomness to your key names the I/O load will be distributed across multiple index partitions. For example, you can compute an MD5 hash of the character sequence that you plan to assign as the key and add 3 or 4 characters from the hash as a prefix to the key name.

More details - https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-performance-improve/

-- As of June 2018

"You can store as many objects as you want within a bucket, and write, read, and delete objects in your bucket. Objects can be up to 5 terabytes in size."

from http://aws.amazon.com/s3/details/ (as of Mar 4th 2015)

@Acyra- performance of object delivery from a single bucket would depend greatly on the names of the objects in it.

If the file names were distanced by random characters then their physical locations would be spread further on the AWS hardware, but if you named everything 'common-x.jpg', 'common-y.jpg' then those objects will be stored together.

This may slow delivery of the files if you request them simultaneously but not by enough to worry you, the greater risk is from data-loss or an outage, since these objects are stored together they will be lost or unavailable together.