How to delete files recursively from an S3 bucket

Question

I have the following folder structure in S3  Is there a way to recursively remove all files under a certain folder  say foo bar1 or foo or foo bar2 1      foo bar1 1    foo bar1 2    foo bar1 3     foo bar2 1    foo bar2 2    foo bar2 3

User · Answer

You might also consider using Amazon S3 Lifecycle to create an expiration for files with the prefix foo/bar1.

Open the S3 browser console and click a bucket. Then click Properties and then LifeCycle.

Create an expiration rule for all files with the prefix foo/bar1 and set the date to 1 day since file was created.

Save and all matching files will be gone within 24 hours.

Just don't forget to remove the rule after you're done!

No API calls, no third party libraries, apps or scripts.

I just deleted several million files this way.

A screenshot showing the Lifecycle Rule window (note in this shot the Prefix has been left blank, affecting all keys in the bucket):

enter image description here

User · Answer

In the S3 management console  click on the checkmark for the bucket  and click on the empty button from the top right

User · Answer

In case if you want to remove all objects with  foo   prefix using Java AWS SDK 2 0  import java util ArrayList  import java util Iterator  import software amazon awssdk services s3 S3Client  import software amazon awssdk services s3 model            ListObjectsRequest listObjectsRequest   ListObjectsRequest builder        bucket bucketName       prefix  foo         build     ListObjectsResponse objectsResponse   s3Client listObjects listObjectsRequest    while  true        ArrayList lt ObjectIdentifier gt  objects   new ArrayList lt  gt          for  Iterator lt   gt  iterator   objectsResponse contents   iterator    iterator hasNext                S3Object s3Object    S3Object iterator next            objects add              ObjectIdentifier builder                    key s3Object key                     build                         s3Client deleteObjects          DeleteObjectsRequest builder                bucket bucketName               delete                  Delete builder                        objects objects                       build                              build               if  objectsResponse isTruncated              objectsResponse   s3Client listObjects listObjectsRequest           continue             break

User · Answer

With the latest aws-cli python command line tools  to recursively delete all the files under a folder in a bucket is just   aws s3 rm --recursive s3   your bucket name foo    Or delete everything under the bucket   aws s3 rm --recursive s3   your bucket name   If what you want is to actually delete the bucket  there is one-step shortcut   aws s3 rb --force s3   your bucket name   which will remove the contents in that bucket recursively then delete the bucket   Note  the s3    protocol prefix is required for these commands to work

User · Answer

The voted up answer is missing a step    Per aws s3 help      Currently  there is no support for the use of UNIX style wildcards in a          command s  path  arguments    However   most  commands  have  --exclude   lt value gt   and --include   lt value gt    parameters  that  can  achieve  the          desired  result          When there are multiple          filters  the rule is the filters that appear later in the command  take          precedence  over filters that appear earlier in the command   For example  if the filter parameters passed to the command were --exclude     --include    txt  All files will be excluded from the command  except  for  files  ending          with   txt   aws s3 rm --recursive s3   bucket  --exclude     --include   folder path

User · Answer

In case using AWS-SKD for ruby V2   s3 list objects bucket  bucket name  prefix   foo    contents each do  obj    next if obj key     foo      resp   s3 delete object       bucket  bucket name      key  obj key       end   attention please  all  foo    under bucket will delete

User · Answer

This used to require a dedicated API call per key  file   but has been greatly simplified due to the introduction of Amazon S3 - Multi-Object Delete in December 2011      Amazon S3 s new Multi-Object Delete gives you the ability to   delete up to 1000 objects from an S3 bucket with a single request    See my answer to the related question delete from S3 using api php using wildcard for more on this and respective examples in PHP  the AWS SDK for PHP supports this since version 1 4 8    Most AWS client libraries have meanwhile introduced dedicated support for this functionality one way or another  e g    Python  You can achieve this with the excellent boto Python interface to AWS roughly as follows  untested  from the top of my head    import boto s3   boto connect s3   bucket   s3 get bucket  bucketname   bucketListResultSet   bucket list prefix  foo bar   result   bucket delete keys  key name for key in bucketListResultSet     Ruby  This is available since version 1 24 of the AWS SDK for Ruby and the release notes provide an example  as well   bucket   AWS  S3 new buckets  mybucket      delete a list of objects by keys  objects are deleted in batches of 1k per   request   Accepts strings  AWS  S3  S3Object  AWS  S3  ObectVersion and    hashes with  key and  version id bucket objects delete  key1    key2    key3           delete all of the objects in a bucket  optionally with a common prefix as shown  bucket objects with prefix  2009    delete all    conditional delete  loads and deletes objects in batches of 1k  only   deleting those that return true from the block bucket objects delete if  object  object key       pdf        empty the bucket and then delete the bucket  objects are deleted in batches of 1k bucket delete    Or   AWS  S3  Bucket delete  your bucket    force   gt  true

User · Answer

Best way is to use lifecycle rule to delete whole bucket contents  Programmatically you can use following code  PHP  to PUT lifecycle rule    expiration   array  Date    gt  date  U   strtotime  GMT midnight       result    s3- gt putBucketLifecycle array               Bucket    gt   bucket-name                Rules    gt  array                  array                       Expiration    gt   expiration                       ID    gt   rule-name                        Prefix    gt                           Status    gt   Enabled                                                   In above case all the objects will be deleted starting Date -  Today GMT midnight     You can also specify Days as follows  But with Days it will wait for at least 24 hrs  1 day is minimum  to start deleting the bucket contents    expiration   array  Days    gt  1

User · Answer

I just removed all files from my bucket by using PowerShell   Get-S3Object -BucketName YOUR BUCKET       Remove-S3Object -BucketName YOUR BUCKET -Key    Key -Force  true

User · Answer

I needed to do the following     def delete bucket   s3   init amazon s3   s3 buckets  BUCKET-NAME   objects each do  obj      obj delete   end end  def init amazon s3   config   YAML load file    Rails root  config s3 yml     AWS config  access key id   gt  config  access key id    secret access key   gt  config  secret access key      s3   AWS  S3 new end

User · Answer

Just saw that Amazon added a  How to Empty a Bucket  option to the AWS console menu   http   docs aws amazon com AmazonS3 latest UG DeletingaBucket html

User · Answer

With s3cmd package installed on a Linux machine  you can do this   s3cmd rm s3   foo bar --recursive

[amazon-s3] How to delete files recursively from an S3 bucket

Examples related to amazon-s3

Examples related to amazon-web-services