Listing files in a specific folder of a AWS S3 bucket

Question

I need to list all files contained in a certain folder contained in my S3 bucket   The folder structure is the following   my-bucket users  lt user-id gt  contacts  lt contact-id gt    I have files related to users and files related to a certain user s contact  I need to list both   To list files I m using this code   ListObjectsRequest listObjectsRequest   new ListObjectsRequest   withBucketName  my-bucket                    withPrefix  some-prefix   withDelimiter       ObjectListing objects   transferManager getAmazonS3Client   listObjects listObjectsRequest     To list a certain user s files I m using this prefix    users  lt user-id gt     and I m correctly getting all files in the directory excluding contacts subdirectory  for example   users  lt user-id gt  file1 txt users  lt user-id gt  file2 txt users  lt user-id gt  file3 txt   To list a certain user contact s files instead I m using this prefix    users  lt user-id gt  contacts  lt contact-id gt    but in this case I m getting also the  directory itself as a returned object   users  lt user-id gt  contacts  lt contact-id gt  file1 txt users  lt user-id gt  contacts  lt contact-id gt  file2 txt users  lt user-id gt  contacts  lt contact-id gt     Why am I getting this behaviour  What s different beetween the two listing requests  I need to list only files in the directory  excluding sub-directories

User · Answer

S3 does not have directories  while you can list files in a pseudo directory manner like you demonstrated  there is no directory  file  per-se  You may of inadvertently created a data file called users  lt user-id gt  contacts  lt contact-id gt

User · Answer

Everything in S3 is an object  To you  it may be files and folders  But to S3  they re just objects   Objects that end with the delimiter    in most cases  are usually perceived as a folder  but it s not always the case  It depends on the application  Again  in your case  you re interpretting it as a folder  S3 is not  It s just another object   In your case above  the object users  lt user-id gt  contacts  lt contact-id gt   exists in S3 as a distinct object  but the object users  lt user-id gt   does not  That s the difference in your responses  Why they re like that  we cannot tell you  but someone made the object in one case  and didn t in the other  You don t see it in the AWS Management Console because the console is interpreting it as a folder and hiding it from you   Since S3 just sees these things as objects  it won t  exclude  certain things for you  It s up to the client to deal with the objects as they should be dealt with   Your Solution  Since you re the one that doesn t want the folder objects  you can exclude it yourself by checking the last character for a    If it is  then ignore the object from the response

User · Answer

If your goal is only to take the files and not the folder  the approach I made was to use the file size as a filter  This property is the current size of the file hosted by AWS  All the folders return 0 in that property  The following is a C  code using linq but it shouldn t be hard to translate to Java  var amazonClient   new AmazonS3Client key  secretKey  region   var listObjectsRequest  new ListObjectsRequest                               BucketName    someBucketName                   Delimiter    someDelimiter                   Prefix    somePrefix                 var objects   amazonClient ListObjects listObjectsRequest   var objectsInFolder   objects S3Objects Where file   gt  file Size  gt  0  ToList

User · Answer

As other have already said  everything in S3 is an object  To you  it may be files and folders  But to S3  they re just objects   If you don t need objects which end with a     you can safely delete them e g  via REST api or AWS Java SDK  I assume you have write access   You will not lose  nested files   there no files  so you will not lose objects whose names are prefixed with the key you delete   AmazonS3 amazonS3   AmazonS3ClientBuilder standard   withCredentials new ProfileCredentialsProvider    withRegion  region   build    amazonS3 deleteObject new DeleteObjectRequest  my-bucket    users  lt user-id gt  contacts  lt contact-id gt         Please note that I m using ProfileCredentialsProvider so that my requests are not anonymous  Otherwise  you will not be able to delete an object  I have my AWS keep key stored in    aws credentials file

User · Answer

While everybody say that there are no directories and files in s3  but only objects  and buckets   which is absolutely true  I would suggest to take advantage of CommonPrefixes  described in this answer  So  you can do following to get list of  quot folders quot   commonPrefixes  and  quot files quot   objectSummaries   ListObjectsV2Request req   new ListObjectsV2Request   withBucketName bucket getName    withPrefix prefix  withDelimiter DELIMITER   ListObjectsV2Result listing   s3Client listObjectsV2 req   for  String commonPrefix   listing getCommonPrefixes              System out println commonPrefix     for  S3ObjectSummary summary  listing getObjectSummaries          System out println summary getKey        In your case  for objectSummaries  files  it should return  in case of correct prefix    users user-id contacts contact-id file1 txt users user-id contacts contact-id file2 txt for commonPrefixes  users user-id contacts contact-id  Reference  https   docs aws amazon com AmazonS3 latest API API ListObjectsV2 html

User · Answer

Based on  davioooh answer  This code is worked for me   ListObjectsRequest listObjectsRequest   new ListObjectsRequest   withBucketName  your-bucket                withPrefix  your folder path    withDelimiter

User · Answer

you can check the type  s3 has a special application x-directory  bucket objects   delimiter  gt       prefix  gt  f1     each    obj  p obj object content type

[java] Listing files in a specific "folder" of a AWS S3 bucket

Examples related to java

Examples related to amazon-web-services

Examples related to amazon-s3