[amazon-web-services] How do you search an amazon s3 bucket?

I have a bucket with thousands of files in it. How can I search the bucket? Is there a tool you can recommend?

There are multiple options, none being simple "one shot" full text solution:

1. Key name pattern search: Searching for keys starting with some string- if you design key names carefully, then you may have rather quick solution.

2. Search metadata attached to keys: when posting a file to AWS S3, you may process the content, extract some meta information and attach this meta information in form of custom headers into the key. This allows you to fetch key names and headers without need to fetch complete content. The search has to be done sequentialy, there is no "sql like" search option for this. With large files this could save a lot of network traffic and time.

3. Store metadata on SimpleDB: as previous point, but with storing the metadata on SimpleDB. Here you have sql like select statements. In case of large data sets you may hit SimpleDB limits, which can be overcome (partition metadata across multiple SimpleDB domains), but if you go really far, you may need to use another metedata type of database.

4. Sequential full text search of the content - processing all the keys one by one. Very slow, if you have too many keys to process.

We are storing 1440 versions of a file a day (one per minute) for couple of years, using versioned bucket, it is easily possible. But getting some older version takes time, as one has to sequentially go version by version. Sometime I use simple CSV index with records, showing publication time plus version id, having this, I could jump to older version rather quickly.

As you see, AWS S3 is not on it's own designed for full text searches, it is simple storage service.

Take a look at this documentation: http://docs.aws.amazon.com/AWSSDKforPHP/latest/index.html#m=amazons3/get_object_list

You can use a Perl-Compatible Regular Expression (PCRE) to filter the names.

Here's a short and ugly way to do search file names using the AWS CLI:

aws s3 ls s3://your-bucket --recursive | grep your-search | cut -c 32-

I did something as below to find patterns in my bucket

def getListOfPrefixesFromS3(dataPath: String, prefix: String, delimiter: String, batchSize: Integer): List[String] = {
    var s3Client = new AmazonS3Client()
    var listObjectsRequest = new ListObjectsRequest().withBucketName(dataPath).withMaxKeys(batchSize).withPrefix(prefix).withDelimiter(delimiter)
    var objectListing: ObjectListing = null
    var res: List[String] = List()

    do {
      objectListing = s3Client.listObjects(listObjectsRequest)
      res = res ++ objectListing.getCommonPrefixes
      listObjectsRequest.setMarker(objectListing.getNextMarker)
    } while (objectListing.isTruncated)
    res
  }

For larger buckets this consumes too much of time since all the object summaries are returned by the Aws and not only the ones that match the prefix and the delimiter. I am looking for ways to improve the performance and so far i've only found that i should name the keys and organise them in buckets properly.

If you're on Windows and have no time finding a nice grep alternative, a quick and dirty way would be:

aws s3 ls s3://your-bucket/folder/ --recursive > myfile.txt

and then do a quick-search in myfile.txt

The "folder" bit is optional.

P.S. if you don't have AWS CLI installed - here's a one liner using Chocolatey package manager

choco install awscli

P.P.S. If you don't have the Chocolatey package manager - get it! Your life on Windows will get 10x better. (I'm not affiliated with Chocolatey in any way, but hey, it's a must-have, really).

Search by Prefix in S3 Console

directly in the AWS Console bucket view.

Copy wanted files using s3-dist-cp

When you have thousands or millions of files another way to get the wanted files is to copy them to another location using distributed copy. You run this on EMR in a Hadoop Job. The cool thing about AWS is that they provide their custom S3 version s3-dist-cp. It allows you to group wanted files using a regular expression in the groupBy field. You can use this for example in a custom step on EMR

[
    {
        "ActionOnFailure": "CONTINUE",
        "Args": [
            "s3-dist-cp",
            "--s3Endpoint=s3.amazonaws.com",
            "--src=s3://mybucket/",
            "--dest=s3://mytarget-bucket/",
            "--groupBy=MY_PATTERN",
            "--targetSize=1000"
        ],
        "Jar": "command-runner.jar",
        "Name": "S3DistCp Step Aggregate Results",
        "Type": "CUSTOM_JAR"
    }
]

I faced the same problem. Searching in S3 should be much more easier than the current situation. That's why, I implemented this open source tool for searching in S3.

SSEARCH is full open source S3 search tool. It has been implemented always keeping mind that the performance is the critical factor and according to the benchmarks it searches the bucket which contains ~1000 files within seconds.

Installation is simple. You only download docker-compose file and running it with

docker-compose up

SSEARCH will be started and you can search anything in any bucket you have.

Given that you are in AWS...I would think you would want to use their CloudSearch tools. Put the data you want to search in their service...have it point to the S3 keys.

http://aws.amazon.com/cloudsearch/

I tried in the following way

aws s3 ls s3://Bucket1/folder1/2019/ --recursive |grep filename.csv

This outputs the actual path where the file exists

2019-04-05 01:18:35     111111 folder1/2019/03/20/filename.csv

The way I did it is: I have thousands of files in s3. I saw the properties panel of one file in the list. You can see the URI of that file and I copy pasted that to the browser - it was a text file and it rendered nicely. Now I replaced the uuid in the url with the uuid that I had at hand and boom there the file is.

I wish AWS had a better way to search a file, but this worked for me.

Another option is to mirror the S3 bucket on your web server and traverse locally. The trick is that the local files are empty and only used as a skeleton. Alternatively, the local files could hold useful meta data that you normally would need to get from S3 (e.g. filesize, mimetype, author, timestamp, uuid). When you provide a URL to download the file, search locally and but provide a link to the S3 address.

Local file traversing is easy and this approach for S3 management is language agnostic. Local file traversing also avoids maintaining and querying a database of files or delays making a series of remote API calls to authenticate and get the bucket contents.

You could allow users to upload files directly to your server via FTP or HTTP and then transfer a batch of new and updated files to Amazon at off peak times by just recursing over the directories for files with any size. On the completion of a file transfer to Amazon, replace the web server file with an empty one of the same name. If a local file has any filesize then serve it directly because its awaiting batch transfer.

There are (at least) two different use cases which could be described as "search the bucket":

1. Search for something inside every object stored at the bucket; this assumes a common format for all the objects in that bucket (say, text files), etc etc. For something like this, you're forced to do what Cody Caughlan just answered. The AWS S3 docs has example code showing how to do this with the AWS SDK for Java: Listing Keys Using the AWS SDK for Java (there you'll also find PHP and C# examples).

2. List item Search for something in the object keys contained in that bucket; S3 does have partial support for this, in the form of allowing prefix exact matches + collapsing matches after a delimiter. This is explained in more detail at the AWS S3 Developer Guide. This allows, for example, to implement "folders" through using as object keys something like

   folder/subfolder/file.txt

   If you follow this convention, most of the S3 GUIs (such as the AWS Console) will show you a folder view of your bucket.

AWS released a new Service to query S3 buckets with SQL: Amazon Athena https://aws.amazon.com/athena/

Fast forward to 2020, and using aws-okta as our 2fa, the following command, while slow as hell to iterate through all of the objects and folders in this particular bucket (+270,000) worked fine.

aws-okta exec dev -- aws s3 ls my-cool-bucket --recursive | grep needle-in-haystax.txt

Use Amazon Athena to query S3 bucket. Also, load data to Amazon Elastic search. Hope this helps.

This is little bit old thread - but maybe help someone who still search - I'm the one who search for that a year.

Solution may be "AWS Athena" where you can search over data like this

'SELECT user_name FROM S3Object WHERE cast(age as int) > 20'

https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/

Currently pricing is $5 for 1TB data - so for example, if your query search over one 1TB file 3times your cost is $15 - but for example if there is only 1column in "converted columnar format" what you want read, you'll pay 1/3 of price means $1.67/TB.

Not a technical answer, but I have built an application which allows for wildcard search: https://bucketsearch.net/

It will asynchronously index your bucket and then allow you to search the results.

It's free to use (donationware).

Just a note to add on here: it's now 3 years later, yet this post is top in Google when you type in "How to search an S3 Bucket."

Perhaps you're looking for something more complex, but if you landed here trying to figure out how to simply find an object (file) by it's title, it's crazy simple:

open the bucket, select "none" on the right hand side, and start typing in the file name.

http://docs.aws.amazon.com/AmazonS3/latest/UG/ListingObjectsinaBucket.html

Status 2018-07: Amazon do have native sql like search for csv and json files!

https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/

Try this command:

aws s3api list-objects --bucket your-bucket --prefix sub-dir-path --output text --query 'Contents[].{Key: Key}'

Then you can pipe this into a grep to get specific file types to do whatever you want with them.