check if a key exists in a bucket in s3 using boto3

Question

I would like to know if a key exists in boto3  I can loop the bucket contents and check the key if it matches   But that seems longer and an overkill  Boto3 official docs explicitly state how to do this    May be I am missing the obvious  Can anybody point me how I can achieve this

User · Answer

I noticed that just for catching the exception using botocore.exceptions.ClientError we need to install botocore. botocore takes up 36M of disk space. This is particularly impacting if we use aws lambda functions. In place of that if we just use exception then we can skip using the extra library!

I am validating for the file extension to be '.csv'
This will not throw an exception if the bucket does not exist!
This will not throw an exception if the bucket exists but object does not exist!
This throws out an exception if the bucket is empty!
This throws out an exception if the bucket has no permissions!

The code looks like this. Please share your thoughts:

import boto3
import traceback

def download4mS3(s3bucket, s3Path, localPath):
    s3 = boto3.resource('s3')

    print('Looking for the csv data file ending with .csv in bucket: ' + s3bucket + ' path: ' + s3Path)
    if s3Path.endswith('.csv') and s3Path != '':
        try:
            s3.Bucket(s3bucket).download_file(s3Path, localPath)
        except Exception as e:
            print(e)
            print(traceback.format_exc())
            if e.response['Error']['Code'] == "404":
                print("Downloading the file from: [", s3Path, "] failed")
                exit(12)
            else:
                raise
        print("Downloading the file from: [", s3Path, "] succeeded")
    else:
        print("csv file not found in in : [", s3Path, "]")
        exit(12)

User · Answer

If you have less than 1000 in a directory or bucket you can get set of them and after check if such key in this set   files in dir    d  Key   split      -1  for d in s3 client list objects v2  Bucket  mybucket   Prefix  my dir   get  Contents   or       Such code works even if my dir is not exists   http   boto3 readthedocs io en latest reference services s3 html S3 Client list objects v2

User · Answer

Here is a solution that works for me  One caveat is that I know the exact format of the key ahead of time  so I am only listing the single file  import boto3    The s3 base class to interact with S3 class S3 object     def   init   self       self s3 client   boto3 client  s3      def check if object exists self  s3 bucket  s3 key       response   self s3 client list objects        Bucket   s3 bucket        Prefix   s3 key             if  ETag  in str response         return True     else        return False  if   name         main       s3    S3     if s3 check if object exists bucket  key       print  Found S3 object     else      print  No object found

User · Answer

S3 REGION  eu-central-1  bucket  mybucket1  name  objectname   import boto3 from botocore client import Config client   boto3 client  s3  region name S3 REGION config Config signature version  s3v4    list   client list objects v2 Bucket bucket Prefix name  for obj in list get  Contents            if obj  Key      name  return True return False

User · Answer

There is one simple way by which we can check if file exists or not in S3 bucket  We donot need to use exception for this  sesssion   boto3 Session aws access key id  aws secret access key  s3   session client  s3    object name    filename  bucket    bucketname  obj status   s3 list objects Bucket   bucket  Prefix   object name  if obj status get  Contents        print  File exists   else      print  File does not exists

User · Answer

FWIW  here are the very simple functions that I am using  import boto3  def get resource config  dict             Loads the s3 resource       Expects AWS ACCESS KEY ID and AWS SECRET ACCESS KEY to be in the environment     or in a config dictionary      Looks in the environment first          s3   boto3 resource  s3                           aws access key id os environ get                               AWS ACCESS KEY ID   config get  AWS ACCESS KEY ID                             aws secret access key os environ get  AWS SECRET ACCESS KEY   config get  AWS SECRET ACCESS KEY         return s3   def get bucket s3  s3 uri  str          Get the bucket from the resource      A thin wrapper  use with caution       Example usage        gt  gt  bucket   get bucket get resource    s3 uri prod         return s3 Bucket s3 uri    def isfile s3 bucket  key  str  - gt  bool         Returns T F whether the file exists         objs   list bucket objects filter Prefix key       return len objs     1 and objs 0  key    key   def isdir s3 bucket  key  str  - gt  bool         Returns T F whether the directory exists         objs   list bucket objects filter Prefix key       return len objs   gt  1

User · Answer

Just following the thread  can someone conclude which one is the most efficient way to check if an object exists in S3    I think head object might win as it just checks the metadata which is lighter than the actual object itself

User · Answer

import boto3 client   boto3 client  s3   s3 key    Your file without bucket name e g  abc bcd txt  bucket    your bucket name  content   client head object Bucket bucket Key s3 key      if content get  ResponseMetadata  None  is not None          print  File exists - s3    s  s     bucket s3 key       else          print  File does not exist - s3    s  s     bucket s3 key

User · Answer

Use this concise oneliner  makes it less intrusive when you have to throw it inside an existing project without modifying much of the code  s3 file exists   lambda filename  bool list bucket objects filter Prefix filename     The above function assumes the bucket variable was already declared  You can extend the lambda to support additional parameter like s3 file exists   lambda filename  bucket  bool list bucket objects filter Prefix filename

User · Answer

This could check both prefix and key  and fetches at most 1 key   def prefix exits bucket  prefix       s3 client   boto3 client  s3       res   s3 client list objects v2 Bucket bucket  Prefix prefix  MaxKeys 1      return  Contents  in res

User · Answer

If you seek a key that is equivalent to a directory then you might want this approach  session   boto3 session Session   resource   session resource  s3   bucket   resource Bucket  mybucket    key    dir-like-or-file-like-key  objects    o for o in bucket objects filter Prefix key  limit 1       has key   len objects   gt  0   This works for a parent key or a key that equates to file or a key that does not exist    I tried the favored approach above and failed on parent keys

User · Answer

Assuming you just want to check if a key exists  instead of quietly over-writing it   do this check first  import boto3  def key exists mykey  mybucket     s3 client   boto3 client  s3     response   s3 client list objects v2 Bucket mybucket  Prefix mykey    if response        for obj in response  Contents              if mykey    obj  Key                  return True   return False  if key exists  someprefix myfile-abc123    my-bucket-name        print  quot key exists quot   else      print  quot safe to put new bucket object quot         try            resp   s3 client put object Body  quot Your string or file-like object quot                                         Bucket mybucket Key mykey           check resp success and ClientError exception for errors

User · Answer

Not only client but bucket too   import boto3 import botocore bucket   boto3 resource  s3   region name  eu-west-1   Bucket  my-bucket    try    bucket Object  my-file   get   except botocore exceptions ClientError as ex    if ex response  Error    Code       NoSuchKey       print  NoSuchKey

User · Answer

It s really simple with get   method      import botocore from boto3 session import Session session   Session aws access key id  AWS ACCESS KEY                   aws secret access key  AWS SECRET ACCESS KEY   s3   session resource  s3   bucket s3   s3 Bucket  bucket name    def not exist file key       try          file details   bucket s3 Object file key  get             print file details    This line prints the file details         return False     except botocore exceptions ClientError as e          if e response  Error    Code       NoSuchKey     or you can check with e reponse  HTTPStatusCode       404              return True         return False   For any other error it s hard to determine whether it exists or not  so based on the requirement feel free to change it to True  False   raise Exception  print not exist  hello world txt

User · Answer

You can use S3Fs  which is essentially a wrapper around boto3 that exposes typical file-system style operations   import s3fs s3   s3fs S3FileSystem   s3 exists  myfile txt

User · Answer

you can use Boto3 for this   import boto3 s3   boto3 resource  s3   bucket   s3 Bucket  my-bucket   objs   list bucket objects filter Prefix key   if len objs  gt 0       print  key exists     else      print  key doesn t exist      Here key is the path you want to check exists or not

User · Answer

I m not a big fan of using exceptions for control flow  This is an alternative approach that works in boto3   import boto3  s3   boto3 resource  s3   bucket   s3 Bucket  my-bucket   key    dootdoot jpg  objs   list bucket objects filter Prefix key   if any  w key    path s3 for w in objs        print  Exists    else      print  Doesn t exist

User · Answer

For boto3  ObjectSummary can be used to check if an object exists      Contains the summary of an object stored in an Amazon S3 bucket  This object doesn t contain contain the object s full metadata or any of its contents   import boto3 from botocore errorfactory import ClientError def path exists path  bucket name          Check to see if an object exists on S3        s3   boto3 resource  s3       try          s3 ObjectSummary bucket name bucket name  key path  load       except ClientError as e          if e response  Error    Code       404               return False         else              raise e     return True  path exists  path to file html     In ObjectSummary load     Calls s3 Client head object to update the attributes of the ObjectSummary resource    This shows that you can use ObjectSummary instead of Object if you are planning on not using get    The load   function does not retrieve the object it only obtains the summary

User · Answer

Check out  bucket get key      key name       headers None       version id None       response headers None       validate True        Check to see if a particular key exists within the bucket  This method   uses a HEAD request to check for the existence of the key  Returns  An   instance of a Key object or None   from Boto S3 Docs  You can just call bucket get key keyname  and check if the returned object is None

User · Answer

Try This simple  import boto3 s3   boto3 resource  s3   bucket   s3 Bucket  mybucket name     just Bucket name file name    A B filename txt         full file path obj   list bucket objects filter Prefix file name   if len obj   gt  0      print  Exists   else      print  Not Exists

User · Answer

The easiest way I found  and probably the most efficient  is this   import boto3 from botocore errorfactory import ClientError  s3   boto3 client  s3   try      s3 head object Bucket  bucket name   Key  file path   except ClientError        Not found     pass

User · Answer

Boto 2 s boto s3 key Key object used to have an exists method that checked if the key existed on S3 by doing a HEAD request and looking at the the result  but it seems that that no longer exists  You have to do it yourself   import boto3 import botocore  s3   boto3 resource  s3    try      s3 Object  my-bucket    dootdoot jpg   load   except botocore exceptions ClientError as e      if e response  Error    Code       404             The object does not exist                  else            Something else has gone wrong          raise else        The object does exist            load   does a HEAD request for a single key  which is fast  even if the object in question is large or you have many objects in your bucket   Of course  you might be checking if the object exists because you re planning on using it  If that is the case  you can just forget about the load   and do a get   or download file   directly  then handle the error case there

User · Answer

In Boto3  if you re checking for either a folder  prefix  or a file using list objects   You can use the existence of  Contents  in the response dict as a check for whether the object exists   It s another way to avoid the try except catches as  EvilPuppetMaster suggests  import boto3 client   boto3 client  s3   results   client list objects Bucket  my-bucket   Prefix  dootdoot jpg   return  Contents  in results

[python] check if a key exists in a bucket in s3 using boto3

Examples related to python

Examples related to amazon-s3

Examples related to boto3