Read file content from S3 bucket with boto3

Question

I read the filenames in my S3 bucket by doing   objs   boto3 client list objects Bucket  my bucket       while  Contents  in objs keys            objs contents   objs  Contents           for i in range len objs contents                filename   objs contents i   Key     Now  I need to get the actual content of the file  similarly to a open filename  readlines    What is the best way

User · Accepted Answer

boto3 offers a resource model that makes tasks like iterating through objects easier  Unfortunately  StreamingBody doesn t provide readline or readlines    s3   boto3 resource  s3   bucket   s3 Bucket  test-bucket     Iterates through all the objects  doing the pagination for you  Each obj   is an ObjectSummary  so it doesn t contain the body  You ll need to call   get to get the whole body  for obj in bucket objects all        key   obj key     body   obj get    Body   read

User · Answer

Using the client instead of resource  s3   boto3 client  s3   bucket  bucket name  result   s3 list objects Bucket   bucket  Prefix   something    for o in result get  Contents        data   s3 get object Bucket bucket  Key o get  Key        contents   data  Body   read       print contents

User · Answer

You might also consider the smart open module  which supports iterators   from smart open import smart open    stream lines from an S3 object for line in smart open  s3   mybucket mykey txt    rb        print line decode  utf8      and context managers   with smart open  s3   mybucket mykey txt    rb   as s3 source      for line in s3 source           print line decode  utf8         s3 source seek 0     seek to the beginning     b1000   s3 source read 1000     read 1000 bytes   Find smart open at https   pypi org project smart open

User · Answer

If you already know the filename  you can use the boto3 builtin download fileobj  import boto3  from io import BytesIO  session   boto3 Session   s3 client   session client  s3    f   BytesIO   s3 client download fileobj bucket name  filename  f  f seek 0  print f getvalue

User · Answer

When you want to read a file with a different configuration than the default one  feel free to use either mpu aws s3 read s3path  directly or the copy-pasted code   def s3 read source  profile name None               Read a file from an S3 source       Parameters     ----------     source   str         Path starting with s3     e g   s3   bucket-name key foo bar      profile name   str  optional         AWS profile      Returns     -------     content   bytes      botocore exceptions NoCredentialsError         Botocore is not able to find your credentials  Either specify         profile name or add the environment variables AWS ACCESS KEY ID          AWS SECRET ACCESS KEY and AWS SESSION TOKEN          See https   boto3 readthedocs io en latest guide configuration html             session   boto3 Session profile name profile name      s3   session client  s3       bucket name  key   mpu aws  s3 path split source      s3 object   s3 get object Bucket bucket name  Key key      body   s3 object  Body       return body read

[python] Read file content from S3 bucket with boto3

Examples related to python

Examples related to amazon-web-services

Examples related to amazon-s3

Examples related to boto3