How to save S3 object to a file using boto3

Question

I m trying to do a  hello world  with new boto3 client for AWS   The use-case I have is fairly simple  get object from S3 and save it to the file   In boto 2 X I would do it like this   import boto key   boto connect s3   get bucket  foo   get key  foo   key get contents to filename   tmp foo     In boto 3   I can t find a clean way to do the same thing  so I m manually iterating over the  Streaming  object   import boto3 key   boto3 resource  s3   Object  fooo    docker my-image tar gz   get   with open   tmp my-image tar gz    w   as f      chunk   key  Body   read 1024 8      while chunk          f write chunk          chunk   key  Body   read 1024 8    or  import boto3 key   boto3 resource  s3   Object  fooo    docker my-image tar gz   get   with open   tmp my-image tar gz    w   as f      for chunk in iter lambda  key  Body   read 4096   b             f write chunk    And it works fine  I was wondering is there any  native  boto3 function that will do the same task

User · Answer

Note  I m assuming you have configured authentication separately  Below code is to download the single object from the S3 bucket          import boto3   initiate s3 client  s3   boto3 resource  s3     Download object to the file     s3 Bucket  mybucket   download file  hello txt     tmp hello txt

User · Answer

For those of you who would like to simulate the set contents from string like boto2 methods  you can try  import boto3 from cStringIO import StringIO  s3c   boto3 client  s3   contents    My string to save to S3 object  target bucket    hello-world by vor  target file    data hello txt  fake handle   StringIO contents     notice if you do fake handle read   it reads like a file handle s3c put object Bucket target bucket  Key target file  Body fake handle read        For Python3   In python3 both StringIO and cStringIO are gone  Use the StringIO import like   from io import StringIO   To support both version   try     from StringIO import StringIO except ImportError     from io import StringIO

User · Answer

If you wish to download a version of a file  you need to use get object  import boto3  bucket    bucketName  prefix    path to file   filename    fileName ext   s3c   boto3 client  s3   s3r   boto3 resource  s3    if   name         main         for version in s3r Bucket bucket  object versions filter Prefix prefix   filename           file   version get           version id   file get  VersionId           obj   s3c get object              Bucket bucket              Key prefix   filename              VersionId version id                    with open f quot  filename   version id  quot    wb   as f              for chunk in obj  Body   iter chunks chunk size 4096                   f write chunk   Ref  https   botocore amazonaws com v1 documentation api latest reference response html

User · Answer

There is a customization that went into Boto3 recently which helps with this  among other things   It is currently exposed on the low-level S3 client  and can be used like this   s3 client   boto3 client  s3   open  hello txt   write  Hello  world       Upload the file to S3 s3 client upload file  hello txt    MyBucket    hello-remote txt      Download the file from S3 s3 client download file  MyBucket    hello-remote txt    hello2 txt   print open  hello2 txt   read      These functions will automatically handle reading writing files as well as doing multipart uploads in parallel for large files   Note that s3 client download file won t create a directory  It can be created as pathlib Path   path to file txt   parent mkdir parents True  exist ok True

User · Answer

When you want to read a file with a different configuration than the default one  feel free to use either mpu aws s3 download s3path  destination  directly or the copy-pasted code   def s3 download source  destination                  exists strategy  raise                   profile name None               Copy a file from an S3 source to a local destination       Parameters     ----------     source   str         Path starting with s3     e g   s3   bucket-name key foo bar      destination   str     exists strategy     raise    replace    abort           What is done when the destination already exists      profile name   str  optional         AWS profile      Raises     ------     botocore exceptions NoCredentialsError         Botocore is not able to find your credentials  Either specify         profile name or add the environment variables AWS ACCESS KEY ID          AWS SECRET ACCESS KEY and AWS SESSION TOKEN          See https   boto3 readthedocs io en latest guide configuration html             exists strategies     raise    replace    abort       if exists strategy not in exists strategies          raise ValueError  exists strategy        is not in                               format exists strategy  exists strategies       session   boto3 Session profile name profile name      s3   session resource  s3       bucket name  key    s3 path split source      if os path isfile destination           if exists strategy is  raise               raise RuntimeError  File        already exists                                   format destination           elif exists strategy is  abort               return     s3 Bucket bucket name  download file key  destination   from collections import namedtuple  S3Path   namedtuple  S3Path     bucket name    key      def  s3 path split s3 path               Split an S3 path into bucket and key       Parameters     ----------     s3 path   str      Returns     -------     splitted    str  str           bucket  key       Examples     --------      gt  gt  gt   s3 path split  s3   my-bucket foo bar jpg       S3Path bucket name  my-bucket   key  foo bar jpg               if not s3 path startswith  s3               raise ValueError               s3 path is expected to start with  s3         but was                  format s3 path                bucket key   s3 path len  s3            bucket name  key   bucket key split      1      return S3Path bucket name  key

User · Answer

Preface  File is json with contents    name    Android    status    ERROR    import boto3 import io  s3   boto3 resource  s3    obj   s3 Object  my-bucket    key-to-file json   data   io BytesIO   obj download fileobj data     object is now a bytes string  Converting it to a dict  new dict   json loads data getvalue   decode  utf-8     print new dict  status       Should print  Error

User · Answer

boto3 now has a nicer interface than the client   resource   boto3 resource  s3   my bucket   resource Bucket  MyBucket   my bucket download file key  local filename    This by itself isn t tremendously better than the client in the accepted answer  although the docs say that it does a better job retrying uploads and downloads on failure  but considering that resources are generally more ergonomic  for example  the s3 bucket and object resources are nicer than the client methods  this does allow you to stay at the resource layer without having to drop down   Resources generally can be created in the same way as clients  and they take all or most of the same arguments and just forward them to their internal clients

[python] How to save S3 object to a file using boto3

Examples related to python

Examples related to amazon-web-services

Examples related to boto

Examples related to boto3