How to write a file or data to an S3 object using boto3

Question

In boto 2  you can write to an S3 object using these methods    Key set contents from string   Key set contents from file   Key set contents from filename   Key set contents from stream     Is there a boto 3 equivalent  What is the boto3 method for saving data to an object stored on S3

User · Answer

boto3 also has a method for uploading a file directly:

s3 = boto3.resource('s3')    
s3.Bucket('bucketname').upload_file('/local/file/here.txt','folder/sub/path/to/s3key')

http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.upload_file

User · Answer

A cleaner and concise version which I use to upload files on the fly to a given S3 bucket and sub-folder-  import boto3  BUCKET NAME    sample bucket name  PREFIX    sub-folder    s3   boto3 resource  s3      Creating an empty file called   DONE  and putting it in the S3 bucket s3 Object BUCKET NAME  PREFIX     DONE   put Body       Note  You should ALWAYS put your AWS credentials  aws access key id and aws secret access key  in a separate file  for example-    aws credentials

User · Answer

Here s a nice trick to read JSON from s3   import json  boto3 s3   boto3 resource  s3   Bucket  bucket   json load s3   lambda f  json load s3 Object key f  get    Body    json dump s3   lambda obj  f  s3 Object key f  put Body json dumps obj     Now you can use json load s3 and json dump s3 with the same API as load and dump  data     test  0  json dump s3 data   key     saves json to s3   bucket key data   json load s3  key     read json from s3   bucket key

User · Answer

In boto 3  the  Key set contents from   methods were replaced by  Object put    Client put object     For example  import boto3  some binary data   b Here we have some data  more binary data   b Here we have some more data     Method 1  Object put   s3   boto3 resource  s3   object   s3 Object  my bucket name    my key including filename txt   object put Body some binary data     Method 2  Client put object   client   boto3 client  s3   client put object Body more binary data  Bucket  my bucket name   Key  my key including anotherfilename txt    Alternatively  the binary data can come from reading a file  as described in the official docs comparing boto 2 and boto 3   Storing Data Storing data from a file  stream  or string is easy    Boto 2 x from boto s3 key import Key key   Key  hello txt   key set contents from file   tmp hello txt      Boto 3 s3 Object  mybucket    hello txt   put Body open   tmp hello txt    rb

User · Answer

it is worth mentioning smart-open that uses boto3 as a back-end   smart-open is a drop-in replacement for python s open that can open files from s3  as well as ftp  http and many other protocols   for example  from smart open import open import json with open  s3   your bucket your key json    r   as f      data   json load f    The aws credentials are loaded via boto3 credentials  usually a file in the    aws  dir or an environment variable

User · Answer

You may use the below code to write  for example an image to S3 in 2019  To be able to connect to S3 you will have to install AWS CLI using command pip install awscli  then enter few credentials using command aws configure   import urllib3 import uuid from pathlib import Path from io import BytesIO from errors import custom exceptions as cex  BUCKET NAME    xxx yyy zzz  POSTERS BASE PATH    assets wallcontent  CLOUDFRONT BASE URL    https   xxx cloudfront net     class S3 object       def   init   self           self client   boto3 client  s3           self bucket name   BUCKET NAME         self posters base path   POSTERS BASE PATH      def   download image self  url           manager   urllib3 PoolManager           try              res   manager request  GET   url          except Exception              print  Could not download the image from URL     url              raise cex ImageDownloadFailed         return BytesIO res data     any file-like object that implements read        def upload image self  url           try              image file   self   download image url          except cex ImageDownloadFailed              raise cex ImageUploadFailed          extension   Path url  suffix         id   uuid uuid1   hex   extension         final path   self posters base path         id         try              self client upload fileobj image file                                         self bucket name                                         final path                                                  except Exception              print  Image Upload Error for URL     url              raise cex ImageUploadFailed          return CLOUDFRONT BASE URL   id

User · Answer

You no longer have to convert the contents to binary before writing to the file in S3  The following example creates a new text file  called newfile txt  in an S3 bucket with string contents   import boto3  s3   boto3 resource       s3       region name  us-east-1       aws access key id KEY ID      aws secret access key ACCESS KEY   content  String content to write to a new S3 file  s3 Object  my-bucket-name    newfile txt   put Body content

[python] How to write a file or data to an S3 object using boto3

Examples related to python

Examples related to amazon-web-services

Examples related to amazon-s3

Examples related to boto

Examples related to boto3