How to upload a file to directory in S3 bucket using boto

Question

I want to copy a file in s3 bucket using python   Ex   I have bucket name   test  And in the bucket  I have 2 folders name  dump   amp   input   Now I want to copy a file from local directory to S3  dump  folder using python    Can anyone help me

User · Answer

This is a three liner  Just follow the instructions on the boto3 documentation   import boto3 s3   boto3 resource service name    s3   s3 meta client upload file Filename    C  foo bar baz filetype   Bucket    yourbucketname   Key    baz filetype     Some important arguments are  Parameters  Filename  str  -- The path to the file to upload  Bucket  str  -- The name of the bucket to upload to  Key  str  -- The name of the that you want to assign to your file in your s3 bucket  This could be the same as the name of the file or a different name of your choice but the filetype should remain the same   Note  I assume that you have saved your credentials in a    aws folder as suggested in the best configuration practices in the boto3 documentation

User · Answer

Using boto3  import logging import boto3 from botocore exceptions import ClientError   def upload file file name  bucket  object name None          Upload a file to an S3 bucket       param file name  File to upload      param bucket  Bucket to upload to      param object name  S3 object name  If not specified then file name is used      return  True if file was uploaded  else False                If S3 object name was not specified  use file name     if object name is None          object name   file name        Upload the file     s3 client   boto3 client  s3       try          response   s3 client upload file file name  bucket  object name      except ClientError as e          logging error e          return False     return True   For more - https   boto3 amazonaws com v1 documentation api latest guide s3-uploading-files html

User · Answer

You should mention the content type as well to omit the file accessing issue  import os image  fly png  s3 filestore path    images fly png  filename  file extension   os path splitext image  content type dict   quot  png quot   quot image png quot   quot  html quot   quot text html quot                   quot  css quot   quot text css quot   quot  js quot   quot application javascript quot                   quot  jpg quot   quot image png quot   quot  gif quot   quot image gif quot                   quot  jpeg quot   quot image jpeg quot    content type content type dict file extension  s3   boto3 client  s3   config boto3 session Config signature version  s3v4                      region name  ap-south-1                     aws access key id S3 KEY                    aws secret access key S3 SECRET  s3 put object Body image  Bucket S3 BUCKET  Key s3 filestore path  ContentType content type

User · Answer

For upload folder example as following code and S3 folder picture  import boto import boto s3 import boto s3 connection import os path import sys        Fill in info on data to upload   destination bucket name bucket name    willie20181121    source directory sourceDir     home willie Desktop x     Linux Path   destination directory name  on s3  destDir     test1      S3 Path   max size in bytes before uploading in parts  between 1 and 5 GB recommended MAX SIZE   20   1000   1000  size of parts when uploading in parts PART SIZE   6   1000   1000  access key    MPBVAQ       IT      secret key    11t63yDV           HgUcgMOSN        conn   boto connect s3          aws access key id   access key          aws secret access key   secret key          host           org tw           is secure False                  uncomment if you are not using ssl         calling format   boto s3 connection OrdinaryCallingFormat              bucket   conn create bucket bucket name          location boto s3 connection Location DEFAULT    uploadFileNames      for  sourceDir  dirname  filename  in os walk sourceDir       uploadFileNames extend filename      break  def percent cb complete  total       sys stdout write          sys stdout flush    for filename in uploadFileNames      sourcepath   os path join sourceDir   filename      destpath   os path join destDir  filename      print   Uploading  s to Amazon S3 bucket  s                  sourcepath  bucket name        filesize   os path getsize sourcepath      if filesize  gt  MAX SIZE          print   quot multipart upload quot           mp   bucket initiate multipart upload destpath          fp   open sourcepath  rb           fp num   0         while  fp tell    lt  filesize               fp num    1             print   quot uploading part  i quot   fp num              mp upload part from file fp  fp num  cb percent cb  num cb 10  size PART SIZE           mp complete upload        else          print   quot singlepart upload quot           k   boto s3 key Key bucket          k key   destpath         k set contents from filename sourcepath                  cb percent cb  num cb 10   PS  For more reference URL

User · Answer

No need to make it that complicated   s3 connection   boto connect s3   bucket   s3 connection get bucket  your bucket name   key   boto s3 key Key bucket   some file zip   with open  some file zip   as f      key send file f

User · Answer

Upload file to s3 within a session with credentials   import boto3  session   boto3 Session      aws access key id  AWS ACCESS KEY ID       aws secret access key  AWS SECRET ACCESS KEY     s3   session resource  s3     Filename - File to upload   Bucket - Bucket to upload to  the top level directory under AWS S3    Key - S3 object name  can contain subdirectories   If not specified then file name is used s3 meta client upload file Filename  input file path   Bucket  bucket name   Key  s3 output key

User · Answer

If you have the aws command line interface installed on your system you can make use of pythons subprocess library  For example  import subprocess def copy file to s3 source  str  target  str  bucket  str      subprocess run   quot aws quot    quot s3 quot     quot cp quot   source  f quot s3    bucket   target  quot     Similarly you can use that logics for all sort of AWS client operations like downloading or listing files etc  It is also possible to get return values  This way there is no need to import boto3  I guess its use is not intended that way but in practice I find it quite convenient that way  This way you also get the status of the upload displayed in your console - for example  Completed 3 5 GiB 3 5 GiB  242 8 MiB s  with 1 file s  remaining  To modify the method to your wishes I recommend having a look into the subprocess reference as well as to the AWS Cli reference  Note  This is a copy of my answer to a similar question

User · Answer

import boto from boto s3 key import Key  AWS ACCESS KEY ID      AWS SECRET ACCESS KEY      END POINT                                 eg  us-east-1 S3 HOST                                   eg  s3 us-east-1 amazonaws com BUCKET NAME    test          FILENAME    upload txt                  UPLOADED FILENAME    dumps upload txt    include folders in file path  If it doesn t exist  it will be created  s3   boto s3 connect to region END POINT                             aws access key id AWS ACCESS KEY ID                             aws secret access key AWS SECRET ACCESS KEY                             host S3 HOST   bucket   s3 get bucket BUCKET NAME  k   Key bucket  k key   UPLOADED FILENAME k set contents from filename FILENAME

User · Answer

Try this     import boto import boto s3 import sys from boto s3 key import Key  AWS ACCESS KEY ID      AWS SECRET ACCESS KEY       bucket name   AWS ACCESS KEY ID lower      -dump  conn   boto connect s3 AWS ACCESS KEY ID          AWS SECRET ACCESS KEY    bucket   conn create bucket bucket name      location boto s3 connection Location DEFAULT   testfile    replace this with an actual filename  print  Uploading  s to Amazon S3 bucket  s          testfile  bucket name   def percent cb complete  total       sys stdout write          sys stdout flush     k   Key bucket  k key    my test file  k set contents from filename testfile      cb percent cb  num cb 10     UPDATE  I am not a pythonist  so thanks for the heads up about the import statements  Also  I d not recommend placing credentials inside your own source code  If you are running this inside AWS use IAM Credentials with Instance Profiles  http   docs aws amazon com IAM latest UserGuide id roles use switch-role-ec2 instance-profiles html   and to keep the same behaviour in your Dev Test environment  use something like Hologram from AdRoll  https   github com AdRoll hologram

User · Answer

from boto3 s3 transfer import S3Transfer import boto3  have all the variables populated which are required below client   boto3 client  s3   aws access key id access key aws secret access key secret key  transfer   S3Transfer client  transfer upload file filepath  bucket name  folder name     filename

User · Answer

I used this and it is very simple to implement  import tinys3  conn   tinys3 Connection  S3 ACCESS KEY   S3 SECRET KEY  tls True   f   open  some file zip   rb   conn upload  some file zip  f  my bucket     https   www smore com labs tinys3

User · Answer

I have something that seems to me has a bit more order  import boto3 from pprint import pprint from botocore exceptions import NoCredentialsError   class S3 object       BUCKET    quot test quot      connection   None      def   init   self           try              vars   get s3 credentials  quot aws quot               self connection   boto3 resource  s3    aws access key id                                                 aws secret access key           except Exception  as error              print error              self connection   None       def upload file self  file to upload path  file name           if file to upload is None or file name is None  return False         try              pprint file to upload              file name    quot your-folder-inside-s3  0  quot  format file name              self connection Bucket self BUCKET  upload file file to upload path                                                                         file name              print  quot Upload Successful quot               return True          except FileNotFoundError              print  quot The file was not found quot               return False          except NoCredentialsError              print  quot Credentials not available quot               return False    There re three important variables here  the BUCKET const  the file to upload and the file name BUCKET  is the name of your S3 bucket file to upload path  must be the path from file you want to upload file name  is the resulting file and path in your bucket  this is where you add folders or what ever  There s many ways but you can reuse this code in another script like this import S3  def some function        S3 S3   upload file path to file  final file name

User · Answer

This will also work   import os  import boto import boto s3 connection from boto s3 key import Key  try       conn   boto s3 connect to region  us-east-1       aws access key id    AWS-Access-Key       aws secret access key    AWS-Secrete-Key         host    s3-website-us-east-1 amazonaws com         is secure True                  uncomment if you are not using ssl     calling format   boto s3 connection OrdinaryCallingFormat               bucket   conn get bucket  YourBucketName       key name    FileToUpload      path    images holiday   Directory Under which file should get upload     full key name   os path join path  key name      k   bucket new key full key name      k set contents from filename key name   except Exception e      print str e      print  error

User · Answer

import boto3  s3   boto3 resource  s3   BUCKET    test   s3 Bucket BUCKET  upload file  your local file    dump file

User · Answer

xmlstr   etree tostring listings   encoding  utf8   method  xml   conn   boto connect s3          aws access key id   access key          aws secret access key   secret key            host     lt bucketName gt  s3 amazonaws com           host    bycket s3 amazonaws com            is secure False                  uncomment if you are not using ssl         calling format   boto s3 connection OrdinaryCallingFormat              conn auth region name    us-west-1   bucket   conn get bucket  resources   validate False  key  bucket get key  filename txt   key set contents from string  SAMPLE TEXT   key set canned acl  public-read

[python] How to upload a file to directory in S3 bucket using boto

Examples related to python

Examples related to amazon-web-services

Examples related to amazon-s3

Examples related to boto