Upload a file to Amazon S3 with NodeJS

Question

I ran into a problem while trying to upload a file to my S3 bucket  Everything works except that my file paramters do not seem appropriate  I am using Amazon S3 sdk to upload from nodejs to s3   These are my routes settings   var multiparty   require  connect-multiparty        multipartyMiddleware   multiparty    app route   api items upload   post multipartyMiddleware  items upload     This is items upload   function   exports upload   function req  res        var file   req files file      var s3bucket   new AWS S3  params   Bucket   mybucketname          s3bucket createBucket function             var params                 Key  file name              Body  file                    s3bucket upload params  function err  data                console log  PRINT FILE    file               if  err                    console log  ERROR MSG     err                 else                   console log  Successfully uploaded data                                           Setting Body param to a string like  hello  works fine  According to doc  Body param must take   Buffer  Typed Array  Blob  String  ReadableStream  Object data  However  uploading a file object fails with the following error message    Error  Unsupported body payload object    This is the file object     fieldName   file     originalFilename   second fnp png     path    var folders ps l8lvygws0w93trqz7yj1t5sr0000gn T 26374-7ttwvc png     headers         content-disposition    form-data  name  file   filename  second fnp png          content-type    image png       ws         writableState           highWaterMark  16384          objectMode  false          needDrain  true          ending  true          ended  true          finished  true          decodeStrings  true          defaultEncoding   utf8           length  0          writing  false          sync  false          bufferProcessing  false          onwrite   Function           writecb  null          writelen  0          buffer              errorEmitted  false         writable  true       domain  null        events    error   Object   close   Object           maxListeners  10       path    var folders ps l8lvygws0w93trqz7yj1t5sr0000gn T 26374-7ttwvc png        fd  null       flags   w        mode  438       start  undefined       pos  undefined       bytesWritten  261937       closed  true      size  261937    name   second fnp png     type   image png      Any help will be greatly appreciated

User · Answer

var express   require  express    app   module exports   express    var secureServer   require  http   createServer app   secureServer listen 3001    var aws   require  aws-sdk   var multer   require  multer   var multerS3   require  multer-s3        aws config update       secretAccessKey   XXXXXXXXXXXXXXXXXXXXXXXXXXXXX       accessKeyId   XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX       region   us-east-1              s3   new aws S3        var upload   multer      storage  multerS3       s3  s3      dirname   uploads       bucket   Your bucket name       key  function  req  file  cb            console log file           cb null   uploads profile images u     Date now       jpg      use        Date now   for unique file keys                     app post   upload   upload single  photos    function req  res  next      console log  Successfully uploaded    req file    res send  Successfully uploaded     req file length     files

User · Answer

I found the following to be a working solution     npm install aws-sdk    Once you ve installed the aws-sdk   use the following code replacing values with your where needed    var AWS   require  aws-sdk    var fs    require  fs     var s3   new AWS S3        Bucket names must be unique across all S3 users  var myBucket    njera    var myKey    jpeg     for text file   fs readFile  demo txt   function  err  data      for Video file   fs readFile  demo avi   function  err  data      for image file                 fs readFile  demo jpg   function  err  data      if  err    throw err            params    Bucket  myBucket  Key  myKey  Body  data          s3 putObject params  function err  data              if  err                  console log err              else                 console log  Successfully uploaded data to myBucket myKey                                  I found the complete tutorial on the subject here in case you re looking for references        How to upload files  text image video  in amazon s3 using node js

User · Answer

Or Using promises   const AWS   require  aws-sdk    AWS config update       accessKeyId   accessKeyId       secretAccessKey   secretAccessKey       region   region       let params         Bucket   yourBucketName       Key   someUniqueKey       Body   someFile     try       let uploadPromise   await new AWS S3   putObject params  promise        console log  Successfully uploaded data to bucket      catch  e        console log  Error uploading data     e

User · Answer

Uploading a file to AWS s3 and sending the url in response for accessing the file   Multer is a node js middleware for handling multipart form-data  which is primarily used for uploading files  It is written on top of busboy for maximum efficiency  check this npm module here   When you are sending the request  make sure the headers  have Content-Type is multipart form-data  We are sending the file location in the response  which will give the url  but if you want to access that url  make the bucket public or else you will not be able to access it   upload router js  const express   require  express    const router   express Router    const AWS   require  aws-sdk    const multer   require  multer    const storage   multer memoryStorage   const upload   multer  storage  storage     const s3Client   new AWS S3       accessKeyId   your access key id       secretAccessKey   your secret access id       region   ur region       const uploadParams              Bucket   ur bucket name             Key         pass key          Body  null     pass file body      router post   api file upload   upload single  file    req res    gt        const params   uploadParams       uploadParams Key   req file originalname      uploadParams Body   req file buffer       s3Client upload params   err  data    gt            if  err                res status 500  json  error  Error - gt      err                      res json  message   File uploaded successfully   filename            req file originalname   location   data Location                 module exports   router    app js  const express   require  express    const app   express     const router   require    app routers upload router js    app use      router       Create a Server   const server   app listen 8080       gt      console log  App listening at 8080

User · Answer

So it looks like there are a few things going wrong here  Based on your post it looks like you are attempting to support file uploads using the connect-multiparty middleware  What this middleware does is take the uploaded file  write it to the local filesystem and then sets req files to the the uploaded file s    The configuration of your route looks fine  the problem looks to be with your items upload   function  In particular with this part   var params       Key  file name    Body  file      As I mentioned at the beginning of my answer connect-multiparty writes the file to the local filesystem  so you ll need to open the file and read it  then upload it  and then delete it on the local filesystem   That said you could update your method to something like the following   var fs   require  fs    exports upload   function  req  res        var file   req files file      fs readFile file path  function  err  data            if  err  throw err     Something went wrong          var s3bucket   new AWS S3  params   Bucket   mybucketname              s3bucket createBucket function                  var params                     Key  file originalFilename    file name doesn t exist as a property                 Body  data                            s3bucket upload params  function  err  data                       Whether there is an error or not  delete the temp file                 fs unlink file path  function  err                        if  err                            console error err                                             console log  Temp File Delete                                         console log  PRINT FILE    file                   if  err                        console log  ERROR MSG     err                       res status 500  send err                     else                       console log  Successfully uploaded data                        res status 200  end                                                               What this does is read the uploaded file from the local filesystem  then uploads it to S3  then it deletes the temporary file and sends a response   There s a few problems with this approach  First off  it s not as efficient as it could be  as for large files you will be loading the entire file before you write it  Secondly  this process doesn t support multi-part uploads for large files  I think the cut-off is 5 Mb before you have to do a multi-part upload    What I would suggest instead is that you use a module I ve been working on called S3FS which provides a similar interface to the native FS in Node JS but abstracts away some of the details such as the multi-part upload and the S3 api  as well as adds some additional functionality like recursive methods    If you were to pull in the S3FS library your code would look something like this   var fs   require  fs        S3FS   require  s3fs        s3fsImpl   new S3FS  mybucketname             accessKeyId  XXXXXXXXXXX          secretAccessKey  XXXXXXXXXXXXXXXXX             Create our bucket if it doesn t exist s3fsImpl create     exports upload   function  req  res        var file   req files file      var stream   fs createReadStream file path       return s3fsImpl writeFile file originalFilename  stream  then function              fs unlink file path  function  err                if  err                    console error err                                     res status 200  end                 What this will do is instantiate the module for the provided bucket and AWS credentials and then create the bucket if it doesn t exist  Then when a request comes through to upload a file we ll open up a stream to the file and use it to write the file to S3 to the specified path  This will handle the multi-part upload piece behind the scenes  if needed  and has the benefit of being done through a stream  so you don t have to wait to read the whole file before you start uploading it   If you prefer  you could change the code to callbacks from Promises  Or use the pipe   method with the event listener to determine the end errors   If you re looking for some additional methods  check out the documentation for s3fs and feel free to open up an issue if you are looking for some additional methods or having issues

User · Answer

Thanks to David as his solution helped me come up with my solution for uploading multi-part files from my Heroku hosted site to S3 bucket  I did it using formidable to handle incoming form and fs to get the file content  Hopefully  it may help you   api service ts  public upload files   Observable lt any gt          const formData  FormData   new FormData         files forEach file   gt             create a new multipart-form for every file        formData append  file   file  file name                             return this http post uploadUrl  formData  pipe        map this extractData         catchError this handleError             server js  app post   api upload   upload   app use   api upload   router     upload js  const IncomingForm   require  formidable   IncomingForm  const fs   require  fs    const AWS   require  aws-sdk     module exports   function upload req  res        var form   new IncomingForm         const bucket   new AWS S3                  signatureVersion   v4           accessKeyId  process env AWS ACCESS KEY ID          secretAccessKey  process env AWS SECRET ACCESS KEY          region   us-east-1                              form on  file    field  file    gt             const fileContent   fs readFileSync file path            const s3Params                 Bucket  process env AWS S3 BUCKET              Key   folder     file name              Expires  60                           Body  fileContent              ACL   public-read                      bucket upload s3Params  function err  data                if  err                    throw err                                        console log  File uploaded to      data Location               fs unlink file path  function  err                  if  err                      console error err                                 console log  Temp File Delete                                                            The second callback is called when the form is completely parsed          In this case  we want to send back a success status code      form on  end        gt                  res status 200  json  upload ok                 form parse req       upload-image component ts  import   Component  OnInit  ViewChild  Output  EventEmitter  Input   from   angular core   import   ApiService   from     api service   import   MatSnackBar   from   angular material snack-bar     Component     selector   app-upload-image     templateUrl     upload-image component html     styleUrls      upload-image component css       export class UploadImageComponent implements OnInit     public files  Set lt File gt    new Set       ViewChild  file     static  false    file    public uploadedFiles  Array lt string gt    new Array lt string gt       public uploadedFileNames  Array lt string gt    new Array lt string gt        Output   filesOutput   new EventEmitter lt Array lt string gt  gt        Input   CurrentImage  string     Input   IsPublic  boolean     Output   valueUpdate   new EventEmitter      strUploadedFiles string         filesUploaded  boolean   false          constructor private api  ApiService  public snackBar  MatSnackBar          ngOnInit                updateValue val          this valueUpdate emit val              reset           this files   new Set        this uploadedFiles   new Array lt string gt         this uploadedFileNames   new Array lt string gt         this filesUploaded   false         upload           this api upload this files  subscribe res   gt             this filesOutput emit this uploadedFiles          if  res     upload ok                   this reset                         err   gt          console log err                  onFilesAdded         var txt           const files     key  string   File     this file nativeElement files       for  let key in files          if   isNaN parseInt key               var currentFile   files key           var sFileExtension   currentFile name split      currentFile name split      length - 1  toLowerCase            var iFileSize   currentFile size           if    sFileExtension      jpg                    sFileExtension      png                     iFileSize  gt  671329                txt    File type       sFileExtension     n n               txt     Size      iFileSize     n n               txt     Please make sure your file is in jpg or png format and less than 655 KB  n n               alert txt               return false                     this files add files key            this uploadedFiles push  https   gourmet-philatelist-assets s3 amazonaws com folder     files key  name           this uploadedFileNames push files key  name           if  this IsPublic  amp  amp  this uploadedFileNames length    1                      this filesUploaded   true            this updateValue files key  name             break                     else if   this IsPublic  amp  amp  this uploadedFileNames length    3                      this strUploadedFiles    files key  name                      this updateValue this strUploadedFiles              this filesUploaded   true            break                    else                     this strUploadedFiles    files key  name                            this updateValue this strUploadedFiles                                             addFiles         this file nativeElement click             openSnackBar message  string  action  string        this snackBar open message  action          duration  2000        verticalPosition   top                      upload-image component html   lt input type  file   file style  display  none   change   onFilesAdded    multiple   gt   amp nbsp  lt button mat-raised-button color  primary             disabled   filesUploaded   click    event preventDefault    addFiles    gt    Add Files  lt  button gt   amp nbsp  lt button class  btn btn-success   disabled   uploadedFileNames length    0   click    event preventDefault    upload    gt    Upload  lt  button gt

User · Answer

Upload CSV Excel  const fs   require  fs    const AWS   require  aws-sdk     const s3   new AWS S3       accessKeyId  XXXXXXXXX      secretAccessKey  XXXXXXXXX      const absoluteFilePath    C   Project  test xlsx    const uploadFile        gt      fs readFile absoluteFilePath   err  data    gt         if  err  throw err       const params              Bucket   testBucket      pass your bucket name          Key   folderName key xlsx      file will be saved in  lt folderName gt  folder          Body  data               s3 upload params  function  s3Err  data                        if  s3Err  throw s3Err                     console log  File uploaded successfully at   data Location                         debugger                                uploadFile

[node.js] Upload a file to Amazon S3 with NodeJS

Examples related to node.js

Examples related to amazon-s3

Examples related to multipartform-data