AmazonS3 putObject with InputStream length example

Question

I am uploading a file to S3 using Java - this is what I got so far   AmazonS3 s3   new AmazonS3Client new BasicAWSCredentials  XX   YY      List lt Bucket gt  buckets   s3 listBuckets     s3 putObject new PutObjectRequest buckets get 0  getName    fileName  stream  new ObjectMetadata        The file is being uploaded but a WARNING is raised when I am not setting the content length    com amazonaws services s3 AmazonS3Client putObject  No content length specified for stream  gt  data   Stream contents will be buffered in memory and could result in out of memory errors     This is a file I am uploading and the stream variable is an InputStream  from which I can get the byte array like this  IOUtils toByteArray stream    So when I try to set the content length and MD5  taken from here  like this      get MD5 base64 hash MessageDigest messageDigest   MessageDigest getInstance  MD5    messageDigest reset    messageDigest update IOUtils toByteArray stream    byte   resultByte   messageDigest digest    String hashtext   new String Hex encodeHex resultByte     ObjectMetadata meta   new ObjectMetadata    meta setContentLength IOUtils toByteArray stream  length   meta setContentMD5 hashtext     It causes the following error to come back from S3      The Content-MD5 you specified was invalid    What am I doing wrong   Any help appreciated   P S  I am on Google App Engine - I cannot write the file to disk or create a temp file because AppEngine does not support FileOutputStream

User · Answer

While writing to S3  you need to specify the length of S3 object to be sure that there are no out of memory errors      Using IOUtils toByteArray stream  is also prone to OOM errors because this is backed by ByteArrayOutputStream   So  the best option is to first write the inputstream to a temp file on local disk and then use that file to write to S3 by specifying the length of temp file

User · Answer

adding log4j-1 2 12 jar file has resolved the issue for me

User · Answer

For uploading  the S3 SDK has two putObject methods   PutObjectRequest String bucketName  String key  File file    and  PutObjectRequest String bucketName  String key  InputStream input  ObjectMetadata metadata    The inputstream ObjectMetadata method needs a minimum metadata of Content Length of your inputstream  If you don t  then it will buffer in-memory to get that information  this could cause OOM  Alternatively  you could do your own in-memory buffering to get the length  but then you need to get a second inputstream   Not asked by the OP  limitations of his environment   but for someone else  such as me  I find it easier  and safer  if you have access to temp file   to write the inputstream to a temp file  and put the temp file  No in-memory buffer  and no requirement to create a second inputstream   AmazonS3 s3Service   new AmazonS3Client awsCredentials   File scratchFile   File createTempFile  prefix    suffix    try       FileUtils copyInputStreamToFile inputStream  scratchFile           PutObjectRequest putObjectRequest   new PutObjectRequest bucketName  id  scratchFile       PutObjectResult putObjectResult   s3Service putObject putObjectRequest      finally       if scratchFile exists              scratchFile delete

User · Answer

I ve created a library that uses multipart uploads in the background to avoid buffering everything in memory and also doesn t write to disk  https   github com alexmojaki s3-stream-upload

User · Answer

Just passing the file object to the putobject method worked for me  If you are getting a stream  try writing it to a temp file before passing it on to S3    amazonS3 putObject bucketName  id fileObject     I am using Aws SDK v1 11 414  The answer at https   stackoverflow com a 35904801 2373449 helped me

User · Answer

If all you are trying to do is solve the content length error from amazon then you could just read the bytes from the input stream to a Long and add that to the metadata         Obtain the Content length of the Input stream for S3 header     try       InputStream is   event getFile   getInputstream        contentBytes   IOUtils toByteArray is     catch  IOException e        System err printf  Failed while reading bytes from  s   e getMessage         Long contentLength   Long valueOf contentBytes length    ObjectMetadata metadata   new ObjectMetadata    metadata setContentLength contentLength          Reobtain the tmp uploaded file as input stream     InputStream inputStream   event getFile   getInputstream           Put the object in S3     try        s3client putObject new PutObjectRequest bucketName  keyName  inputStream  metadata       catch  AmazonServiceException ase        System out println  Error Message         ase getMessage         System out println  HTTP Status Code      ase getStatusCode         System out println  AWS Error Code        ase getErrorCode         System out println  Error Type            ase getErrorType         System out println  Request ID            ase getRequestId       catch  AmazonClientException ace        System out println  Error Message      ace getMessage       finally       if  inputStream    null            inputStream close              You ll need to read the input stream twice using this exact method so if you are uploading a very large file you might need to look at reading it once into an array and then reading it from there

User · Answer

i am actually doing somewhat same thing but on my AWS S3 storage -  Code for servlet which is receiving uploaded file -  import java io IOException  import java io PrintWriter  import java util List   import javax servlet ServletException  import javax servlet http HttpServlet  import javax servlet http HttpServletRequest  import javax servlet http HttpServletResponse   import org apache commons fileupload FileItem  import org apache commons fileupload disk DiskFileItemFactory  import org apache commons fileupload servlet ServletFileUpload   import com src code s3 S3FileUploader   public class FileUploadHandler extends HttpServlet        protected void doGet HttpServletRequest request  HttpServletResponse response  throws ServletException  IOException           doPost request  response              protected void doPost HttpServletRequest request  HttpServletResponse response  throws ServletException  IOException           PrintWriter out   response getWriter             try              List lt FileItem gt  multipartfiledata   new ServletFileUpload new DiskFileItemFactory    parseRequest request                  upload to S3             S3FileUploader s3   new S3FileUploader                String result   s3 fileUploader multipartfiledata                out print result             catch Exception e               System out println e getMessage                         Code which is uploading this data as AWS object -  import java io ByteArrayInputStream  import java io IOException  import java util List  import java util UUID   import org apache commons fileupload FileItem   import com amazonaws AmazonClientException  import com amazonaws AmazonServiceException  import com amazonaws auth ClasspathPropertiesFileCredentialsProvider  import com amazonaws services s3 AmazonS3  import com amazonaws services s3 AmazonS3Client  import com amazonaws services s3 model ObjectMetadata  import com amazonaws services s3 model PutObjectRequest  import com amazonaws services s3 model S3Object   public class S3FileUploader         private static String bucketName           NAME OF YOUR BUCKET          private static String keyName           Object-  UUID randomUUID         public String fileUploader List lt FileItem gt  fileData  throws IOException           AmazonS3 s3   new AmazonS3Client new ClasspathPropertiesFileCredentialsProvider             String result    Upload unsuccessfull because            try                S3Object s3Object   new S3Object                 ObjectMetadata omd   new ObjectMetadata                omd setContentType fileData get 0  getContentType                 omd setContentLength fileData get 0  getSize                 omd setHeader  filename   fileData get 0  getName                  ByteArrayInputStream bis   new ByteArrayInputStream fileData get 0  get                  s3Object setObjectContent bis               s3 putObject new PutObjectRequest bucketName  keyName  bis  omd                s3Object close                 result    Uploaded Successfully              catch  AmazonServiceException ase               System out println  Caught an AmazonServiceException  which means your request made it to Amazon S3  but was                      rejected with an error response for some reason                 System out println  Error Message         ase getMessage                System out println  HTTP Status Code      ase getStatusCode                System out println  AWS Error Code        ase getErrorCode                System out println  Error Type            ase getErrorType                System out println  Request ID            ase getRequestId                 result   result   ase getMessage              catch  AmazonClientException ace               System out println  Caught an AmazonClientException  which means the client encountered an internal error while                      trying to communicate with S3  such as not being able to access the network                 result   result   ace getMessage              catch  Exception e                 result   result   e getMessage                      return result            Note  - I am using aws properties file for credentials   Hope this helps

User · Answer

Because the original question was never answered  and I had to run into this same problem  the solution for the MD5 problem is that S3 doesn t want the Hex encoded MD5 string we normally think about    Instead  I had to do this       content is a passed in InputStream byte   resultByte   DigestUtils md5 content   String streamMD5   new String Base64 encodeBase64 resultByte    metaData setContentMD5 streamMD5     Essentially what they want for the MD5 value is the Base64 encoded raw MD5 byte-array  not the Hex string  When I switched to this it started working great for me

[java] AmazonS3 putObject with InputStream length example

Examples related to java

Examples related to google-app-engine

Examples related to amazon-s3

Examples related to md5

Examples related to inputstream