[shell] Using curl to upload POST data with files

I would like to use cURL to not only send data parameters in HTTP POST but to also upload files with specific form name. How should I go about doing that ?

HTTP Post parameters:

userid = 12345 filecomment = This is an image file

HTTP File upload: File location = /home/user1/Desktop/test.jpg Form name for file = image (correspond to the $_FILES['image'] at the PHP side)

I figured part of the cURL command as follows:

curl -d "userid=1&filecomment=This is an image file" --data-binary @"/home/user1/Desktop/test.jpg" localhost/uploader.php

The problem I am getting is as follows:

Notice: Undefined index: image in /var/www/uploader.php

The problem is I am using $_FILES['image'] to pick up files in the PHP script.

How do I adjust my cURL commands accordingly ?

This question is related to shell file curl post file-upload

The answer is


You need to use the -F option:
-F/--form <name=content> Specify HTTP multipart POST data (H)

Try this:

curl \
  -F "userid=1" \
  -F "filecomment=This is an image file" \
  -F "image=@/home/user1/Desktop/test.jpg" \
  localhost/uploader.php

The issue that lead me here turned out to be a basic user error - I wasn't including the @ sign in the path of the file and so curl was posting the path/name of the file rather than the contents. The Content-Length value was therefore 8 rather than the 479 I expected to see given the legnth of my test file.

The Content-Length header will be automatically calculated when curl reads and posts the file.

curl -i -H "Content-Type: application/xml" --data "@test.xml" -v -X POST https://<url>/<uri/

... < Content-Length: 479 ...

Posting this here to assist other newbies in future.


Here is my solution, I have been reading a lot of posts and they were really helpful. Finally I wrote some code for small files, with cURL and PHP that I think its really useful.

public function postFile()
{    
        $file_url = "test.txt";  //here is the file route, in this case is on same directory but you can set URL too like "http://examplewebsite.com/test.txt"
        $eol = "\r\n"; //default line-break for mime type
        $BOUNDARY = md5(time()); //random boundaryid, is a separator for each param on my post curl function
        $BODY=""; //init my curl body
        $BODY.= '--'.$BOUNDARY. $eol; //start param header
        $BODY .= 'Content-Disposition: form-data; name="sometext"' . $eol . $eol; // last Content with 2 $eol, in this case is only 1 content.
        $BODY .= "Some Data" . $eol;//param data in this case is a simple post data and 1 $eol for the end of the data
        $BODY.= '--'.$BOUNDARY. $eol; // start 2nd param,
        $BODY.= 'Content-Disposition: form-data; name="somefile"; filename="test.txt"'. $eol ; //first Content data for post file, remember you only put 1 when you are going to add more Contents, and 2 on the last, to close the Content Instance
        $BODY.= 'Content-Type: application/octet-stream' . $eol; //Same before row
        $BODY.= 'Content-Transfer-Encoding: base64' . $eol . $eol; // we put the last Content and 2 $eol,
        $BODY.= chunk_split(base64_encode(file_get_contents($file_url))) . $eol; // we write the Base64 File Content and the $eol to finish the data,
        $BODY.= '--'.$BOUNDARY .'--' . $eol. $eol; // we close the param and the post width "--" and 2 $eol at the end of our boundary header.



        $ch = curl_init(); //init curl
        curl_setopt($ch, CURLOPT_HTTPHEADER, array(
                         'X_PARAM_TOKEN : 71e2cb8b-42b7-4bf0-b2e8-53fbd2f578f9' //custom header for my api validation you can get it from $_SERVER["HTTP_X_PARAM_TOKEN"] variable
                         ,"Content-Type: multipart/form-data; boundary=".$BOUNDARY) //setting our mime type for make it work on $_FILE variable
                    );
        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/1.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0'); //setting our user agent
        curl_setopt($ch, CURLOPT_URL, "api.endpoint.post"); //setting our api post url
        curl_setopt($ch, CURLOPT_COOKIEJAR, $BOUNDARY.'.txt'); //saving cookies just in case we want
        curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); // call return content
        curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1); navigate the endpoint
        curl_setopt($ch, CURLOPT_POST, true); //set as post
        curl_setopt($ch, CURLOPT_POSTFIELDS, $BODY); // set our $BODY 


        $response = curl_exec($ch); // start curl navigation

     print_r($response); //print response

}

With this we should be get on the "api.endpoint.post" the following vars posted. You can easily test with this script, and you should be receive this debugs on the function postFile() at the last row.

print_r($response); //print response

public function getPostFile()
{

    echo "\n\n_SERVER\n";
    echo "<pre>";
    print_r($_SERVER['HTTP_X_PARAM_TOKEN']);
    echo "/<pre>";
    echo "_POST\n";
    echo "<pre>";
    print_r($_POST['sometext']);
    echo "/<pre>";
    echo "_FILES\n";
    echo "<pre>";
    print_r($_FILEST['somefile']);
    echo "/<pre>";
}

It should work well, they may be better solutions but this works and is really helpful to understand how the Boundary and multipart/from-data mime works on PHP and cURL library.


Here is how to correctly escape arbitrary filenames of uploaded files with bash:

#!/bin/bash
set -eu

f="$1"
f=${f//\\/\\\\}
f=${f//\"/\\\"}
f=${f//;/\\;}

curl --silent --form "uploaded=@\"$f\"" "$2"

As an alternative to curl, you can use HTTPie, it'a CLI, cURL-like tool for humans.

  1. Installation instructions: https://github.com/jakubroztocil/httpie#installation

  2. Then, run:

    http -f POST http://localhost:4040/api/users username=johnsnow photo@images/avatar.jpg
    
    HTTP/1.1 200 OK
    Access-Control-Expose-Headers: X-Frontend
    Cache-control: no-store
    Connection: keep-alive
    Content-Encoding: gzip
    Content-Length: 89
    Content-Type: text/html; charset=windows-1251
    Date: Tue, 26 Jun 2018 11:11:55 GMT
    Pragma: no-cache
    Server: Apache
    Vary: Accept-Encoding
    X-Frontend: front623311
    
    ...
    

if you are uploading binary file such as csv, use below format to upload file

curl -X POST \
    'http://localhost:8080/workers' \
    -H 'authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6ImFjY2VzcyIsInR5cGUiOiJhY2Nlc3MifQ.eyJ1c2VySWQiOjEsImFjY291bnRJZCI6MSwiaWF0IjoxNTExMzMwMzg5LCJleHAiOjE1MTM5MjIzODksImF1ZCI6Imh0dHBzOi8veW91cmRvbWFpbi5jb20iLCJpc3MiOiJmZWF0aGVycyIsInN1YiI6ImFub255bW91cyJ9.HWk7qJ0uK6SEi8qSeeB6-TGslDlZOTpG51U6kVi8nYc' \
    -H 'content-type: application/x-www-form-urlencoded' \
    --data-binary '@/home/limitless/Downloads/iRoute Masters - Workers.csv'

Catching the user id as path variable (recommended):

curl -i -X POST -H "Content-Type: multipart/form-data" 
-F "[email protected]" http://mysuperserver/media/1234/upload/

Catching the user id as part of the form:

curl -i -X POST -H "Content-Type: multipart/form-data" 
-F "[email protected];userid=1234" http://mysuperserver/media/upload/

or:

curl -i -X POST -H "Content-Type: multipart/form-data" 
-F "[email protected]" -F "userid=1234" http://mysuperserver/media/upload/

After a lot of tries this command worked for me:

curl -v -F filename=image.jpg -F [email protected] http://localhost:8080/api/upload

I got it worked with this command curl -F 'filename=@/home/yourhomedirextory/file.txt' http://yourserver/upload


Examples related to shell

Comparing a variable with a string python not working when redirecting from bash script Get first line of a shell command's output How to run shell script file using nodejs? Run bash command on jenkins pipeline Way to create multiline comments in Bash? How to do multiline shell script in Ansible How to check if a file exists in a shell script How to check if an environment variable exists and get its value? Curl to return http status code along with the response docker entrypoint running bash script gets "permission denied"

Examples related to file

Gradle - Move a folder from ABC to XYZ Difference between opening a file in binary vs text Angular: How to download a file from HttpClient? Python error message io.UnsupportedOperation: not readable java.io.FileNotFoundException: class path resource cannot be opened because it does not exist Writing JSON object to a JSON file with fs.writeFileSync How to read/write files in .Net Core? How to write to a CSV line by line? Writing a dictionary to a text file? What are the pros and cons of parquet format compared to other formats?

Examples related to curl

What is the incentive for curl to release the library for free? curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number Converting a POSTMAN request to Curl git clone error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054 How to post raw body data with curl? Curl : connection refused How to use the curl command in PowerShell? Curl to return http status code along with the response How to install php-curl in Ubuntu 16.04 curl: (35) SSL connect error

Examples related to post

How to post query parameters with Axios? How can I add raw data body to an axios request? HTTP POST with Json on Body - Flutter/Dart How do I POST XML data to a webservice with Postman? How to set header and options in axios? Redirecting to a page after submitting form in HTML How to post raw body data with curl? How do I make a https post in Node Js without any third party module? How to convert an object to JSON correctly in Angular 2 with TypeScript Postman: How to make multiple requests at the same time

Examples related to file-upload

bootstrap 4 file input doesn't show the file name How to post a file from a form with Axios File Upload In Angular? How to set the max size of upload file The request was rejected because no multipart boundary was found in springboot Send multipart/form-data files with angular using $http File upload from <input type="file"> How to upload files in asp.net core? REST API - file (ie images) processing - best practices Angular - POST uploaded file