How can I run multiple curl requests processed sequentially

Question

Assuming I m a big unix rookie   I m running a curl request through cron every 15 minutes   Curl basically is used to load a web page  PHP  that given some arguments  acts as a script like  curl http   example com  update  1    What I would like to achieve is to run another  quot script quot  using this curl technique   every time the other script is run before the other script is run  I have read that curl accepts multiple URLs in one command but I m unsure if this would process the ulrs sequentially or in  quot parallel quot

User · Answer

Write a script with two curl requests in desired order and run it by cron  like     bin bash curl http   mysite com  update  1 curl http   mysite com  the other thing

User · Answer

It would most likely process them sequentially  why not just test it   But you can also do this   make a file called curlrequests sh  put it in a file like thus  curl http   example com  update  1 curl http   example com  update  3 curl http   example com  update  234 curl http   example com  update  65   save the file and make it executable with chmod  chmod  x curlrequests sh   run your file    curlrequests sh    or     path to file curlrequests sh  As a side note  you can chain requests with  amp  amp   like this     curl http   example com  update  1  amp  amp  curl http   example com  update  2  amp  amp  curl http   example com update  3   And execute in parallel using  amp      curl http   example com  update  1  amp  curl http   example com  update  2  amp  curl http   example com  update  3

User · Answer

According to the curl man page      You can specify any amount of URLs on the command line  They will be   fetched in a sequential  manner in the specified order    So the simplest and most efficient  curl will send them all down a single TCP connection  those to the same origin   approach would be put them all on a single invocation of curl e g    curl http   example com  update  1 http   example com  update  2

User · Answer

Another crucial method not mentioned here is using the same TCP connection for multiple HTTP requests  and exactly one curl command for this  This is very useful to save network bandwidth  client and server resources  and overall the need of using multiple curl commands  as curl by default closes the connection when end of command is reached  Keeping the connection open and reusing it is very common for standard clients running a web-app  Starting curl version 7 36 0  the --next or -  command-line option allows to chain multiple requests  and usable both in command-line and scripting  For example   Sending multiple requests on the same TCP connection    curl http   example com  update  1 -  http   example com foo   Sending multiple different HTTP requests on the same connection    curl http   example com  update  1 -  -d  quot I am posting this string quot  http   example com  update  2    Sending multiple HTTP requests with different curl options for each request    curl -o  my output file  http   example com  update  1 -  -d  quot my data quot  -s -m 10 http   example com foo -  -o  dev null http   example com random  From the curl manpage   -   --next Tells curl to use a separate operation for the following URL and associated options  This allows you to send several URL requests  each with their own specific options  for example  such as different user names or custom requests for each  -   --next will reset all local options and only global ones will have their values survive over to the operation following the -   --next instruction  Global options include -v  --verbose  --trace  --trace-ascii and --fail-early  For example  you can do both a GET and a POST in a single command line  curl www1 example com --next -d postthis www2 example com Added in 7 36 0

User · Answer

This will do what you want  uses an input file and is super fast     bin bash IFS    n  file  path to input txt lines   cat   file   for line in   lines   do    curl    line   done IFS    exit        one entry per line on your input file  it will follow the order of your input file  save it as whatever sh and make it executable

User · Answer

You can specify any amount of URLs on the command line  They will be fetched in a sequential manner in the specified order

User · Answer

I think this uses more native capabilities    printing the links to a file   echo  https   stackoverflow com questions 3110444  https   stackoverflow com questions 8445445  https   stackoverflow com questions 4875446    gt  links file txt     xargs curl  lt  links file txt   Enjoy

[curl] How can I run multiple curl requests processed sequentially?

Examples related to curl

Examples related to request