How can I send cookies using PHP curl in addition to CURLOPT COOKIEFILE

Question

I am scraping some content from a website after a form submission  The problem is that the script is failing every now and then  say 2 times out of 5 the script fails  I am using php curl  COOKIEFILE and COOKIEJAR to handle the cookie  However when I observed the sent headers of my browser  when visiting the target website from my browser and using live http headers  and the headers sent by php and saw there are many differences    My browser sent a lot more cookie variables than php curl  I think this difference might be because javascript is resposible for setting most of the cookies  however I m not sure about this   I am using the below code to do the scraping and I am showing the sent headers of my browser and of php curl    ckfile   tempnam    tmp    cookiename      url    https   www domain com firststep    poststring    variable1 4 amp variable2 5    ch   curl init   url   curl setopt   ch  CURLOPT COOKIEJAR   ckfile   curl setopt   ch  CURLOPT COOKIEFILE   ckfile   curl setopt  ch  CURLOPT CONNECTTIMEOUT  30   curl setopt  ch  CURLOPT RETURNTRANSFER  true   curl setopt  ch  CURLOPT SSL VERIFYPEER  false   curl setopt  ch  CURLOPT FOLLOWLOCATION  1   curl setopt   ch  CURLOPT POST  1   curl setopt   ch  CURLOPT POSTFIELDS   poststring    output   curl exec   ch   curl close  ch       url    https   www domain com nextstep    poststring    variableB1 4 amp variableB2 5    ch   curl init   url   curl setopt   ch  CURLOPT COOKIEJAR   ckfile   curl setopt   ch  CURLOPT COOKIEFILE   ckfile   curl setopt  ch  CURLOPT CONNECTTIMEOUT  30   curl setopt  ch  CURLOPT RETURNTRANSFER  true   curl setopt  ch  CURLOPT SSL VERIFYPEER  false   curl setopt  ch  CURLOPT FOLLOWLOCATION  1   curl setopt   ch  CURLOPT POST  1   curl setopt   ch  CURLOPT POSTFIELDS   poststring   curl setopt  ch  CURLINFO HEADER OUT  true    output   curl exec   ch    headers   curl getinfo  ch  CURLINFO HEADER OUT   curl close  ch    print r  headers       Gives  POST  d-cobs-web doffers html jsessionid 7BC2A5277A4EB07D9A7237A707BE1366 HTTP 1 1 User-Agent  Mozilla Host  domain subdomain nl Accept      Cookie  JSESSIONID 7BC2A5277A4EB07D9A7237A707BE1366  www-20480 MIFBNLFDFAAA Content-Length  187 Content-Type  application x-www-form-urlencoded     Where live http headers gives  POST  d-cobs-web doffers html jsessionid 7BC2A5277A4EB07D9A7237A707BE1366 HTTP 1 1 Host  domain subdomain nl User-Agent  Mozilla 5 0  Windows NT 6 1  WOW64  rv 21 0  Gecko 20100101 Firefox 21 0 Accept  text html application xhtml xml application xml q 0 9     q 0 8 Accept-Language  nl en-us q 0 7 en q 0 3 Accept-Encoding  gzip  deflate Content-Type  application x-www-form-urlencoded  charset UTF-8 Referer  https   domain subdomain nl dd doffers html returnUrl https 3A 2F 2Fttcc subdomain nl 2Fdd 2Fpreferences html 3FValueChanged 3Dfalse amp BEGBA  amp departureDate 13-06-2013 amp extChangeTime  amp pax2 0 amp bp  amp pax1 1 amp pax4 0 amp bk  amp pax3 0 amp shopId  amp xtpage  amp partner NSINT amp bc  amp xt pc  amp ov  amp departureTime  amp comfortClass 2 amp destination DEBHF amp thalysTicketless  amp beneUser  amp debugDOffer  amp logonId  amp valueChanged  amp iDomesticOrigin  amp rp  amp returnTime  amp locale nl NL amp vu  amp thePassWeekend false amp returnDate  amp xtsite  amp pax A amp lc2  amp lc1  amp lc4  amp lc3  amp lc6  amp lc5  amp BECRA  amp passType2  amp custId  amp lc9  amp iDomesticDestination  amp passType1 A amp lc7  amp lc8  amp origin NLASC amp toporef  amp pid  amp passType4  amp returnTimeType 1 amp passType3  amp departureTimeType 1 amp socusId  amp idr3  amp xtn2  amp loyaltyCard  amp idr2  amp idr1  amp thePassBusiness false amp cid 14812 Content-Length  219 Cookie  subdomainPARTNER NSINT  JSESSIONID CB3FEB3AC72AD61A80BFED91D3FD96CA  www-20480 MHFBNLFDFAAA  campaignPos 5  www-47873 MGFBNLFDFAAA    utma 1 993399624 1370027094 1370040145 1370082133 5    utmc 1    utmz 1 1370027094 1 1 utmcsr  direct  utmccn  direct  utmcmd  none   BCSessionID 5dc05787-c2c8-43e1-9abe-93989970b087  BCPermissionLevel PERSONAL    utmb 1 1 10 1370082133 Connection  keep-alive Pragma  no-cache Cache-Control  no-cache AJAXREQUEST  viewRoot amp doffersForm doffersForm amp doffersForm 3AvalueChanged  amp doffersForm 3ArequestValid true amp javax faces ViewState j id3 amp doffersForm 3Aj id937 doffersForm 3Aj id937 amp valueChanged false amp AJAX 3AEVENTS COUNT 1 amp    I would like to use    headers     array     headers      Cookie       cookie    and   curl setopt  ch  CURLOPT HTTPHEADER   headers     where    cookie    subdomainPARTNER NSINT  JSESSIONID CB3FEB3AC72AD61A80BFED91D3FD96CA  www-20480 MHFBNLFDFAAA  campaignPos 5  www-47873 MGFBNLFDFAAA    utma 1 993399624 1370027094 1370040145 1370082133 5    utmc 1    utmz 1 1370027094 1 1 utmcsr  direct  utmccn  direct  utmcmd  none   BCSessionID 5dc05787-c2c8-43e1-9abe-93989970b087  BCPermissionLevel PERSONAL    utmb 1 1 10 1370082133     Some of the parameters in the cookie above I might be able to scrape from the content of the website  but not all  Some of them I might be able to read from the  ckfile  but I don t know how to do that  Especially the utma utmc  utmz  utmcsr  utmccn  utmcmd I am not able to get from anywhere  I think these are generated by the javascript    Question 1  Am I doing something wrong with the cookie handling in the current code as very few cookie variables are sent by php curl and a lot more by the browser  Further  can other differences between sent headers by browser and php curl be a problem to return the right content   Question 2  Are the missing cookie variables due to the javascript setting those cookies   Question 3  What is the best way to handle the cookies to make sure that all required cookies are being sent to the remote server   Your help is very welcome

User · Answer

I think the only cookie you need is JSESSIONID=xxx..

Also NEVER share your cookies, becasuse someone may access your personal data that way. Specially when the cookies are session. These cookies will stop working once you logout the site.

User · Answer

Here is a list of examples for sending cookies - https   github com andriichuk php-curl-cookbook cookies   curlHandler   curl init     curl setopt array  curlHandler    CURLOPT URL   gt   https   httpbin org cookies   CURLOPT RETURNTRANSFER   gt  true   CURLOPT COOKIEFILE    gt   cookieFile  CURLOPT COOKIE   gt   foo bar baz foo           Or set header    CURLOPT HTTPHEADER   gt            Cookie  foo bar baz foo                  response   curl exec  curlHandler   curl close  curlHandler    echo  response

User · Answer

Try below code    cookieFile    cookies txt   if  file exists  cookieFile          fh   fopen  cookieFile   w        fwrite  fh           fclose  fh        ch   curl init    curl setopt  ch  CURLOPT URL   apiCall   curl setopt  ch  CURLOPT POST  TRUE   curl setopt  ch  CURLOPT POSTFIELDS   jsonDataEncoded   curl setopt  ch  CURLOPT RETURNTRANSFER  TRUE   curl setopt  ch  CURLOPT SSL VERIFYHOST  FALSE   curl setopt  ch  CURLOPT SSL VERIFYPEER  FALSE   curl setopt  ch  CURLOPT HTTPHEADER  array  Content-Type  application json     curl setopt  ch  CURLOPT COOKIEFILE   cookieFile      Cookie aware curl setopt  ch  CURLOPT COOKIEJAR   cookieFile      Cookie aware curl setopt  ch  CURLOPT VERBOSE  true   if  curl exec  ch        die  Error       curl error  ch       - Code      curl errno  ch      else       response   curl exec  ch      curl close  ch    result   json decode  response  true    echo   lt pre gt    var dump  result   echo  lt  pre gt      I hope this will help you    Best regards  Dasitha

User · Answer

If the cookie is generated from script  then you can send the cookie manually along with the cookie from the file using cookie-file option   For example     sending manually set cookie curl setopt  ch  CURLOPT HTTPHEADER  array  Cookie  test cookie        sending cookies from file curl setopt  ch  CURLOPT COOKIEFILE   ckfile     In this case curl will send your defined cookie along with the cookies from the file   If the cookie is generated through javascrript  then you have to trace it out how its generated and then you can send it using the above method through http-header    The utma utmc  utmz are seen when cookies are sent from Mozilla  You shouldn t bet worry about these things anymore   Finally  the way you are doing is alright  Just make sure you are using absolute path for the file names i e   var dir cookie txt  instead of relative one   Always enable the verbose mode when working with curl  It will help you a lot on tracing the requests  Also it will save lot of your times   curl setopt  ch  CURLOPT VERBOSE  true

[php] How can I send cookies using PHP curl in addition to CURLOPT_COOKIEFILE?

Examples related to php

Examples related to cookies

Examples related to curl

Examples related to setcookie