How can I emulate a get request exactly like a web browser

Question

There are websites that when I open specific ajax request on browser  I get the resulted page  But when I try to load them with curl  I receive an error from the server  How can I properly emulate a get request to the server that will simulate a browser  That is what I am doing   url  quot https   new aol com productsweb subflows ScreenNameFlow AjaxSNAction do s username amp f firstname amp l lastname quot   ini set  user agent    Mozilla 4 0  compatible  MSIE 6 0  Windows NT 5 1  SV1   NET CLR 1 0 3705   NET CLR 1 1 4322      ch   curl init     curl setopt  ch  CURLOPT URL  url    result curl exec  ch   print  result

User · Accepted Answer

Are you sure the curl module honors ini set  user agent        There is an option CURLOPT USERAGENT described at http   docs php net function curl-setopt  Could there also be a cookie tested by the server  That you can handle by using CURLOPT COOKIE  CURLOPT COOKIEFILE and or CURLOPT COOKIEJAR   edit  Since the request uses https there might also be error in verifying the certificate  see CURLOPT SSL VERIFYPEER    url  https   new aol com productsweb subflows ScreenNameFlow AjaxSNAction do s username amp f firstname amp l lastname    agent   Mozilla 4 0  compatible  MSIE 6 0  Windows NT 5 1  SV1   NET CLR 1 0 3705   NET CLR 1 1 4322      ch   curl init    curl setopt  ch  CURLOPT SSL VERIFYPEER  false   curl setopt  ch  CURLOPT VERBOSE  true   curl setopt  ch  CURLOPT RETURNTRANSFER  true   curl setopt  ch  CURLOPT USERAGENT   agent   curl setopt  ch  CURLOPT URL  url    result curl exec  ch   var dump  result

User · Answer

i ll make an example   first decide what browser you want to emulate  in this case i chose Firefox 60 6 1esr  64-bit   and check what GET request it issues  this can be obtained with a simple netcat server  MacOS bundles netcat  most linux distributions bunles netcat  and Windows users can get netcat from   Cygwin org   among other places    setting up the netcat server to listen on port 9999  nc -l 9999  now hitting http   127 0 0 1 9999 in firefox  i get     nc -l 9999 GET   HTTP 1 1 Host  127 0 0 1 9999 User-Agent  Mozilla 5 0  Windows NT 6 1  Win64  x64  rv 60 0  Gecko 20100101 Firefox 60 0 Accept  text html application xhtml xml application xml q 0 9     q 0 8 Accept-Language  en-US en q 0 5 Accept-Encoding  gzip  deflate Connection  keep-alive Upgrade-Insecure-Requests  1   now let us compare that with this simple script    lt  php  ch curl init  http   127 0 0 1 9999    curl exec  ch     i get     nc -l 9999 GET   HTTP 1 1 Host  127 0 0 1 9999 Accept        there are several missing headers here  they can all be added with the CURLOPT HTTPHEADER option of curl setopt  but the User-Agent specifically should be set with CURLOPT USERAGENT instead  it will be persistent across multiple calls to curl exec   and if you use CURLOPT FOLLOWLOCATION then it will persist across http redirections as well   and the Accept-Encoding header should be set with CURLOPT ENCODING instead  if they re set with CURLOPT ENCODING then curl will automatically decompress the response if the server choose to compress it  but if you set it via CURLOPT HTTPHEADER then you must manually detect and decompress the content yourself  which is a pain in the ass and completely unnecessary  generally speaking  so adding those we get    lt  php  ch curl init  http   127 0 0 1 9999    curl setopt array  ch array          CURLOPT USERAGENT  gt  Mozilla 5 0  Windows NT 6 1  Win64  x64  rv 60 0  Gecko 20100101 Firefox 60 0           CURLOPT ENCODING  gt  gzip  deflate           CURLOPT HTTPHEADER  gt array                   Accept  text html application xhtml xml application xml q 0 9     q 0 8                    Accept-Language  en-US en q 0 5                    Connection  keep-alive                    Upgrade-Insecure-Requests  1                  curl exec  ch     now running that code  our netcat server gets     nc -l 9999 GET   HTTP 1 1 Host  127 0 0 1 9999 User-Agent  Mozilla 5 0  Windows NT 6 1  Win64  x64  rv 60 0  Gecko 20100101 Firefox 60 0 Accept-Encoding  gzip  deflate Accept  text html application xhtml xml application xml q 0 9     q 0 8 Accept-Language  en-US en q 0 5 Connection  keep-alive Upgrade-Insecure-Requests  1   and voila  our php-emulated  browser GET request should now be indistinguishable from the real firefox GET request     this next part is just nitpicking  but if you look very closely  you ll see that the headers are stacked in the wrong order  firefox put the Accept-Encoding header in line 6  and our emulated GET request puts it in line 3   to fix this  we can manually put the Accept-Encoding header in the right line     lt  php  ch curl init  http   127 0 0 1 9999    curl setopt array  ch array          CURLOPT USERAGENT  gt  Mozilla 5 0  Windows NT 6 1  Win64  x64  rv 60 0  Gecko 20100101 Firefox 60 0           CURLOPT ENCODING  gt  gzip  deflate           CURLOPT HTTPHEADER  gt array                   Accept  text html application xhtml xml application xml q 0 9     q 0 8                    Accept-Language  en-US en q 0 5                    Accept-Encoding  gzip  deflate                    Connection  keep-alive                    Upgrade-Insecure-Requests  1                  curl exec  ch     running that  our netcat server gets      nc -l 9999 GET   HTTP 1 1 Host  127 0 0 1 9999 User-Agent  Mozilla 5 0  Windows NT 6 1  Win64  x64  rv 60 0  Gecko 20100101 Firefox 60 0 Accept  text html application xhtml xml application xml q 0 9     q 0 8 Accept-Language  en-US en q 0 5 Accept-Encoding  gzip  deflate Connection  keep-alive Upgrade-Insecure-Requests  1   problem solved  now the headers is even in the correct order  and the request seems to be COMPLETELY INDISTINGUISHABLE from the real firefox request     i don t actually recommend this last step  it s a maintenance burden to keep CURLOPT ENCODING in sync with the custom Accept-Encoding header  and i ve never experienced a situation where the order of the headers are significant

[php] How can I emulate a get request exactly like a web browser?

Examples related to php

Examples related to curl