Get final URL after curl is redirected

Question

I need to get the final URL after a page redirect preferably with curl or wget   For example http   google com may redirect to http   www google com   The contents are easy to get ex  curl --max-redirs 10 http   google com -L   but I m only interested in the final url  in the former case http   www google com    Is there any way of doing this by using only Linux built-in tools   command line only

User · Answer

Thanks  that helped me  I made some improvements and wrapped that in a helper script  finalurl       bin bash curl  1 -s -L -I -o  dev null -w    url effective      -o output to  dev null -I don t actually download  just discover the final URL -s silent mode  no progressbars   This made it possible to call the command from other scripts like this   echo  finalurl http   someurl

User · Answer

Can you try with it      bin bash  LOCATION  curl -I  http   your-domain com url redirect r something amp a values-VALUES FILES amp e zip    perl -n -e    Location          amp  amp  print   1 n     echo   LOCATION    Note  when you execute the command curl -I http   your-domain com have to use single quotes in the command like  curl -I  http   your-domain com

User · Answer

The parameters -L  --location  and -I  --head  still doing unnecessary HEAD-request to the location-url  If you are sure that you will have no more than one redirect  it is better to disable follow location and use a curl-variable   redirect url   This code do only one HEAD-request to the specified URL and takes redirect url from location-header  curl --head --silent --write-out  quot   redirect url  n quot  --output  dev null  quot https    quot  quot goo gl QeJeQ4 quot    Speed test all videos link txt - 50 links of goo gl bit ly which redirect to youtube 1  With follow location time while read -r line  do     curl -kIsL -w  quot   url effective  n quot  -o  dev null   line done  lt  all videos link txt  Results  real    1m40 832s user    0m9 266s sys     0m15 375s  2  Without follow location time while read -r line  do     curl -kIs -w  quot   redirect url  n quot  -o  dev null   line done  lt  all videos link txt  Results  real    0m51 037s user    0m5 297s sys     0m8 094s

User · Answer

You can do this with wget usually   wget --content-disposition  url  additionally if you add -O  dev null you will not be actually saving the file    wget -O  dev null --content-disposition example com

User · Answer

as another option     curl -i http   google com HTTP 1 1 301 Moved Permanently Location  http   www google com  Content-Type  text html  charset UTF-8 Date  Sat  19 Jun 2010 04 15 10 GMT Expires  Mon  19 Jul 2010 04 15 10 GMT Cache-Control  public  max-age 2592000 Server  gws Content-Length  219 X-XSS-Protection  1  mode block   lt HTML gt  lt HEAD gt  lt meta http-equiv  content-type  content  text html charset utf-8  gt   lt TITLE gt 301 Moved lt  TITLE gt  lt  HEAD gt  lt BODY gt   lt H1 gt 301 Moved lt  H1 gt  The document has moved  lt A HREF  http   www google com   gt here lt  A gt    lt  BODY gt  lt  HTML gt    But it doesn t go past the first one

User · Answer

This would work    curl -I somesite com   perl -n -e    Location          amp  amp  print   1 n

User · Answer

Thank you  I ended up implementing your suggestions  curl -i   grep  curl -i http   google com -L   egrep -A 10  301 Moved Permanently 302 Found    grep  Location    awk -F       print  2     tail -1   Returns blank if the website doesn t redirect  but that s good enough for me as it works on consecutive redirections   Could be buggy  but at a glance it works ok

User · Answer

curl can only follow http redirects  To also follow meta refresh directives and javascript redirects  you need a full-blown browser like headless chrome     bin bash real url          printf  location href nquit n          chromium-browser --headless --disable-gpu --disable-software-rasterizer       --disable-dev-shm-usage --no-sandbox --repl  quot    quot  2 gt   dev null         tr -d   gt  gt  gt      jq -r   result value     If you don t have chrome installed  you can use it from a docker container     bin bash real url          printf  location href nquit n          docker run -i --rm --user  quot   id -u  quot  USER quot   quot  --volume  quot   pwd  quot   usr src app       zenika alpine-chrome --no-sandbox --repl  quot    quot  2 gt   dev null         tr -d   gt  gt  gt      jq -r   result value     Like so    real url http   dx doi org 10 1016 j pgeola 2020 06 005  https   www sciencedirect com science article abs pii S0016787820300638 via 3Dihub

User · Answer

You could use grep  doesn t wget tell you where it s redirecting too  Just grep that out

User · Answer

curl s -w option and the sub variable url effective is what you are looking for  Something like curl -Ls -o  dev null -w   url effective  http   google com  More info  -L         Follow redirects -s         Silent mode  Don t output anything -o FILE    Write output to  lt file gt  instead of stdout -w FORMAT  What to output after completion  More You might want to add -I  that is an uppercase i  as well  which will make the command not download any  quot body quot   but it then also uses the HEAD method  which is not what the question included and risk changing what the server does  Sometimes servers don t respond well to HEAD even when they respond fine to GET

User · Answer

I m not sure how to do it with curl  but libwww-perl installs the GET alias     GET -S -d -e http   google com GET http   google com -- gt  301 Moved Permanently GET http   www google com  -- gt  302 Found GET http   www google ca  -- gt  200 OK Cache-Control  private  max-age 0 Connection  close Date  Sat  19 Jun 2010 04 11 01 GMT Server  gws Content-Type  text html  charset ISO-8859-1 Expires  -1 Client-Date  Sat  19 Jun 2010 04 11 01 GMT Client-Peer  74 125 155 105 80 Client-Response-Num  1 Set-Cookie  PREF ID a1925ca9f8af11b9 TM 1276920661 LM 1276920661 S ULFrHqOiFDDzDVFB  expires Mon  18-Jun-2012 04 11 01 GMT  path    domain  google ca Title  Google X-XSS-Protection  1  mode block

[linux] Get final URL after curl is redirected

Examples related to linux

Examples related to redirect

Examples related to curl

Examples related to wget