How to get the contents of a webpage in a shell variable

Question

In Linux how can I fetch an URL and get its contents in a variable in shell script

User · Accepted Answer

You can use wget command to download the page and read it into a variable as   content   wget google com -q -O -  echo  content   We use the -O option of wget which allows us to specify the name of the file into which wget dumps the page contents  We specify - to get the dump onto standard output and collect that into the variable content  You can add the -q quiet option to turn off s wget output    You can use the curl command for this aswell as   content   curl -L google com  echo  content   We need to use the -L option as the page we are requesting might have moved  In which case we need to get the page from the new location  The -L or --location option helps us with this

User · Answer

content  wget -O -  url

User · Answer

You can use curl or wget to retrieve the raw data  or you can use w3m -dump to have a nice text representation of a web page     foo   w3m -dump http   www example com    echo  foo You have reached this web page by typing  example com    example net   example org  or  example edu  into your web browser  These domain names are reserved for use in documentation and are not available for registration  See RFC 2606  Section 3

User · Answer

There are many ways to get a page from the command line    but it also depends if you want the code source or the page itself   If you need the code source   with curl   curl  url   with wget   wget -O -  url   but if you want to get what you can see with a browser  lynx can be useful   lynx -dump  url   I think you can find so many solutions for this little problem  maybe you should read all man pages for those commands  And don t forget to replace  url by your URL     Good luck

User · Answer

If you have LWP installed  it provides a binary simply named  GET       GET http   example com  lt  DOCTYPE HTML PUBLIC  -  W3C  DTD HTML 4 01 Transitional  EN  gt   lt HTML gt   lt HEAD gt     lt META http-equiv  Content-Type  content  text html  charset utf-8  gt     lt TITLE gt Example Web Page lt  TITLE gt   lt  HEAD gt    lt body gt     lt p gt You have reached this web page by typing  amp quot example com amp quot    amp quot example net amp quot   amp quot example org amp quot   or  amp quot example edu amp quot  into your web browser  lt  p gt   lt p gt These domain names are reserved for use in documentation and are not available    for registration  See  lt a href  http   www rfc-editor org rfc rfc2606 txt  gt RFC    2606 lt  a gt   Section 3  lt  p gt   lt  BODY gt   lt  HTML gt    wget -O-  curl  and lynx -source behave similarly

User · Answer

There is the wget command or the curl   You can now use the file you downloaded with wget  Or you can handle a stream with curl     Resources     linux die - man wget linux die - man curl

[linux] How to get the contents of a webpage in a shell variable?

Examples related to linux

Examples related to bash

Examples related to shell

Examples related to wget