I had the same problem. My solution was to do the login via Chrome and save the cookies data to a text file. This is easily done with this Chrome extension: Chrome cookie.txt export extension.
When you get the cookies data, there is also an example on how to use them with wget. A simple copy-paste command line is provided to you.
If they're using basic authentication:
wget http://username:[email protected]/page.html
If they're using POSTed form data, you'll need to use something like cURL instead.
I directly gave cookies of an existing connection to wget
with --no-cookies
and the Cookie HTTP request header. In my case it was a Moodle university login where logging in looks more complex (using multiple requests with a login ticket). I added --post-data
because it was a POST
request.
For example, get all Moodle users list:
wget --no-cookies --header "Cookie: <name>=<value>" --post-data 'tab=search&name=+&personsubmit=Rechercher&keywords=&keywordsoption=allmine' https://moodle.unistra.fr/message/index.php
You can install this plugin in Firefox: https://addons.mozilla.org/en-US/firefox/addon/cliget/?src=cb-dl-toprated Start downloading what you want and click on the plugin. It gives you the whole command either for wget or curl to download the file on the serer. Very easy!
You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2'
works just fine. Note: you can also pass a file name to wget with the POST data in the file.
Example to download with wget on server a big file link that can be obtained in your browser.
In example using Google Chrome.
Login where you need, and press download. Go to download and copy your link.
Then open DevTools on a page where you where login, go to Console and get your cookies, by entering document.cookie
Now, go to server and download your file: wget --header "Cookie: <YOUR_COOKIE_OUTPUT_FROM_CONSOLE>" <YOUR_DOWNLOAD_LINK>
I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:
wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'
(This is one line, though it likely wraps on your browser)
If you want the output saved to a file, change -O -
to -O /some/file/name.ext
I use this chrome extension. It'll give you the wget command for any download link you open.
Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work
When you want to use wget to download some file from a site which requires login, you just need a cookie file. In order to generate the cookie file, I choose lynx. lynx is a text web browser. First you need a configure file for lynx to save cookie. Create a file lynx.cfg. Write these configuration into the file.
SET_COOKIES:TRUE
ACCEPT_ALL_COOKIES:TRUE
PERSISTENT_COOKIES:TRUE
COOKIE_FILE:cookie.file
Then start lynx with this command:
lynx -cfg=lynx.cfg http://the.site.com/login
After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout. The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.
Then wget can download file from the site with this command.
wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz
You can log in via browser and copy the needed headers afterwards:
Use "Copy as cURL" in the Network tab of browser developer tools and replace curl's flag -H
with wget's --header
(and also --data
with --post-data
if needed).
Source: Stackoverflow.com