[linux] CURL to access a page that requires a login from a different page

I have 2 pages: xyz.com/a and xyz.com/b. I can only access xyz.com/b if and only if I login to xyz.com/a first. If accessing xyz.com/b without going through the other, I simply get access denied (no redirect to login) via the browser. Once I login at xyz.com/a, I can access the other.

My problem is doing this using the curl command. I can login successfully to xyz.com/a using curl, but then try xyx.com/b and I get access denied.

I use the following:

curl --user user:pass https://xyz.com/a  #works ok
curl https://xyz.com/b #doesn't work

I've tried using the second line with & without the user/password part and still doesn't work. Both pages uses the same CA, so that's not a problem. Any suggestions? Thanks

This question is related to linux bash cookies curl http-authentication

The answer is


After some googling I found this:

curl -c cookie.txt -d "LoginName=someuser" -d "password=somepass" https://oursite/a
curl -b cookie.txt https://oursite/b

No idea if it works, but it might lead you in the right direction.


My answer is a mod of some prior answers from @JoeMills and @user.

  1. Get a cURL command to log into server:

    • Load login page for website and open Network pane of Developer Tools
      • In firefox, right click page, choose 'Inspect Element (Q)' and click on Network tab
    • Go to login form, enter username, password and log in
    • After you have logged in, go back to Network pane and scroll to the top to find the POST entry. Right click and choose Copy -> Copy as CURL
    • Paste this to a text editor and try this in command prompt to see if it works
      • Its possible that some sites have hardening that will block this type of login spoofing that would require more steps below to bypass.
  2. Modify cURL command to be able to save session cookie after login

    • Remove the entry -H 'Cookie: <somestuff>'
    • Add after curl at beginning -c login_cookie.txt
    • Try running this updated curl command and you should get a new file 'login_cookie.txt' in the same folder
  3. Call a new web page using this new cookie that requires you to be logged in

    • curl -b login_cookie.txt <url_that_requires_log_in>

I have tried this on Ubuntu 20.04 and it works like a charm.


Also you might want to log in via browser and get the command with all headers including cookies:

Open the Network tab of Developer Tools, log in, navigate to the needed page, use "Copy as cURL".

screenshot


Examples related to linux

grep's at sign caught as whitespace How to prevent Google Colab from disconnecting? "E: Unable to locate package python-pip" on Ubuntu 18.04 How to upgrade Python version to 3.7? Install Qt on Ubuntu Get first line of a shell command's output Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running? Run bash command on jenkins pipeline How to uninstall an older PHP version from centOS7 How to update-alternatives to Python 3 without breaking apt?

Examples related to bash

Comparing a variable with a string python not working when redirecting from bash script Zipping a file in bash fails How do I prevent Conda from activating the base environment by default? Get first line of a shell command's output Fixing a systemd service 203/EXEC failure (no such file or directory) /bin/sh: apt-get: not found VSCode Change Default Terminal Run bash command on jenkins pipeline How to check if the docker engine and a docker container are running? How to switch Python versions in Terminal?

Examples related to cookies

SameSite warning Chrome 77 How to fix "set SameSite cookie to none" warning? Set cookies for cross origin requests Make Axios send cookies in its requests automatically How can I set a cookie in react? Fetch API with Cookie How to use cookies in Python Requests How to set cookies in laravel 5 independently inside controller Where does Chrome store cookies? Sending cookies with postman

Examples related to curl

What is the incentive for curl to release the library for free? curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number Converting a POSTMAN request to Curl git clone error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054 How to post raw body data with curl? Curl : connection refused How to use the curl command in PowerShell? Curl to return http status code along with the response How to install php-curl in Ubuntu 16.04 curl: (35) SSL connect error

Examples related to http-authentication

Basic HTTP authentication with Node and Express 4 CURL to access a page that requires a login from a different page Authentication issues with WWW-Authenticate: Negotiate How to set the authorization header using curl How to do HTTP authentication in android?