Is it possible to use pip to install a package from a private GitHub repository

Question

I am trying to install a Python package from a private GitHub repository  For a public repository  I can issue the following command which works fine   pip install git git   github com django django git   However  if I try this for a private repository   pip install git git   github com echweb echweb-utils git   I get the following output   Downloading unpacking git git   github com echweb echweb-utils git Cloning Git repository git   github com echweb echweb-utils git to  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-VRsIoo-build Complete output from command  usr local bin git clone git   github com echweb echweb-utils git  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-VRsIoo-build  fatal  The remote end hung up unexpectedly  Cloning into  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-VRsIoo-build     ---------------------------------------- Command  usr local bin git clone git   github com echweb echweb-utils git  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-VRsIoo-build failed with error code 128   I guess this is because I am trying to access a private repository without providing any authentication  I therefore tried to use Git   ssh hoping that pip would use my SSH public key to authenticate   pip install git ssh   github com echweb echweb-utils git   This gives the following output   Downloading unpacking git ssh   github com echweb echweb-utils git Cloning Git repository ssh   github com echweb echweb-utils git to  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-DQB8s4-build Complete output from command  usr local bin git clone ssh   github com echweb echweb-utils git  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-DQB8s4-build  Cloning into  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-DQB8s4-build     Permission denied  publickey    fatal  The remote end hung up unexpectedly  ---------------------------------------- Command  usr local bin git clone ssh   github com echweb echweb-utils git  var folders cB cB85g9P7HM4jcPn7nrvWRU   TI -Tmp- pip-DQB8s4-build failed with error code 128   Is what I am trying to achieve even possible  If so  how can I do it

User · Answer

As an additional technique  if you have the private repository cloned locally  you can do   pip install git file   c  repo directory   More modernly  you can just do this  and the -e will mean you don t have to commit changes before they re reflected    pip install -e C  repo directory

User · Answer

If you want to install dependencies from a requirements file within a CI server or alike  you can do this   git config --global credential helper  cache  echo  protocol https host example com username   GIT USER  password   GIT PASS      git credential approve pip install -r requirements txt   In my case  I used GIT USER gitlab-ci-token and GIT PASS   CI JOB TOKEN    This method has a clear advantage  You have a single requirements file which contains all of your dependencies

User · Answer

I figured out a way to automagically  pip install  a GitLab private repository that requires no password prompt  This approach uses GitLab  Deploy Keys  and an SSH configuration file  so you can deploy using keys other than your personal SSH keys  in my case  for use by a  bot   Perhaps someone kind soul can verify using GitHub   Create a New SSH key   ssh-keygen -t rsa -C  GitLab Robot Deploy Key    The file should show up as    ssh GitLab Robot Deploy Key and    ssh GitLab Robot Deploy Key pub   Copy and paste the contents of the    ssh GitLab Robot Deploy Key pub file into the GitLab  Deploy Keys  dialog   Test the New Deploy Key  The following command tells SSH to use your new deploy key to set up the connection  On success  you should get the message   Welcome to GitLab  UserName    ssh -T -i    ssh GitLab Robot Deploy Key git gitlab mycorp com   Create the SSH Configuration File  Next  use an editor to create a    ssh config file  Add the following contents  The  Host  value can be anything you want  just remember it  because you ll be using it later   The HostName is the URL to your GitLab instance  The IdentifyFile is path to the SSH key file you created in the first step   Host GitLab   HostName gitlab mycorp com   IdentityFile    ssh GitLab Robot Deploy Key   Point SSH to the Configuration file  oxyum gave us the recipe for using pip with SSH   pip install git ssh   git gitlab mycorp com my name my repo git   We just need to modify it a bit to make SSH use our new Deploy Key  We do that by pointing SSH to the Host entry in the SSH configuration file  Just replace the  gitlab mycorp com  in the command to the host name we used in the SSH configuration file   pip install git ssh   git GitLab my name my repo git   The package should now install without any password prompt   Reference A  Reference B

User · Answer

You may try   pip install git git gitlab mycorp com my name my repo git   without ssh      That works for me

User · Answer

You can do it directly with the HTTPS URL like this   pip install git https   github com username repo git   This also works just appending that line in the requirements txt in a Django project  for instance

User · Answer

If you don t want to use SSH  you could add the username and password in the HTTPS URL   The code below assumes that you have a file called  pass  in the working directory that contains your password   export PASS   cat pass  pip install git https    lt username gt   PASS github com echweb echweb-utils git

User · Answer

The syntax for the requirements file is given here   https   pip pypa io en latest reference pip install html requirements-file-format  So for example  use   -e git http   github com rwillmer django-behave egg django-behave   if you want the source to stick around after installation   Or just  git http   github com rwillmer django-behave egg django-behave   if you just want it to be installed

User · Answer

Just copy the remote from the original git clone command  or from git remote -v   You will get something like this    Bitbucket  git ssh   git bitbucket org your account my pro git GitHub  git ssh   git github com your account my pro git   Next  you need to replace   with   next to the domain name   So install using   pip install git ssh   git bitbucket org your account my pro git

User · Answer

If you have your own library package on GitHub  GitLab  etc   you have to add a tag to commit with a concrete version of the library  for example  v2 0  and then you can install your package   pip install git ssh   link name repo git v2 0   This works for me  Other solutions haven t worked for me

User · Answer

My case was kind of more complicated than most of the ones described in the answers  I was the owner of two private repositories repo-A and repo-B in a Github organization and needed to pip install repo-A during the python unittests of repo-B  as a Github action  Steps I followed to solve this task   Created a Personal Access Token for my account  As for its permissions  I only needed to keep the default ones   i e  repo - Full control of private repositories  Created a repository secret under repo-B  pasted my Personal Access Token in there and named it PERSONAL ACCESS TOKEN  This was important because  unlike the solution proposed by Jamie  I didn t need to explicitly expose my precious raw Personal Access Token inside the github action  yml file  Finally  I pip installed the package from source via HTTPS as follows export PERSONAL ACCESS TOKEN     secrets PERSONAL ACCESS TOKEN     amp  amp  pip install git https     PERSONAL ACCESS TOKEN  github com MY ORG NAME MY REPO NAME git

User · Answer

If you need to do this in  say  a command line one-liner  it s also possible  I was able to do this for deployment on Google Colab   Create a Personal Access Token  https   docs github com en github authenticating-to-github creating-a-personal-access-token Run  pip install git https    lt PERSONAL ACCESS TOKEN gt  github com  lt USERNAME gt   lt REPOSITORY gt  git

User · Answer

When I was installing from GitHub I was able to use   pip install git ssh   git github com  lt username gt   lt projectname gt  git egg  lt eggname gt    But  since I had to run pip as sudo  the SSH keys were not working with GitHub any more  and  git clone  failed on  Permission denied  publickey    Using git https allowed me to run the command as sudo  and have GitHub ask me for my user password   sudo pip install git https   github com  lt username gt   lt projectname gt  git egg  lt eggname gt

User · Answer

I found it much easier to use tokens than SSH keys  I couldn t find much good documentation on this  so I came across this solution mainly through trial and error  Further  installing from pip and setuptools have some subtle differences  but this way should work for both   GitHub don t  currently  as of August 2016  offer an easy way to get the zip   tarball of private repositories  So you need to point setuptools to tell setuptools that you re pointing to a Git repository   from setuptools import setup import os   Get the deploy key from https   help github com articles git-automation-with-oauth-tokens  github token   os environ  GITHUB TOKEN    setup                install requires  package       dependency links          git https    github token  github com user  package  git   version  egg  package -0           format github token github token  package package  version master              A couple of notes here    For private repositories  you need to authenticate with GitHub  the simplest way I found is to create an OAuth token  drop that into your environment  and then include it with the URL You need to include some version number  here is 0  at the end of the link  even if there s isn t any package on PyPI  This has to be a actual number  not a word  You need to preface with git  to tell setuptools it s to clone the repository  rather than pointing at a zip   tarball version can be a branch  a tag  or a commit hash You need to supply --process-dependency-links if installing from pip

User · Answer

You can use the git ssh URI scheme  but you must set a username  Notice the git  part in the URI  pip install git ssh   git github com echweb echweb-utils git  Also read about deploy keys  PS  In my installation  the  quot git ssh quot  URI scheme works only with  quot editable quot  requirements  pip install -e URI egg EggName  Remember  Change the   character that git remote -v prints to a   character before using the remote s address in the pip command    git remote -v origin  git github com echweb echweb-utils git  fetch                          change this to a     character  If you forget  you will get this error  ssh  Could not resolve hostname github com echweb           nodename nor servname provided  or not known

User · Answer

You can also install a private repository dependency via git https   github com     URL by providing login credentials  login and password  or deploy token  for curl with the  netrc file   echo  machine github com login ei-grad password mypasswordshouldbehere   gt     netrc pip install  git https   github com ei-grad my private repo git egg my private repo

User · Answer

It also works with Bitbucket   pip install git ssh   git bitbucket org username projectname git   Pip will use your SSH keys in this case

User · Answer

oxyum s solution is OK for this answer  I just want to point out that you need to be careful if you are installing using sudo as the keys must be stored for root too  for example   root  ssh    Then you can type  sudo pip install git ssh   git github com echweb echweb-utils git

User · Answer

Here s a quick method that worked for me  Simply fork the repo and install it from your own GitHub account with   pip install git https   github com yourName repoName

[python] Is it possible to use pip to install a package from a private GitHub repository?

Examples related to python

Examples related to git

Examples related to github

Examples related to pip