What's the best way to download a python package and it's dependencies from pypi for offline installation on another machine? Is there any easy way to do this with pip or easy_install? I'm trying to install the requests library on a FreeBSD box that is not connected to the internet.
This question is related to
python
pip
freebsd
easy-install
python-requests
For Pip 8.1.2 you can use pip download -r requ.txt
to download packages to your local machine.
Download the tarball, transfer it to your FreeBSD machine and extract it, afterwards run python setup.py install
and you're done!
EDIT: Just to add on that, you can also install the tarballs with pip now.
Download the wheel file (for example dlb-0.5.0-py3-none-any.whl) from Pypi and
pip install dlb-0.5.0-py3-none-any.whl
The pip download
command lets you download packages without installing them:
pip download -r requirements.txt
(In previous versions of pip, this was spelled pip install --download -r requirements.txt
.)
Then you can use pip install --no-index --find-links /path/to/download/dir/ -r requirements.txt
to install those downloaded sdists, without accessing the network.
If you want install python libs and their dependencies offline, finish following these steps on a machine with the same os, network connected, and python installed:
1) Create a requirements.txt
file with similar content (Note - these are the libraries you wish to download):
Flask==0.12
requests>=2.7.0
scikit-learn==0.19.1
numpy==1.14.3
pandas==0.22.0
One option for creating the requirements file is to use pip freeze > requirements.txt
. This will list all libraries in your environment. Then you can go in to requirements.txt
and remove un-needed ones.
2) Execute command mkdir wheelhouse && pip download -r requirements.txt -d wheelhouse
to download libs and their dependencies to directory wheelhouse
3) Copy requirements.txt into wheelhouse
directory
4) Archive wheelhouse into wheelhouse.tar.gz
with tar -zcf wheelhouse.tar.gz wheelhouse
Then upload wheelhouse.tar.gz
to your target machine:
1) Execute tar -zxf wheelhouse.tar.gz
to extract the files
2) Execute pip install -r wheelhouse/requirements.txt --no-index --find-links wheelhouse
to install the libs and their dependencies
I had a similar problem. And i had to make it install the same way, we do from pypi.
I did the following things:
Make a directory to store all the packages in the machine that have internet access.
mkdir -p /path/to/packages/
Download all the packages to the path
Edit: You can also try:
python3 -m pip wheel --no-cache-dir -r requirements.txt -w /path/to/packages
pip download -r requirements.txt -d /path/to/packages
Eg:- ls /root/wheelhouse/ # **/root/wheelhouse** is my **/path/to/packages/**
total 4524
-rw-r--r--. 1 root root 16667 May 23 2017 incremental-17.5.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 34713 Sep 1 10:21 attrs-18.2.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 3088398 Oct 15 14:41 Twisted-18.9.0.tar.bz2
-rw-r--r--. 1 root root 133356 Jan 28 15:58 chardet-3.0.4-py2.py3-none-any.whl
-rw-r--r--. 1 root root 154154 Jan 28 15:58 certifi-2018.11.29-py2.py3-none-any.whl
-rw-r--r--. 1 root root 57987 Jan 28 15:58 requests-2.21.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 58594 Jan 28 15:58 idna-2.8-py2.py3-none-any.whl
-rw-r--r--. 1 root root 118086 Jan 28 15:59 urllib3-1.24.1-py2.py3-none-any.whl
-rw-r--r--. 1 root root 47229 Jan 28 15:59 tqdm-4.30.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 7922 Jan 28 16:13 constantly-15.1.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 164706 Jan 28 16:14 zope.interface-4.6.0-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r--. 1 root root 573841 Jan 28 16:14 setuptools-40.7.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 37638 Jan 28 16:15 Automat-0.7.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 37905 Jan 28 16:15 hyperlink-18.0.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 52311 Jan 28 16:15 PyHamcrest-1.9.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 10586 Jan 28 16:15 six-1.12.0-py2.py3-none-any.whl
Tar the packages directory and copy it to the Machine that doesn't have internet access. Then do,
cd /path/to/packages/
tar -cvzf packages.tar.gz . # not the . (dot) at the end
Copy the packages.tar.gz into the destination machine that doesn't have internet access.
In the machine that doesn't have internet access, do the following (Assuming you copied the tarred packages to /path/to/package/ in the current machine)
cd /path/to/packages/
tar -xvzf packages.tar.gz
mkdir -p $HOME/.config/pip/
vi $HOME/.config/pip/pip.conf
and paste the following content inside and save it.
[global]
timeout = 10
find-links = file:///path/to/package/
no-cache-dir = true
no-index = true
Finally, i suggest you use, some form of virtualenv
for installing the packages.
virtualenv -p python2 venv # use python3, if you are on python3
source ./venv/bin/activate
pip install <package>
You should be able to download all the modules that are in the directory /path/to/package/.
Note: I only did this, because i couldn't add options or change the way we install the modules. Otherwise i'd have done
pip install --no-index --find-links /path/to/download/dir/ -r requirements.txt
offline python. for doing this I use virtualenv (isolated Python environment)
1) install virtualenv online with pip:
pip install virtualenv --user
or offline with whl: go to this link , download last version (.whl or tar.gz) and install that with this command:
pip install virtualenv-15.1.0-py2.py3-none-any.whl --user
by using --user
you don't need to use sudo pip…
.
2) use virtualenv
on online machine select a directory with terminal cd
and run this code:
python -m virtualenv myenv
cd myenv
source bin/activate
pip install Flask
after installing all the packages, you have to generate a requirements.txt
so while your virtualenv is active, write
pip freeze > requirements.txt
open a new terminal and create another env like myenv2
.
python -m virtualenv myenv2
cd myenv2
source bin/activate
cd -
ls
now you can go to your offline folder where your requirements.txt
and tranferred_packages
folder are in there. download the packages with following code and put all of them to tranferred_packages
folder.
pip download -r requirements.txt
take your offline folder to offline computer and then
python -m virtualenv myenv2
cd myenv2
source bin/activate
cd -
cd offline
pip install --no-index --find-links="./tranferred_packages" -r requirements.txt
what is in the folder offline [requirements.txt , tranferred_packages {Flask-0.10.1.tar.gz, ...}]
check list of your package
pip list
note: as we are in 2017 it is better to use python 3. you can create python 3 virtualenv with this command.
virtualenv -p python3 envname
As a continues to @chaokunyang answer, I want to put here the script I write that does the work:
"requirements.txt" file
docker==4.4.0
Packer side: file name: "create-offline-python3.6-dependencies-repository.sh"
#!/usr/bin/env bash
# This script follows the steps described in this link:
# https://stackoverflow.com/a/51646354/8808983
LIBRARIES_DIR="python3.6-wheelhouse"
if [ -d ${LIBRARIES_DIR} ]; then
rm -rf ${LIBRARIES_DIR}/*
else
mkdir ${LIBRARIES_DIR}
fi
pip download -r requirements.txt -d ${LIBRARIES_DIR}
files_to_add=("requirements.txt" "install-python-libraries-offline.sh")
for file in "${files_to_add[@]}"; do
echo "Adding file ${file}"
cp "$file" ${LIBRARIES_DIR}
done
tar -cf ${LIBRARIES_DIR}.tar ${LIBRARIES_DIR}
Installer side: file name: "install-python-libraries-offline.sh"
#!/usr/bin/env bash
# This script follows the steps described in this link:
# https://stackoverflow.com/a/51646354/8808983
# This file should run during the installation process from inside the libraries directory, after it was untared:
# pythonX-wheelhouse.tar -> untar -> pythonX-wheelhouse
# |
# |--requirements.txt
# |--install-python-libraries-offline.sh
pip3 install -r requirements.txt --no-index --find-links .
Let me go through the process step by step:
$ mkdir packages
$ cd packages
open up a command prompt or shell and execute the following command:
Suppose the package you want is tensorflow
$ pip download tensorflow
Now, on the target computer, copy the packages
folder and apply the following command
$ cd packages
$ pip install 'tensorflow-xyz.whl' --no-index --find-links '.'
Note that the tensorflow-xyz.whl
must be replaced by the original name of the required package.
Using wheel
compiled packages.
bundle up:
$ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX)
$ pip wheel -r requirements.txt --wheel-dir=$tempdir
$ cwd=`pwd`
$ (cd "$tempdir"; tar -cjvf "$cwd/bundled.tar.bz2" *)
copy tarball and install:
$ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX)
$ (cd $tempdir; tar -xvf /path/to/bundled.tar.bz2)
$ pip install --force-reinstall --ignore-installed --upgrade --no-index --no-deps $tempdir/*
Note wheel
binary packages are not across machines.
More ref. here: https://pip.pypa.io/en/stable/user_guide/#installation-bundles
Source: Stackoverflow.com