Basic http file downloading and saving to disk in python

Question

I m new to Python and I ve been going through the Q amp A on this site  for an answer to my question  However  I m a beginner and I find it difficult to understand some of the solutions  I need a very basic solution   Could someone please explain a simple solution to  Downloading a file through http  and  Saving it to disk  in Windows   to me   I m not sure how to use shutil and os modules  either   The file I want to download is under 500 MB and is an  gz archive file If someone can explain how to extract the archive and utilise the files in it also  that would be great   Here s a partial solution  that I wrote from various answers combined   import requests import os import shutil  global dump  def download file        global dump     url    http   randomsite com file gz      file   requests get url  stream True      dump   file raw  def save file        global dump     location   os path abspath  D  folder file gz       with open  file gz    wb   as location          shutil copyfileobj dump  location      del dump   Could someone point out errors  beginner level  and explain any easier methods to do this   Thanks

User · Answer

For Python3  URLopener is deprecated  And when used you will get error as below      url opener   urllib URLopener   AttributeError  module  urllib  has no   attribute  URLopener    So  try   import urllib request  urllib request urlretrieve url  filename

User · Answer

For text files  you can use  import requests  url    https   WEBSITE com  req   requests get url  path    quot C   YOUR  FILE html quot   with open path   wb   as f      f write req content

User · Answer

Exotic Windows Solution  import subprocess  subprocess run  powershell Invoke-WebRequest    -OutFile     format your url  filename   shell True

User · Answer

As mentioned here   import urllib urllib urlretrieve   http   randomsite com file gz    file gz     EDIT  If you still want to use requests  take a look at this question or this one

User · Answer

Another clean way to save the file is this   import csv import urllib  urllib retrieve  your url goes here     output csv

User · Answer

Four methods using wget  urllib and request      usr bin python import requests from StringIO import StringIO from PIL import Image import profile as profile import urllib import wget   url    https   tinypng com images social website jpg   def testRequest        image name    test1 jpg      r   requests get url  stream True      with open image name   wb   as f          for chunk in r iter content                f write chunk   def testRequest2        image name    test2 jpg      r   requests get url      i   Image open StringIO r content       i save image name   def testUrllib        image name    test3 jpg      testfile   urllib URLopener       testfile retrieve url  image name   def testwget        image name    test4 jpg      wget download url  image name   if   name         main         profile run  testRequest         profile run  testRequest2         profile run  testUrllib         profile run  testwget       testRequest  - 4469882 function calls  4469842 primitive calls  in 20 236 seconds  testRequest2 - 8580 function calls  8574 primitive calls  in 0 072 seconds  testUrllib   - 3810 function calls  3775 primitive calls  in 0 036 seconds  testwget     - 3489 function calls in 0 020 seconds

User · Answer

I started down this path because ESXi s wget is not compiled with SSL and I wanted to download an OVA from a vendor s website directly onto the ESXi host which is on the other side of the world   I had to disable the firewall lazy  enable https out by editing the rules proper   created the python script   import ssl import shutil import tempfile import urllib request context   ssl  create unverified context    dlurl  https   somesite path whatever  with urllib request urlopen durl  context context  as response      with open  file ova    wb   as tmp file          shutil copyfileobj response  tmp file    ESXi libraries are kind of paired down but the open source weasel installer seemed to use urllib for https    so it inspired me to go down this path

User · Answer

I use wget   Simple and good library if you want to example   import wget  file url    http   johndoe com download zip   file name   wget download file url    wget module support python 2 and python 3 versions

User · Answer

A clean way to download a file is   import urllib  testfile   urllib URLopener   testfile retrieve  http   randomsite com file gz    file gz     This downloads a file from a website and names it file gz  This is one of my favorite solutions  from Downloading a picture via urllib and python   This example uses the urllib library  and it  will directly retrieve the file form a source

User · Answer

import urllib urllib request urlretrieve  quot https   raw githubusercontent com dnishimoto python-deep-learning master list 20iterators 20and 20generators ipynb quot    quot test ipynb quot    downloads a single raw juypter notebook to file

[python] Basic http file downloading and saving to disk in python?

Examples related to python

Examples related to file

Examples related to download

Examples related to save