Send file using POST from a Python script

Question

Is there a way to send a file using POST from a Python script

User · Answer

Chris Atlee s poster library works really well for this  particularly the convenience function poster encode multipart encode     As a bonus  it supports streaming of large files without loading an entire file into memory  See also Python issue 3244

User · Answer

From  https   requests readthedocs io en latest user quickstart  post-a-multipart-encoded-file  Requests makes it very simple to upload Multipart-encoded files   with open  report xls    rb   as f      r   requests post  http   httpbin org post   files   report xls   f     That s it  I m not joking - this is one line of code  The file was sent  Let s check    gt  gt  gt  r text      origin    179 13 100 4      files          report xls     lt censored   binary   data gt           form          url    http   httpbin org post      args          headers          Content-Length    3196        Accept-Encoding    identity  deflate  compress  gzip        Accept               User-Agent    python-requests 0 8 0        Host    httpbin org 80        Content-Type    multipart form-data  boundary 127 0 0 1 502 21746 1321131593 786 1          data

User · Answer

Yes  You d use the urllib2 module  and encode using the multipart form-data content type  Here is some sample code to get you started -- it s a bit more than just file uploading  but you should be able to read through it and see how it works   user agent    image uploader  default message    Image  current of  total   import logging import os from os path import abspath  isabs  isdir  isfile  join import random import string import sys import mimetypes import urllib2 import httplib import time import re  def random string  length       return    join  random choice  string letters  for ii in range  length   1    def encode multipart data  data  files       boundary   random string  30       def get content type  filename           return mimetypes guess type  filename  0  or  application octet-stream       def encode field  field name           return   --    boundary                   Content-Disposition  form-data  name   s     field name                      str  data  field name         def encode file  field name           filename   files  field name          return   --    boundary                   Content-Disposition  form-data  name   s   filename   s      field name  filename                    Content-Type   s    get content type filename                       open  filename   rb   read          lines          for name in data          lines extend  encode field  name       for name in files          lines extend  encode file  name       lines extend    -- s--    boundary           body     r n  join  lines       headers     content-type    multipart form-data  boundary     boundary                  content-length   str  len  body         return body  headers  def send post  url  data  files       req   urllib2 Request  url      connection   httplib HTTPConnection  req get host         connection request   POST   req get selector                              encode multipart data  data  files       response   connection getresponse        logging debug   response    s   response read         logging debug   Code   s  s   response status  response reason   def make upload file  server  thread  delay   15  message   None                        username   None  email   None  password   None        delay   max  int  delay or  0    15       def upload file  path  current  total           assert isabs  path          assert isfile  path           logging debug   Uploading  r to  r   path  server          message template   string Template  message or default message           data     MAX FILE SIZE    3145728                    sub                        mode    regist                    com   message template safe substitute  current   current  total   total                    resto   thread                   name   username or                      email   email or                      pwd   password or random string  20            files     upfile   path           send post  server  data  files           logging info   Uploaded  r   path          rand delay   random randint  delay  delay   5          logging debug   Sleeping for   2f seconds------------------------------ n n   rand delay          time sleep  rand delay       return upload file  def upload directory  path  upload file       assert isabs  path      assert isdir  path       matching filenames          file matcher   re compile  r      jpe g gif png     re IGNORECASE       for dirpath  dirnames  filenames in os walk  path           for name in filenames              file path   join  dirpath  name              logging debug   Testing file path  r   file path              if file matcher search  file path                   matching filenames append  file path              else                  logging info   Ignoring non-image file  r   path       total count   len  matching filenames      for index  file path in enumerate  matching filenames           upload file  file path  index   1  total count   def run upload  options  paths       upload file   make upload file    options       for arg in paths          path   abspath  arg          if isdir  path               upload directory  path  upload file          elif isfile  path               upload file  path          else              logging error   No such path   r    path       logging info   Done

User · Answer

You may also want to have a look at httplib2  with examples  I find using httplib2 is more concise than using the built-in HTTP modules

User · Answer

The only thing that stops you from using urlopen directly on a file object is the fact that the builtin file object lacks a len definition  A simple way is to create a subclass  which provides urlopen with the correct file   I have also modified the Content-Type header in the file below   import os import urllib2 class EnhancedFile file       def   init   self   args    keyws           file   init   self   args    keyws       def   len   self           return int os fstat self fileno    6    theFile   EnhancedFile  a xml    r   theUrl    http   example com abcde  theHeaders    Content-Type    text xml    theRequest   urllib2 Request theUrl  theFile  theHeaders   response   urllib2 urlopen theRequest   theFile close     for line in response      print line

User · Answer

def visit v2 device code  camera code       image1   MultipartParam from file  files     home yuzx 1 txt       image2   MultipartParam from file  files     home yuzx 2 txt       datagen  headers   multipart encode    device code   device code     position   3     person data   person data   image1  image2       print    join datagen      if server port    80          port str          else          port str      s     server port       url str    http       server ip   port str     adopen device visit v2      headers  nothing      nothing      request   urllib2 Request url str  datagen  headers      try          response   urllib2 urlopen request          resp   response read           print  http status     response code         result   json loads resp          print resp         return result     except urllib2 HTTPError  e          print  http status     e code         print e read

User · Answer

Looks like python requests does not handle extremely large multi-part files   The documentation recommends you look into requests-toolbelt   Here s the pertinent page from their documentation

User · Answer

The only thing that stops you from using urlopen directly on a file object is the fact that the builtin file object lacks a len definition  A simple way is to create a subclass  which provides urlopen with the correct file   I have also modified the Content-Type header in the file below   import os import urllib2 class EnhancedFile file       def   init   self   args    keyws           file   init   self   args    keyws       def   len   self           return int os fstat self fileno    6    theFile   EnhancedFile  a xml    r   theUrl    http   example com abcde  theHeaders    Content-Type    text xml    theRequest   urllib2 Request theUrl  theFile  theHeaders   response   urllib2 urlopen theRequest   theFile close     for line in response      print line

User · Answer

Chris Atlee s poster library works really well for this  particularly the convenience function poster encode multipart encode     As a bonus  it supports streaming of large files without loading an entire file into memory  See also Python issue 3244

User · Answer

From  https   requests readthedocs io en latest user quickstart  post-a-multipart-encoded-file  Requests makes it very simple to upload Multipart-encoded files   with open  report xls    rb   as f      r   requests post  http   httpbin org post   files   report xls   f     That s it  I m not joking - this is one line of code  The file was sent  Let s check    gt  gt  gt  r text      origin    179 13 100 4      files          report xls     lt censored   binary   data gt           form          url    http   httpbin org post      args          headers          Content-Length    3196        Accept-Encoding    identity  deflate  compress  gzip        Accept               User-Agent    python-requests 0 8 0        Host    httpbin org 80        Content-Type    multipart form-data  boundary 127 0 0 1 502 21746 1321131593 786 1          data

User · Answer

Yes  You d use the urllib2 module  and encode using the multipart form-data content type  Here is some sample code to get you started -- it s a bit more than just file uploading  but you should be able to read through it and see how it works   user agent    image uploader  default message    Image  current of  total   import logging import os from os path import abspath  isabs  isdir  isfile  join import random import string import sys import mimetypes import urllib2 import httplib import time import re  def random string  length       return    join  random choice  string letters  for ii in range  length   1    def encode multipart data  data  files       boundary   random string  30       def get content type  filename           return mimetypes guess type  filename  0  or  application octet-stream       def encode field  field name           return   --    boundary                   Content-Disposition  form-data  name   s     field name                      str  data  field name         def encode file  field name           filename   files  field name          return   --    boundary                   Content-Disposition  form-data  name   s   filename   s      field name  filename                    Content-Type   s    get content type filename                       open  filename   rb   read          lines          for name in data          lines extend  encode field  name       for name in files          lines extend  encode file  name       lines extend    -- s--    boundary           body     r n  join  lines       headers     content-type    multipart form-data  boundary     boundary                  content-length   str  len  body         return body  headers  def send post  url  data  files       req   urllib2 Request  url      connection   httplib HTTPConnection  req get host         connection request   POST   req get selector                              encode multipart data  data  files       response   connection getresponse        logging debug   response    s   response read         logging debug   Code   s  s   response status  response reason   def make upload file  server  thread  delay   15  message   None                        username   None  email   None  password   None        delay   max  int  delay or  0    15       def upload file  path  current  total           assert isabs  path          assert isfile  path           logging debug   Uploading  r to  r   path  server          message template   string Template  message or default message           data     MAX FILE SIZE    3145728                    sub                        mode    regist                    com   message template safe substitute  current   current  total   total                    resto   thread                   name   username or                      email   email or                      pwd   password or random string  20            files     upfile   path           send post  server  data  files           logging info   Uploaded  r   path          rand delay   random randint  delay  delay   5          logging debug   Sleeping for   2f seconds------------------------------ n n   rand delay          time sleep  rand delay       return upload file  def upload directory  path  upload file       assert isabs  path      assert isdir  path       matching filenames          file matcher   re compile  r      jpe g gif png     re IGNORECASE       for dirpath  dirnames  filenames in os walk  path           for name in filenames              file path   join  dirpath  name              logging debug   Testing file path  r   file path              if file matcher search  file path                   matching filenames append  file path              else                  logging info   Ignoring non-image file  r   path       total count   len  matching filenames      for index  file path in enumerate  matching filenames           upload file  file path  index   1  total count   def run upload  options  paths       upload file   make upload file    options       for arg in paths          path   abspath  arg          if isdir  path               upload directory  path  upload file          elif isfile  path               upload file  path          else              logging error   No such path   r    path       logging info   Done

User · Answer

You may also want to have a look at httplib2  with examples  I find using httplib2 is more concise than using the built-in HTTP modules

User · Answer

def visit v2 device code  camera code       image1   MultipartParam from file  files     home yuzx 1 txt       image2   MultipartParam from file  files     home yuzx 2 txt       datagen  headers   multipart encode    device code   device code     position   3     person data   person data   image1  image2       print    join datagen      if server port    80          port str          else          port str      s     server port       url str    http       server ip   port str     adopen device visit v2      headers  nothing      nothing      request   urllib2 Request url str  datagen  headers      try          response   urllib2 urlopen request          resp   response read           print  http status     response code         result   json loads resp          print resp         return result     except urllib2 HTTPError  e          print  http status     e code         print e read

User · Answer

You may also want to have a look at httplib2  with examples  I find using httplib2 is more concise than using the built-in HTTP modules

User · Answer

Looks like python requests does not handle extremely large multi-part files   The documentation recommends you look into requests-toolbelt   Here s the pertinent page from their documentation

User · Answer

You may also want to have a look at httplib2  with examples  I find using httplib2 is more concise than using the built-in HTTP modules

User · Answer

Yes  You d use the urllib2 module  and encode using the multipart form-data content type  Here is some sample code to get you started -- it s a bit more than just file uploading  but you should be able to read through it and see how it works   user agent    image uploader  default message    Image  current of  total   import logging import os from os path import abspath  isabs  isdir  isfile  join import random import string import sys import mimetypes import urllib2 import httplib import time import re  def random string  length       return    join  random choice  string letters  for ii in range  length   1    def encode multipart data  data  files       boundary   random string  30       def get content type  filename           return mimetypes guess type  filename  0  or  application octet-stream       def encode field  field name           return   --    boundary                   Content-Disposition  form-data  name   s     field name                      str  data  field name         def encode file  field name           filename   files  field name          return   --    boundary                   Content-Disposition  form-data  name   s   filename   s      field name  filename                    Content-Type   s    get content type filename                       open  filename   rb   read          lines          for name in data          lines extend  encode field  name       for name in files          lines extend  encode file  name       lines extend    -- s--    boundary           body     r n  join  lines       headers     content-type    multipart form-data  boundary     boundary                  content-length   str  len  body         return body  headers  def send post  url  data  files       req   urllib2 Request  url      connection   httplib HTTPConnection  req get host         connection request   POST   req get selector                              encode multipart data  data  files       response   connection getresponse        logging debug   response    s   response read         logging debug   Code   s  s   response status  response reason   def make upload file  server  thread  delay   15  message   None                        username   None  email   None  password   None        delay   max  int  delay or  0    15       def upload file  path  current  total           assert isabs  path          assert isfile  path           logging debug   Uploading  r to  r   path  server          message template   string Template  message or default message           data     MAX FILE SIZE    3145728                    sub                        mode    regist                    com   message template safe substitute  current   current  total   total                    resto   thread                   name   username or                      email   email or                      pwd   password or random string  20            files     upfile   path           send post  server  data  files           logging info   Uploaded  r   path          rand delay   random randint  delay  delay   5          logging debug   Sleeping for   2f seconds------------------------------ n n   rand delay          time sleep  rand delay       return upload file  def upload directory  path  upload file       assert isabs  path      assert isdir  path       matching filenames          file matcher   re compile  r      jpe g gif png     re IGNORECASE       for dirpath  dirnames  filenames in os walk  path           for name in filenames              file path   join  dirpath  name              logging debug   Testing file path  r   file path              if file matcher search  file path                   matching filenames append  file path              else                  logging info   Ignoring non-image file  r   path       total count   len  matching filenames      for index  file path in enumerate  matching filenames           upload file  file path  index   1  total count   def run upload  options  paths       upload file   make upload file    options       for arg in paths          path   abspath  arg          if isdir  path               upload directory  path  upload file          elif isfile  path               upload file  path          else              logging error   No such path   r    path       logging info   Done

User · Answer

I am trying to test django rest api and its working for me   def test upload file self           filename     Users Ranvijay tests test price matrix csv          data     file   open filename   rb            client   APIClient             client credentials HTTP AUTHORIZATION  Token     token key          response   client post reverse  price-matrix-csv    data  format  multipart            print response         self assertEqual response status code  status HTTP 200 OK

User · Answer

I am trying to test django rest api and its working for me   def test upload file self           filename     Users Ranvijay tests test price matrix csv          data     file   open filename   rb            client   APIClient             client credentials HTTP AUTHORIZATION  Token     token key          response   client post reverse  price-matrix-csv    data  format  multipart            print response         self assertEqual response status code  status HTTP 200 OK

User · Answer

Yes  You d use the urllib2 module  and encode using the multipart form-data content type  Here is some sample code to get you started -- it s a bit more than just file uploading  but you should be able to read through it and see how it works   user agent    image uploader  default message    Image  current of  total   import logging import os from os path import abspath  isabs  isdir  isfile  join import random import string import sys import mimetypes import urllib2 import httplib import time import re  def random string  length       return    join  random choice  string letters  for ii in range  length   1    def encode multipart data  data  files       boundary   random string  30       def get content type  filename           return mimetypes guess type  filename  0  or  application octet-stream       def encode field  field name           return   --    boundary                   Content-Disposition  form-data  name   s     field name                      str  data  field name         def encode file  field name           filename   files  field name          return   --    boundary                   Content-Disposition  form-data  name   s   filename   s      field name  filename                    Content-Type   s    get content type filename                       open  filename   rb   read          lines          for name in data          lines extend  encode field  name       for name in files          lines extend  encode file  name       lines extend    -- s--    boundary           body     r n  join  lines       headers     content-type    multipart form-data  boundary     boundary                  content-length   str  len  body         return body  headers  def send post  url  data  files       req   urllib2 Request  url      connection   httplib HTTPConnection  req get host         connection request   POST   req get selector                              encode multipart data  data  files       response   connection getresponse        logging debug   response    s   response read         logging debug   Code   s  s   response status  response reason   def make upload file  server  thread  delay   15  message   None                        username   None  email   None  password   None        delay   max  int  delay or  0    15       def upload file  path  current  total           assert isabs  path          assert isfile  path           logging debug   Uploading  r to  r   path  server          message template   string Template  message or default message           data     MAX FILE SIZE    3145728                    sub                        mode    regist                    com   message template safe substitute  current   current  total   total                    resto   thread                   name   username or                      email   email or                      pwd   password or random string  20            files     upfile   path           send post  server  data  files           logging info   Uploaded  r   path          rand delay   random randint  delay  delay   5          logging debug   Sleeping for   2f seconds------------------------------ n n   rand delay          time sleep  rand delay       return upload file  def upload directory  path  upload file       assert isabs  path      assert isdir  path       matching filenames          file matcher   re compile  r      jpe g gif png     re IGNORECASE       for dirpath  dirnames  filenames in os walk  path           for name in filenames              file path   join  dirpath  name              logging debug   Testing file path  r   file path              if file matcher search  file path                   matching filenames append  file path              else                  logging info   Ignoring non-image file  r   path       total count   len  matching filenames      for index  file path in enumerate  matching filenames           upload file  file path  index   1  total count   def run upload  options  paths       upload file   make upload file    options       for arg in paths          path   abspath  arg          if isdir  path               upload directory  path  upload file          elif isfile  path               upload file  path          else              logging error   No such path   r    path       logging info   Done

[python] Send file using POST from a Python script

Examples related to python

Examples related to post

Examples related to file-upload

Examples related to http-post