How do I download a binary file over HTTP

Question

How do I download and save a binary file over HTTP using Ruby   The URL is http   somedomain net flv sample sample flv   I am on the Windows platform and I would prefer not to run any external program

User · Answer

There are more api-friendly libraries than Net  HTTP  for example httparty   require  httparty  File open   tmp my file flv    wb   do  f     f write HTTParty get  http   somedomain net flv sample sample flv   parsed response end

User · Answer

Here is my Ruby http to file using open name   rest   amp block   require  quot open-uri quot  require  quot fileutils quot   def download url  path    case io   open url    when StringIO then File open path   w      f  f write io read      when Tempfile then io close  FileUtils mv io path  path    end end  The main advantage here it is concise and simple  because open does much of the heavy lifting  And it does not read the whole response in memory  The open method will stream responses  gt  1kb to a Tempfile  We can exploit this knowledge to implement this lean download to file method  See the OpenURI  Buffer implementation here  Please be careful with user provided input  open name   rest   amp block  is unsafe if name is coming from user input

User · Answer

Example 3 in the Ruby s net http documentation shows how to download a document over HTTP  and to output the file instead of just loading it into memory  substitute puts with a binary write to a file  e g  as shown in Dejw s answer   More complex cases are shown further down in the same document

User · Answer

I know that this is an old question  but Google threw me here and I think I found a simpler answer   In Railscasts  179  Ryan Bates used the Ruby standard class OpenURI to do much of what was asked like this    Warning  untested code  You might need to change tweak it    require  open-uri   File open   my local path sample flv    wb   do  saved file      the following  open  is provided by open-uri   open  http   somedomain net flv sample sample flv    rb   do  read file      saved file write read file read    end end

User · Answer

Expanding on Dejw s answer  edit2    File open filename  w     f    uri   URI parse url    Net  HTTP start uri host uri port    http       http request get uri path    res         res read body   seg          f  lt  lt  seg  hack -- adjust to suit          sleep 0 005                        where filename and url are strings   The sleep command is a hack that can dramatically reduce CPU usage when the network is the limiting factor   Net  HTTP doesn t wait for the buffer  16kB in v1 9 2  to fill before yielding  so the CPU busies itself moving small chunks around  Sleeping for a moment gives the buffer a chance to fill between writes  and CPU usage is comparable to a curl solution  4-5x difference in my application  A more robust solution might examine progress of f pos and adjust the timeout to target  say  95  of the buffer size -- in fact that s how I got the 0 005 number in my example   Sorry  but I don t know a more elegant way of having Ruby wait for the buffer to fill   Edit   This is a version that automatically adjusts itself to keep the buffer just at or below capacity  It s an inelegant solution  but it seems to be just as fast  and to use as little CPU time  as it s calling out to curl    It works in three stages  A brief learning period with a deliberately long sleep time establishes the size of a full buffer  The drop period reduces the sleep time quickly with each iteration  by multiplying it by a larger factor  until it finds an under-filled buffer  Then  during the normal period  it adjusts up and down by a smaller factor   My Ruby s a little rusty  so I m sure this can be improved upon  First of all  there s no error handling   Also  maybe it could be separated into an object  away from the downloading itself  so that you d just call autosleep sleep f pos  in your loop  Even better  Net  HTTP could be changed to wait for a full buffer before yielding  -   def http to file filename url opt       opt          init pause   gt  0 1      start by waiting this long each time                              it s deliberately long so we can see                               what a full buffer looks like      learn period   gt  0 3    keep the initial pause for at least this many seconds      drop   gt  1 5            fast reducing factor to find roughly optimized pause time      adjust   gt  1 05         during the normal period  adjust up or down by this factor     merge opt    pause   opt  init pause    learn   1    opt  learn period  pause  to i   drop period   true   delta   0   max delta   0   last pos   0   File open filename  w     f      uri   URI parse url      Net  HTTP start uri host uri port    http        http request get uri path    res          res read body   seg            f  lt  lt  seg           delta   f pos - last pos           last pos    delta           if delta  gt  max delta then max delta   delta end           if learn  lt   0 then             learn -  1           elsif delta    max delta then             if drop period then               pause    opt  drop factor              else               pause    opt  adjust              end           elsif delta  lt  max delta then             drop period   false             pause    opt  adjust            end           sleep pause                              end

User · Answer

The simplest way is the platform-specific solution       usr bin env ruby  wget http   somedomain net flv sample sample flv    Probably you are searching for   require  net http    Must be somedomain net instead of somedomain net   otherwise  it will throw exception  Net  HTTP start  somedomain net   do  http      resp   http get   flv sample sample flv       open  sample flv    wb   do  file          file write resp body      end end puts  Done     Edit  Changed  Thank You   Edit2  The solution which saves part of a file while downloading     instead of http get f   open  sample flv   begin     http request get   sample flv   do  resp          resp read body do  segment              f write segment          end     end ensure     f close   end

User · Answer

if you looking for a way how to download temporary file  do stuff and delete it try this gem https   github com equivalent pull tempfile  require  pull tempfile   PullTempfile transaction url   https   mycompany org stupid-csv-report csv   original filename   dont-care csv   do  tmp file    CSV foreach tmp file path  do  row               end end

User · Answer

Following solutions will first read the whole content to memory before writing it to disc  for more i o efficient solutions look at the other answers   You can use open-uri  which is a one liner require  open-uri  content   open  http   example com   read  Or by using net http require  net http  File write  quot file name quot   Net  HTTP get URI parse  quot http   url com quot

User · Answer

I had problems  if the file contained German Umlauts             I could solve the problem by using   ec   Encoding  Converter new  iso-8859-1    utf-8       f  lt  lt  ec convert seg

[ruby] How do I download a binary file over HTTP?

Examples related to ruby

Examples related to download