Example 3 in the Ruby's net/http documentation shows how to download a document over HTTP, and to output the file instead of just loading it into memory, substitute puts with a binary write to a file, e.g. as shown in Dejw's answer.
More complex cases are shown further down in the same document.
I had problems, if the file contained German Umlauts (ä,ö,ü). I could solve the problem by using:
ec = Encoding::Converter.new('iso-8859-1', 'utf-8')
...
f << ec.convert(seg)
...
Following solutions will first read the whole content to memory before writing it to disc (for more i/o efficient solutions look at the other answers).
You can use open-uri, which is a one liner
require 'open-uri'
content = open('http://example.com').read
Or by using net/http
require 'net/http'
File.write("file_name", Net::HTTP.get(URI.parse("http://url.com")))
I know that this is an old question, but Google threw me here and I think I found a simpler answer.
In Railscasts #179, Ryan Bates used the Ruby standard class OpenURI to do much of what was asked like this:
(Warning: untested code. You might need to change/tweak it.)
require 'open-uri'
File.open("/my/local/path/sample.flv", "wb") do |saved_file|
# the following "open" is provided by open-uri
open("http://somedomain.net/flv/sample/sample.flv", "rb") do |read_file|
saved_file.write(read_file.read)
end
end
Expanding on Dejw's answer (edit2):
File.open(filename,'w'){ |f|
uri = URI.parse(url)
Net::HTTP.start(uri.host,uri.port){ |http|
http.request_get(uri.path){ |res|
res.read_body{ |seg|
f << seg
#hack -- adjust to suit:
sleep 0.005
}
}
}
}
where filename and url are strings.
The sleep command is a hack that can dramatically reduce CPU usage when the network is the limiting factor. Net::HTTP doesn't wait for the buffer (16kB in v1.9.2) to fill before yielding, so the CPU busies itself moving small chunks around. Sleeping for a moment gives the buffer a chance to fill between writes, and CPU usage is comparable to a curl solution, 4-5x difference in my application. A more robust solution might examine progress of f.pos and adjust the timeout to target, say, 95% of the buffer size -- in fact that's how I got the 0.005 number in my example.
Sorry, but I don't know a more elegant way of having Ruby wait for the buffer to fill.
Edit:
This is a version that automatically adjusts itself to keep the buffer just at or below capacity. It's an inelegant solution, but it seems to be just as fast, and to use as little CPU time, as it's calling out to curl.
It works in three stages. A brief learning period with a deliberately long sleep time establishes the size of a full buffer. The drop period reduces the sleep time quickly with each iteration, by multiplying it by a larger factor, until it finds an under-filled buffer. Then, during the normal period, it adjusts up and down by a smaller factor.
My Ruby's a little rusty, so I'm sure this can be improved upon. First of all, there's no error handling. Also, maybe it could be separated into an object, away from the downloading itself, so that you'd just call autosleep.sleep(f.pos) in your loop? Even better, Net::HTTP could be changed to wait for a full buffer before yielding :-)
def http_to_file(filename,url,opt={})
opt = {
:init_pause => 0.1, #start by waiting this long each time
# it's deliberately long so we can see
# what a full buffer looks like
:learn_period => 0.3, #keep the initial pause for at least this many seconds
:drop => 1.5, #fast reducing factor to find roughly optimized pause time
:adjust => 1.05 #during the normal period, adjust up or down by this factor
}.merge(opt)
pause = opt[:init_pause]
learn = 1 + (opt[:learn_period]/pause).to_i
drop_period = true
delta = 0
max_delta = 0
last_pos = 0
File.open(filename,'w'){ |f|
uri = URI.parse(url)
Net::HTTP.start(uri.host,uri.port){ |http|
http.request_get(uri.path){ |res|
res.read_body{ |seg|
f << seg
delta = f.pos - last_pos
last_pos += delta
if delta > max_delta then max_delta = delta end
if learn <= 0 then
learn -= 1
elsif delta == max_delta then
if drop_period then
pause /= opt[:drop_factor]
else
pause /= opt[:adjust]
end
elsif delta < max_delta then
drop_period = false
pause *= opt[:adjust]
end
sleep(pause)
}
}
}
}
end
There are more api-friendly libraries than Net::HTTP, for example httparty:
require "httparty"
File.open("/tmp/my_file.flv", "wb") do |f|
f.write HTTParty.get("http://somedomain.net/flv/sample/sample.flv").parsed_response
end
Here is my Ruby http to file using open(name, *rest, &block).
require "open-uri"
require "fileutils"
def download(url, path)
case io = open(url)
when StringIO then File.open(path, 'w') { |f| f.write(io.read) }
when Tempfile then io.close; FileUtils.mv(io.path, path)
end
end
The main advantage here it is concise and simple, because open does much of the heavy lifting. And it does not read the whole response in memory.
The open method will stream responses > 1kb to a Tempfile. We can exploit this knowledge to implement this lean download to file method.
See the OpenURI::Buffer implementation here.
Please be careful with user provided input!
open(name, *rest, &block) is unsafe if name is coming from user input!
if you looking for a way how to download temporary file, do stuff and delete it try this gem https://github.com/equivalent/pull_tempfile
require 'pull_tempfile'
PullTempfile.transaction(url: 'https://mycompany.org/stupid-csv-report.csv', original_filename: 'dont-care.csv') do |tmp_file|
CSV.foreach(tmp_file.path) do |row|
# ....
end
end
Source: Stackoverflow.com