Using R to download zipped data file extract and import data

Question

EZGraphs on Twitter writes   Lots of online csvs are zipped  Is there a way to download  unzip the archive  and load the data to a data frame using R   Rstats   I was also trying to do this today  but ended up just downloading the zip file manually   I tried something like   fileName  lt -  http   www newcl org data zipfiles a1 zip  con1  lt - unz fileName  filename  a1 dat   open    r     but I feel as if I m a long way off  Any thoughts

User · Accepted Answer

Zip archives are actually more a  filesystem  with content metadata etc  See help unzip  for details   So to do what you sketch out above you need to    Create a temp  file name  eg tempfile    Use download file   to fetch the file into the temp  file Use unz   to extract the target file from temp  file Remove the temp file via unlink     which in code  thanks for basic example  but this is simpler  looks like  temp  lt - tempfile   download file  http   www newcl org data zipfiles a1 zip  temp  data  lt - read table unz temp   a1 dat    unlink temp    Compressed   z  or gzipped   gz  or bzip2ed   bz2  files are just the file and those you can read directly from a connection   So get the data provider to use that instead

User · Answer

To do this using data table  I found that the following works  Unfortunately  the link does not work anymore  so I used a link for another data set   library data table  temp  lt - tempfile   download file  https   www bls gov tus special requests atusact 0315 zip   temp  timeUse  lt - fread unzip temp  files    atusact 0315 dat    rm temp    I know this is possible in a single line since you can pass bash scripts to fread  but I am not sure how to download a  zip file  extract  and pass a single file from that to fread

User · Answer

Try this code  It works for me   unzip zipfile   lt directory and filename gt          exdir   lt directory where the content will be extracted gt      Example   unzip zipfile    data Data zip  exdir    data

User · Answer

rio   would be very suitable for this - it uses the file extension of a file name to determine what kind of file it is  so it will work with a large variety of file types  I ve also used unzip   to list the file names within the zip file  so its not necessary to specify the file name s  manually  library rio     create a temporary directory td  lt - tempdir      create a temporary file tf  lt - tempfile tmpdir td  fileext  quot  zip quot      download file from internet into temporary location download file  quot http   download companieshouse gov uk BasicCompanyData-part1 zip quot   tf     list zip archive file names  lt - unzip tf  list TRUE     extract files from zip file unzip tf  exdir td  overwrite TRUE     use when zip file has only one file data  lt - import file path td  file names Name 1       use when zip file has multiple files data multiple  lt - lapply file names Name  function x  import file path td  x       delete the files and directories unlink td

User · Answer

Here is an example that works for files which cannot be read in with the read table function  This example reads a  xls file   url  lt - https   www1 toronto ca City Of Toronto Information Technology Open Data Data Sets Assets Files fire stns zip   temp  lt - tempfile   temp2  lt - tempfile    download file url  temp  unzip zipfile   temp  exdir   temp2  data  lt - read xls file path temp2   fire station x y xls     unlink c temp  temp2

User · Answer

I used CRAN package  downloader  found at http   cran r-project org web packages downloader index html   Much easier   download url  dest  dataset zip   mode  wb    unzip   dataset zip   exdir

User · Answer

Just for the record  I tried translating Dirk s answer into code  -P  temp  lt - tempfile   download file  http   www newcl org data zipfiles a1 zip  temp  con  lt - unz temp   a1 dat   data  lt - matrix scan con  ncol 4 byrow TRUE  unlink temp

User · Answer

For Mac  and I assume Linux      If the zip archive contains a single file  you can use the bash command funzip  in conjuction with fread from the data table package   library data table  dt  lt - fread  curl http   www newcl org data zipfiles a1 zip   funzip     In cases where the archive contains multiple files  you can use tar instead to extract a specific file to stdout   dt  lt - fread  curl http   www newcl org data zipfiles a1 zip   tar -xf- --to-stdout  a1 dat

User · Answer

I found that the following worked for me  These steps come from BTD s YouTube video  Managing Zipfile s in R  zip url  lt -  quot url address zip quot   dir  lt - getwd    zip file  lt -  quot file name zip quot   zip combine  lt - as character paste dir  zip file  sep    quot   quot     download file zip url  destfile   zip combine   unzip zip file

[r] Using R to download zipped data file, extract, and import data

Examples related to r

Examples related to zip

Examples related to connection