Tricks to manage the available memory in an R session

Question

What tricks do people use to manage the available memory of an interactive R session   I use the functions below  based on postings by Petr Pikal and David Hinds to the r-help list in 2004  to list  and or sort  the largest objects and to occassionally rm   some of them  But by far the most effective solution was     to run under 64-bit Linux with ample memory    Any other nice tricks folks want to share   One per post  please     improved list of objects  ls objects  lt - function  pos   1  pattern  order by                          decreasing FALSE  head FALSE  n 5        napply  lt - function names  fn  sapply names  function x                                           fn get x  pos   pos        names  lt - ls pos   pos  pattern   pattern      obj class  lt - napply names  function x  as character class x   1       obj mode  lt - napply names  mode      obj type  lt - ifelse is na obj class   obj mode  obj class      obj size  lt - napply names  object size      obj dim  lt - t napply names  function x                          as numeric dim x   1 2        vec  lt - is na obj dim    1   amp   obj type     function       obj dim vec  1   lt - napply names  length  vec      out  lt - data frame obj type  obj size  obj dim      names out   lt - c  Type    Size    Rows    Columns       if   missing order by           out  lt - out order out  order by    decreasing decreasing         if  head          out  lt - head out  n      out     shorthand lsos  lt - function      n 10         ls objects      order by  Size   decreasing TRUE  head TRUE  n n

User · Answer

Running  for  i in 1 10       gc reset   T    from time to time also helps R to free unused but still not released memory

User · Answer

I m fortunate and my large data sets are saved by the instrument in  chunks   subsets  of roughly 100 MB  32bit binary   Thus I can do pre-processing steps  deleting uninformative parts  downsampling  sequentially before fusing the data set   Calling gc     by hand  can help if the size of the data get close to available memory  Sometimes a different algorithm needs much less memory  Sometimes there s a trade off between vectorization and memory use  compare  split  amp  lapply vs  a for loop  For the sake of fast  amp  easy data analysis  I often work first with a small random subset  sample     of the data  Once the data analysis script  Rnw is finished data analysis code and the complete data go to the calculation server for over night   over weekend       calculation

User · Answer

I never save an R workspace  I use import scripts and data scripts and output any especially large data objects that I don t want to recreate often to files  This way I always start with a fresh workspace and don t need to clean out large objects  That is a very nice function though

User · Answer

This adds nothing to the above  but is written in the simple and heavily commented style that I like  It yields a table with the objects ordered in size   but without some of the detail given in the examples above    Find the objects        MemoryObjects   ls        Create an array MemoryAssessmentTable array NA dim c length MemoryObjects  2    Name the columns colnames MemoryAssessmentTable  c  object   bytes    Define the first column as the objects MemoryAssessmentTable  1  MemoryObjects  Define a function to determine size         MemoryAssessmentFunction function x  object size get x     Apply the function to the objects MemoryAssessmentTable  2  t t sapply MemoryAssessmentTable  1  MemoryAssessmentFunction     Produce a table with the largest objects first noquote MemoryAssessmentTable rev order as numeric MemoryAssessmentTable  2

User · Answer

Tip for dealing with objects requiring heavy intermediate calculation  When using objects that require a lot of heavy calculation and intermediate steps to create  I often find it useful to write a chunk of code with the function to create the object  and then a separate chunk of code that gives me the option either to generate and save the object as an rmd file  or load it externally from an rmd file I have already previously saved   This is especially easy to do in R Markdown using the following code-chunk structure       r Create OBJECT   COMPLICATED FUNCTION  lt - function        Do heavy calculations needing lots of memory                                          Output OBJECT             r Generate or load OBJECT   LOAD  lt - TRUE   NOTE  Set LOAD to TRUE if you want to load saved file  NOTE  Set LOAD to FALSE if you want to generate and save  if LOAD    TRUE    OBJECT  lt - readRDS file    MySavedObject rds      else                    OBJECT  lt - COMPLICATED FUNCTION x  y  z                                saveRDS file    MySavedObject rds   object   OBJECT            With this code structure  all I need to do is to change LOAD depending on whether I want to generate and save the object  or load it directly from an existing saved file    Of course  I have to generate it and save it the first time  but after this I have the option of loading it    Setting LOAD   TRUE bypasses use of my complicated function and avoids all of the heavy computation therein   This method still requires enough memory to store the object of interest  but it saves you from having to calculate it each time you run your code   For objects that require a lot of heavy calculation of intermediate steps  e g   for calculations involving loops over large arrays  this can save a substantial amount of time and computation

User · Answer

I use the data table package   With its    operator you can     Add columns by reference Modify subsets of existing columns by reference  and by group by reference Delete columns by reference   None of these operations copy the  potentially large  data table at all  not even once    Aggregation is also particularly fast because data table uses much less working memory    Related links     News from data table  London R presentation  2012 When should I use the    operator in data table

User · Answer

For both speed and memory purposes  when building a large data frame via some complex series of steps  I ll periodically flush it  the in-progress data set being built  to disk  appending to anything that came before  and then restart it  This way the intermediate steps are only working on smallish data frames  which is good as  e g   rbind slows down considerably with larger objects   The entire data set can be read back in at the end of the process  when all the intermediate objects have been removed   dfinal  lt - NULL first  lt - TRUE tempfile  lt -  dfinal temp csv  for  i in bigloop         if   i    10000              print  i     flushing to disk               write table  dfinal  file tempfile  append  first  col names first           first  lt - FALSE         dfinal  lt - NULL     nuke it                  complex operations here that add data to  dfinal  data frame     print   Loop done  flushing to disk and re-reading entire data set       write table  dfinal  file tempfile  append TRUE  col names FALSE   dfinal  lt - read table  tempfile

User · Answer

With only 4GB of RAM  running Windows 10  so make that about 2 or more realistically 1GB  I ve had to be real careful with the allocation   I use data table almost exclusively    The  fread  function allows you to subset information by field names on import  only import the fields that are actually needed to begin with  If you re using base R read  null the spurious columns immediately after import   As 42- suggests  where ever possible I will then subset within the columns immediately after importing the information    I frequently rm   objects from the environment as soon as they re no longer needed  e g  on the next line after using them to subset something else  and call gc      fread  and  fwrite  from data table can be very fast by comparison with base R reads and writes   As kpierce8 suggests  I almost always fwrite everything out of the environment and fread it back in  even with thousand   hundreds of thousands of tiny files to get through  This not only keeps the environment  clean  and keeps the memory allocation low but  possibly due to the severe lack of RAM available  R has a propensity for frequently crashing on my computer  really frequently  Having the information backed up on the drive itself as the code progresses through various stages means I don t have to start right from the beginning if it crashes   As of 2017  I think the fastest SSDs are running around a few GB per second through the M2 port  I have a really basic 50GB Kingston V300  550MB s  SSD that I use as my primary disk  has Windows and R on it   I keep all the bulk information on a cheap 500GB WD platter  I move the data sets to the SSD when I start working on them  This  combined with  fread ing and  fwrite ing everything has been working out great  I ve tried using  ff  but prefer the former  4K read write speeds can create issues with this though  backing up a quarter of a million 1k files  250MBs worth  from the SSD to the platter can take hours  As far as I m aware  there isn t any R package available yet that can automatically optimise the  chunkification  process  e g  look at how much RAM a user has  test the read write speeds of the RAM   all the drives connected and then suggest an optimal  chunkification  protocol  This could produce some significant workflow improvements   resource optimisations  e g  split it to     MB for the ram -  split it to     MB for the SSD -  split it to     MB on the platter -  split it to     MB on the tape  It could sample data sets beforehand to give it a more realistic gauge stick to work from   A lot of the problems I ve worked on in R involve forming combination and permutation pairs  triples etc  which only makes having limited RAM more of a limitation as they will often at least exponentially expand at some point  This has made me focus a lot of attention on the quality as opposed to quantity of information going into them to begin with  rather than trying to clean it up afterwards  and on the sequence of operations in preparing the information to begin with  starting with the simplest operation and increasing the complexity   e g  subset  then merge   join  then form combinations   permutations etc   There do seem to be some benefits to using base R read and write in some instances  For instance  the error detection within  fread  is so good it can be difficult trying to get really messy information into R to begin with to clean it up  Base R also seems to be a lot easier if you re using Linux  Base R seems to work fine in Linux  Windows 10 uses  20GB of disc space whereas Ubuntu only needs a few GB  the RAM needed with Ubuntu is slightly lower  But I ve noticed large quantities of warnings and errors when installing third party packages in  L Ubuntu  I wouldn t recommend drifting too far away from  L Ubuntu or other stock distributions with Linux as you can loose so much overall compatibility it renders the process almost pointless  I think  unity  is due to be cancelled in Ubuntu as of 2017   I realise this won t go down well with some Linux users but some of the custom distributions are borderline pointless beyond novelty  I ve spent years using Linux alone    Hopefully some of that might help others out

User · Answer

If you are working on Linux and want to use several processes and only have to do read operations on one or more large objects use makeForkCluster instead of a makePSOCKcluster  This also saves you the time sending the large object to the other processes

User · Answer

You also can get some benefit using knitr and puting your script in Rmd chuncks    I usually divide the code in different chunks and select which one will save a checkpoint to cache or to a RDS file  and   Over there you can set a chunk to be saved to  cache   or you can decide to run or not a particular chunk  In this way  in a first run you can process only  part 1   another execution you can select only  part 2   etc    Example   part1     r corpus  warning FALSE  cache TRUE  message FALSE  eval TRUE  corpusTw  lt - corpus twitter     build the corpus     part2     r trigrams  warning FALSE  cache TRUE  message FALSE  eval FALSE  dfmTw  lt - dfm corpusTw  verbose TRUE  removeTwitter TRUE  ngrams 3        As a side effect  this also could save you some headaches in terms of reproducibility

User · Answer

I quite like the improved objects function developed by Dirk  Much of the time though  a more basic output with the object name and size is sufficient for me  Here s a simpler function with a similar objective  Memory use can be ordered alphabetically or by size  can be limited to a certain number of objects  and can be ordered ascending or descending  Also  I often work with data that are 1GB   so the function changes units accordingly   showMemoryUse  lt - function sort  size   decreasing FALSE  limit       objectList  lt - ls parent frame       oneKB  lt - 1024   oneMB  lt - 1048576   oneGB  lt - 1073741824    memoryUse  lt - sapply objectList  function x  as numeric object size eval parse text x         memListing  lt - sapply memoryUse  function size            if  size  gt   oneGB  return paste round size oneGB 2    GB            else if  size  gt   oneMB  return paste round size oneMB 2    MB            else if  size  gt   oneKB  return paste round size oneKB 2    kB            else return paste size   bytes                memListing  lt - data frame objectName names memListing  memorySize memListing row names NULL     if  sort   alphabetical   memListing  lt - memListing order memListing objectName decreasing decreasing       else memListing  lt - memListing order memoryUse decreasing decreasing     will run if sort not specified or  size     if  missing limit   memListing  lt - memListing 1 limit      print memListing  row names FALSE    return invisible memListing       And here is some example output    gt  showMemoryUse decreasing TRUE  limit 5        objectName memorySize        coherData  713 75 MB  spec pgram mine  149 63 kB        stoch reg  145 88 kB       describeBy    82 5 kB       lmBandpass   68 41 kB

User · Answer

That s a good trick     One other suggestion is to use memory efficient objects wherever possible  for instance  use a matrix instead of a data frame   This doesn t really address memory management  but one important function that isn t widely known is memory limit     You can increase the default using this command  memory limit size 2500   where the size is in MB   As Dirk mentioned  you need to be using 64-bit in order to take real advantage of this

User · Answer

I really appreciate some of the answers above  following  hadley and  Dirk that suggest closing R and issuing source and using command line I come up with a solution that worked very well for me  I had to deal with hundreds of mass spectras  each occupies around 20 Mb of memory so I used two R scripts  as follows   First a wrapper      usr bin Rscript --vanilla --default-packages utils  for l in 1 length fdir         for k in 1 length fds          system paste  Rscript runConsensus r   l  k            with this script I basically control what my main script do runConsensus r  and I write the data answer for the output  With this  each time the wrapper calls the script it seems the R is reopened and the memory is freed   Hope it helps

User · Answer

I try to keep the amount of objects small when working in a larger project with a lot of intermediate steps  So instead of creating many unique objects called  dataframe-  step1 -  step2 -  step3 -  result  raster-  multipliedRast -  meanRastF -  sqrtRast -  resultRast  I work with temporary objects that I call temp   dataframe -  temp -  temp -  temp -  result  Which leaves me with less intermediate files and more overview   raster   lt - raster  file tif   temp  lt - raster   10 temp  lt - mean temp  resultRast  lt - sqrt temp    To save more memory I can simply remove temp when no longer needed    rm temp    If I need several intermediate files  I use temp1  temp2  temp3   For testing I use test  test2

User · Answer

Saw this on a twitter post and think it s an awesome function by Dirk  Following on from JD Long s answer  I would do this for user friendly reading     improved list of objects  ls objects  lt - function  pos   1  pattern  order by                          decreasing FALSE  head FALSE  n 5        napply  lt - function names  fn  sapply names  function x                                           fn get x  pos   pos        names  lt - ls pos   pos  pattern   pattern      obj class  lt - napply names  function x  as character class x   1       obj mode  lt - napply names  mode      obj type  lt - ifelse is na obj class   obj mode  obj class      obj prettysize  lt - napply names  function x                               format utils  object size x   units    auto          obj size  lt - napply names  object size      obj dim  lt - t napply names  function x                          as numeric dim x   1 2        vec  lt - is na obj dim    1   amp   obj type     function       obj dim vec  1   lt - napply names  length  vec      out  lt - data frame obj type  obj size  obj prettysize  obj dim      names out   lt - c  Type    Size    PrettySize    Length Rows    Columns       if   missing order by           out  lt - out order out  order by    decreasing decreasing         if  head          out  lt - head out  n      out      shorthand lsos  lt - function      n 10         ls objects      order by  Size   decreasing TRUE  head TRUE  n n     lsos     Which results in something like the following                         Type   Size PrettySize Length Rows Columns pca res                 PCA 790128   771 6 Kb          7      NA DF               data frame 271040   264 7 Kb        669      50 factor AgeGender   factanal  12888    12 6 Kb         12      NA dates            data frame   9016     8 8 Kb        669       2 sd                  numeric   3808     3 7 Kb         51      NA napply             function   2256     2 2 Kb         NA      NA lsos               function   1944     1 9 Kb         NA      NA load               loadings   1768     1 7 Kb         12       2 ind sup             integer    448  448 bytes        102      NA x                 character     96   96 bytes          1      NA   NOTE  The main part I added was  again  adapted from JD s answer     obj prettysize  lt - napply names  function x                               print object size x   units    auto

User · Answer

If you really want to avoid the leaks  you should avoid creating any big objects in the global environment   What I usually do is to have a function that does the job and returns NULL     all data is read and manipulated in this function or others that it calls

User · Answer

To further illustrate the common strategy of frequent restarts  we can use littler which allows us to run simple expressions directly from the command-line  Here is an example I sometimes use to time different BLAS for a simple crossprod    r -e N lt -3 10 3  M lt -matrix rnorm N N  ncol N   print system time crossprod M       Likewise    r -lMatrix -e example spMatrix     loads the Matrix package  via the --packages   -l switch   and runs the examples of the spMatrix function   As r always starts  fresh   this method is also a good test during package development   Last but not least r also work great for automated batch mode in scripts using the     usr bin r  shebang-header    Rscript is an alternative where littler is unavailable  e g  on Windows

User · Answer

Unfortunately I did not have time to test it extensively but here is a memory tip that I have not seen before  For me the required memory was reduced with more than 50   When you read stuff into R with for example read csv they require a certain amount of memory  After this you can save them with save  Destinationfile  list ls    The next time you open R you can use load  Destinationfile   Now the memory usage might have decreased  It would be nice if anyone could confirm whether this produces similar results with a different dataset

User · Answer

Just to note that data table package s tables   seems to be a pretty good replacement for Dirk s  ls objects   custom function  detailed in earlier answers   although just for data frames tables and not e g  matrices  arrays  lists

User · Answer

The use of environments instead of lists to handle collections of objects which occupy a significant amount of working memory   The reason  each time an element of a list structure is modified  the whole list is temporarily duplicated  This becomes an issue if the storage requirement of the list is about half the available working memory  because then data has to be swapped to the slow hard disk  Environments  on the other hand  aren t subject to this behaviour and they can be treated similar to lists   Here is an example   get data  lt - function x        get some data based on x   return paste  data from  x      collect data  lt - function i x env        get some data   data  lt - get data x  i        store data into environment   element name  lt - paste  V  i sep       env  element name    lt - data   return NULL       better list  lt - new env   filenames  lt - c  file1   file2   file3   lapply seq along filenames  collect data x filenames env better list     read write access print better list   V1     better list   V2     lt -  testdata    number of list elements length ls better list     In conjunction with structures such as big matrix or data table which allow for altering their content in-place  very efficient memory usage can be achieved

User · Answer

Ensure you record your work in a reproducible script   From time-to-time  reopen R  then source   your script   You ll clean out anything you re no longer using  and as an added benefit will have tested your code

User · Answer

The llfunction in gData package can show the memory usage of each object as well   gdata  ll unit  MB

User · Answer

As well as the more general memory management techniques given in the answers above  I always try to reduce the size of my objects as far as possible  For example  I work with very large but very sparse matrices  in other words matrices where most values are zero  Using the  Matrix  package  capitalisation important  I was able to reduce my average object sizes from  2GB to  200MB as simply as   my matrix  lt - Matrix my matrix    The Matrix package includes data formats that can be used exactly like a regular matrix  no need to change your other code  but are able to store sparse data much more efficiently  whether loaded into memory or saved to disk   Additionally  the raw files I receive are in  long  format where each data point has variables x  y  z  i  Much more efficient to transform the data into an x   y   z dimension array with only variable i   Know your data and use a bit of common sense

User · Answer

Based on  Dirk s and  Tony s answer I have made a slight update  The result was outputting  1  before the pretty size values  so I took out the capture output which solved the problem    ls objects  lt - function  pos   1  pattern  order by                       decreasing FALSE  head FALSE  n 5    napply  lt - function names  fn  sapply names  function x      fn get x  pos   pos    names  lt - ls pos   pos  pattern   pattern  obj class  lt - napply names  function x  as character class x   1   obj mode  lt - napply names  mode  obj type  lt - ifelse is na obj class   obj mode  obj class  obj prettysize  lt - napply names  function x        format utils  object size x    units    auto      obj size  lt - napply names  utils  object size   obj dim  lt - t napply names  function x      as numeric dim x   1 2    vec  lt - is na obj dim    1   amp   obj type     function   obj dim vec  1   lt - napply names  length  vec  out  lt - data frame obj type  obj size  obj prettysize  obj dim  names out   lt - c  Type    Size    PrettySize    Rows    Columns   if   missing order by       out  lt - out order out  order by    decreasing decreasing     if  head      out  lt - head out  n   return out       shorthand lsos  lt - function      n 10         ls objects      order by  Size   decreasing TRUE  head TRUE  n n     lsos

User · Answer

I make aggressive use of the subset parameter with selection of only the required variables when passing dataframes to the data  argument of regression functions  It does result in some errors if I forget to add variables to both the formula and the select  vector  but it still saves a lot of time due to decreased copying of objects and reduces the memory footprint significantly  Say I have 4 million records with 110 variables  and I do   Example     library rms   library Hmisc  for the cph and rcs functions Mayo PrCr rbc mdl  lt -  cph formula   Surv surv yr  death    age   Sex   nsmkr   rcs Mayo  4                                          rcs PrCr rat  3     rbc cat   Sex        data   subset set1HLI   gdlab2  amp  HIVfinal     Negative                               select   c  surv yr    death    PrCr rat    Mayo                                           age    Sex    nsmkr    rbc cat                       By way of setting context and the strategy   the gdlab2 variable is a logical vector that was constructed for subjects in a dataset that had all normal or almost normal values for a bunch of laboratory tests and HIVfinal was a character vector that summarized preliminary and confirmatory testing for HIV

User · Answer

I love Dirk s  ls objects   script but I kept squinting to count characters in the size column  So I did some ugly hacks to make it present with pretty formatting for the size    ls objects  lt - function  pos   1  pattern  order by                          decreasing FALSE  head FALSE  n 5        napply  lt - function names  fn  sapply names  function x                                           fn get x  pos   pos        names  lt - ls pos   pos  pattern   pattern      obj class  lt - napply names  function x  as character class x   1       obj mode  lt - napply names  mode      obj type  lt - ifelse is na obj class   obj mode  obj class      obj size  lt - napply names  object size      obj prettysize  lt - sapply obj size  function r  prettyNum r  big mark              obj dim  lt - t napply names  function x                          as numeric dim x   1 2        vec  lt - is na obj dim    1   amp   obj type     function       obj dim vec  1   lt - napply names  length  vec      out  lt - data frame obj type  obj size obj prettysize  obj dim      names out   lt - c  Type    Size    PrettySize    Rows    Columns       if   missing order by           out  lt - out order out  order by    decreasing decreasing             out  lt - out c  Type    PrettySize    Rows    Columns            names out   lt - c  Type    Size    Rows    Columns       if  head          out  lt - head out  n      out

User · Answer

This is a newer answer to this excellent old question  From Hadley s Advanced R   install packages  pryr    library pryr   object size 1 10     88 B  object size mean     832 B  object size mtcars     6 74 kB    http   adv-r had co nz memory html

[memory-management] Tricks to manage the available memory in an R session

Examples related to memory-management

Examples related to r