How to prevent tensorflow from allocating the totality of a GPU memory

Question

I work in an environment in which computational resources are shared  i e   we have a few server machines equipped with a few Nvidia Titan X GPUs each  For small to moderate size models  the 12 GB of the Titan X is usually enough for 2   3 people to run training concurrently on the same GPU  If the models are small enough that a single model does not take full advantage of all the computational units of the GPU  this can actually result in a speedup compared with running one training process after the other  Even in cases where the concurrent access to the GPU does slow down the individual training time  it is still nice to have the flexibility of having multiple users simultaneously train on the GPU  The problem with TensorFlow is that  by default  it allocates the full amount of available GPU memory when it is launched  Even for a small two-layer neural network  I see that all 12 GB of the GPU memory is used up  Is there a way to make TensorFlow only allocate  say  4 GB of GPU memory  if one knows that this is enough for a given model

User · Answer

For Tensorflow version 2 0 and 2 1 use the following snippet    import tensorflow as tf  gpu devices   tf config experimental list physical devices  GPU    tf config experimental set memory growth gpu devices 0   True    For prior versions   following snippet used to work for me   import tensorflow as tf tf config tf ConfigProto   tf config gpu options allow growth True sess   tf Session config tf config

User · Answer

Tensorflow 2 0 Beta and  probably  beyond  The API changed again  It can be now found in   tf config experimental set memory growth      device      enable     Aliases    tf compat v1 config experimental set memory growth tf compat v2 config experimental set memory growth   References    https   www tensorflow org versions r2 0 api docs python tf config experimental set memory growth https   www tensorflow org guide gpu limiting gpu memory growth   See also  Tensorflow - Use a GPU  https   www tensorflow org guide gpu  for Tensorflow 2 0 Alpha see  this answer

User · Answer

If you re using Tensorflow 2 try the following   config   tf compat v1 ConfigProto   config gpu options allow growth   True session   tf compat v1 Session config config

User · Answer

i tried to train unet on voc data set but because of huge image size  memory finishes  i tried all the above tips  even tried with batch size  1  yet to no improvement  sometimes TensorFlow version also causes the memory issues  try by using      pip install tensorflow-gpu  1 8 0

User · Answer

You can set the fraction of GPU memory to be allocated when you construct a tf Session by passing a tf GPUOptions as part of the optional config argument     Assume that you have 12GB of GPU memory and want to allocate  4GB  gpu options   tf GPUOptions per process gpu memory fraction 0 333   sess   tf Session config tf ConfigProto gpu options gpu options     The per process gpu memory fraction acts as a hard upper bound on the amount of GPU memory that will be used by the process on each GPU on the same machine  Currently  this fraction is applied uniformly to all of the GPUs on the same machine  there is no way to set this on a per-GPU basis

User · Answer

config   tf ConfigProto   config gpu options allow growth True sess   tf Session config config    https   github com tensorflow tensorflow issues 1578

User · Answer

You can use   TF FORCE GPU ALLOW GROWTH true   in your environment variables   In tensorflow code   bool GPUBFCAllocator  GetAllowGrowthValue const GPUOptions amp  gpu options      const char  force allow growth string         std  getenv  TF FORCE GPU ALLOW GROWTH      if  force allow growth string    nullptr        return gpu options allow growth

User · Answer

Shameless plug  If you install the GPU supported Tensorflow  the session will first allocate all GPU whether you set it to use only CPU or GPU  I may add my tip that even you set the graph to use CPU only you should set the same configuration as answered above     to prevent the unwanted GPU occupation  And in an interactive interface like IPython and Jupyter  you should also set that configure  otherwise  it will allocate all memory and left almost none for others  This is sometimes hard to notice

User · Answer

Well I am new to tensorflow  I have Geforce 740m or something GPU with 2GB ram  I was running mnist handwritten kind of example for a native language with training data containing of 38700 images and 4300 testing images and was trying to get precision   recall   F1 using following code as sklearn was not giving me precise reults  once i added this to my existing code i started getting GPU errors   TP   tf count nonzero predicted   actual  TN   tf count nonzero  predicted - 1     actual - 1   FP   tf count nonzero predicted    actual - 1   FN   tf count nonzero  predicted - 1    actual   prec   TP    TP   FP  recall   TP    TP   FN  f1   2   prec   recall    prec   recall    plus my model was heavy i guess  i was getting memory error after 147  148 epochs  and then I thought why not create functions for the tasks so I dont know if it works this way in tensrorflow  but I thought if a local variable is used and when out of scope it may release memory and i defined the above elements for training and testing in modules  I was able to achieve 10000 epochs without any issues  I hope this will help

User · Answer

All the answers above refer to either setting the memory to a certain extent in TensorFlow 1 X versions or to allow memory growth in TensorFlow 2 X  The method  tf config experimental set memory growth indeed works for allowing dynamic growth during the allocation preprocessing  Nevertheless one may like to allocate from the start a specific GPU memory  The logic behind allocating a specific GPU memory would also be to prevent OOM memory during training sessions  For example  if one trains while opening video-memory consuming Chrome tabs any other video consumption process  the tf config experimental set memory growth gpu  True  could result in OOM errors thrown  hence the necessity of allocating from the start more memory in certain cases  The recommended and correct way in which to allot memory per GPU in TensorFlow 2 X is done in the following manner  gpus   tf config experimental list physical devices  GPU   if gpus      Restrict TensorFlow to only allocate 1GB of memory on the first GPU   try      tf config experimental set virtual device configuration          gpus 0            tf config experimental VirtualDeviceConfiguration memory limit 1024

User · Answer

For TensorFlow 2 0 and 2 1  docs   import tensorflow as tf tf config gpu set per process memory growth True   For TensorFlow 2 2   docs   import tensorflow as tf gpus   tf config experimental list physical devices  GPU   for gpu in gpus    tf config experimental set memory growth gpu  True   The docs also list some more methods   Set environment variable TF FORCE GPU ALLOW GROWTH to true  Use tf config experimental set virtual device configuration to set a hard limit on a Virtual GPU device

User · Answer

Here is an excerpt from the Book Deep Learning with TensorFlow      In some cases it is desirable for the process to only allocate a subset of the available memory  or to only grow the memory usage as it is needed by the process  TensorFlow provides two configuration options on the session to control this  The first is the allow growth option  which attempts to allocate only as much GPU memory based on runtime allocations  it starts out allocating very little memory  and as sessions get run and more GPU memory is needed  we extend the GPU memory region needed by the TensorFlow process    1  Allow growth   more flexible   config   tf ConfigProto   config gpu options allow growth   True session   tf Session config config         The second method is per process gpu memory fraction option  which determines the fraction of the overall amount of memory that each visible GPU should be allocated  Note  No release of memory needed  it can even worsen memory fragmentation when done   2  Allocate fixed memory   To only allocate 40  of the total memory of each GPU by   config   tf ConfigProto   config gpu options per process gpu memory fraction   0 4 session   tf Session config config         Note   That s only useful though if you truly want to bind the amount of GPU memory available on the TensorFlow process

User · Answer

allocate 60  of GPU memory  from keras backend tensorflow backend import set session import tensorflow as tf  config   tf ConfigProto   config gpu options per process gpu memory fraction   0 6 set session tf Session config config

User · Answer

For Tensorflow 2 0 this this solution worked for me    TF-GPU 2 0  Windows 10  GeForce RTX 2070   physical devices   tf config experimental list physical devices  GPU   assert len physical devices   gt  0   Not enough GPU hardware devices available  tf config experimental set memory growth physical devices 0   True

User · Answer

All the answers above assume execution with a sess run   call  which is becoming the exception rather than the rule in recent versions of TensorFlow   When using the tf Estimator framework  TensorFlow 1 4 and above  the way to pass the fraction along to the implicitly created MonitoredTrainingSession is   opts   tf GPUOptions per process gpu memory fraction 0 333  conf   tf ConfigProto gpu options opts  trainingConfig   tf estimator RunConfig session config conf       tf estimator Estimator model fn                              config trainingConfig    Similarly in Eager mode  TensorFlow 1 5 and above     opts   tf GPUOptions per process gpu memory fraction 0 333  conf   tf ConfigProto gpu options opts  tfe enable eager execution config conf    Edit  11-04-2018 As an example  if you are to use tf contrib gan train  then you can use something similar to bellow   tf contrib gan gan train           config conf

User · Answer

this code has worked for me  import tensorflow as tf config   tf compat v1 ConfigProto   config gpu options allow growth   True session   tf compat v1 InteractiveSession config config

[python] How to prevent tensorflow from allocating the totality of a GPU memory?

Examples related to python

Examples related to tensorflow

Examples related to tensorflow2.0

Examples related to tensorflow2.x

Examples related to nvidia-titan