How to check if pytorch is using the GPU

Question

I would like to know if pytorch is using my GPU  It s possible to detect with nvidia-smi if there is any activity from the GPU during the process  but I want something written in a python script   Is there a way to do so

User · Answer

Create a tensor on the GPU as follows     python  gt  gt  gt  import torch  gt  gt  gt  print torch rand 3 3  cuda       Do not quit  open another terminal and check if the python process is using the GPU using     nvidia-smi

User · Answer

Simply from command prompt or Linux environment run the following command    python -c  import torch  print torch cuda is available       The above should print True  python -c  import torch  print torch rand 2 3  cuda       This one should print the following   tensor   0 7997  0 6170  0 7042    0 4174  0 1494  0 0516    device  cuda 0

User · Answer

After you start running the training loop  if you want to manually watch it from the terminal whether your program is utilizing the GPU resources and to what extent  then you can simply use watch as in     watch -n 2 nvidia-smi   This will continuously update the usage stats for every 2 seconds until you press ctrl c    If you need more control on more GPU stats you might need  you can use more sophisticated version of nvidia-smi with --query-gpu      Below is a simple illustration of this     watch -n 3 nvidia-smi --query-gpu index gpu name memory total memory used memory free temperature gpu pstate utilization gpu utilization memory --format csv   which would output the stats something like     Note  There should not be any space between the comma separated query names in --query-gpu      Else those values will be ignored and no stats are returned     Also  you can check whether your installation of PyTorch detects your CUDA installation correctly by doing   In  13   import  torch  In  14   torch cuda is available   Out 14   True   True status means that PyTorch is configured correctly and is using the GPU although you have to move place the tensors with necessary statements in your code     If you want to do this inside Python code  then look into this module   https   github com jonsafari nvidia-ml-py or in pypi here  https   pypi python org pypi nvidia-ml-py

User · Answer

This is going to work    In  1   import torch  In  2   torch cuda current device   Out 2   0  In  3   torch cuda device 0  Out 3    lt torch cuda device at 0x7efce0b03be0 gt   In  4   torch cuda device count   Out 4   1  In  5   torch cuda get device name 0  Out 5    GeForce GTX 950M   In  6   torch cuda is available   Out 6   True   This tells me the GPU GeForce GTX 950M is being used by PyTorch

User · Answer

As it hasn t been proposed here  I m adding a method using torch device  as this is quite handy  also when initializing tensors on the correct device    setting device on GPU if available  else CPU device   torch device  cuda  if torch cuda is available   else  cpu   print  Using device    device  print     Additional Info when using cuda if device type     cuda       print torch cuda get device name 0       print  Memory Usage        print  Allocated    round torch cuda memory allocated 0  1024  3 1    GB       print  Cached       round torch cuda memory reserved 0  1024  3 1    GB    Edit  torch cuda memory cached has been renamed to torch cuda memory reserved  So use memory cached for older versions  Output  Using device  cuda  Tesla K80 Memory Usage  Allocated  0 3 GB Cached     0 6 GB  As mentioned above  using device it is possible to   To move tensors to the respective device    torch rand 10  to device    To create a tensor directly on the device    torch rand 10  device device     Which makes switching between CPU and GPU comfortable without changing the actual code   Edit  As there has been some questions and confusion about the cached and allocated memory I m adding some additional information about it   torch cuda max memory cached device None  Returns the maximum GPU memory managed by the caching allocator in bytes for a given device   torch cuda memory allocated device None  Returns the current GPU memory usage by tensors in bytes for a given device    You can either directly hand over a device as specified further above in the post or you can leave it None and it will use the current device     Additional note  Old graphic cards with Cuda compute capability 3 0 or lower may be visible but cannot be used by Pytorch  Thanks to hekimgil for pointing this out  -  quot Found GPU0 GeForce GT 750M which is of cuda capability 3 0  PyTorch no longer supports this GPU because it is too old  The minimum cuda capability that we support is 3 5  quot

User · Answer

From practical standpoint just one minor digression  import torch dev   torch device  quot cuda quot   if torch cuda is available   else torch device  quot cpu quot    This dev now knows if cuda or cpu  And there is a difference how you deal with model and with tensors when moving to cuda  It is a bit strange at first  import torch import torch nn as nn dev   torch device  quot cuda quot   if torch cuda is available   else torch device  quot cpu quot   t1   torch randn 1 2  t2   torch randn 1 2  to dev  print t1     tensor   -0 2678   1 9252    print t2     tensor    0 5117  -3 6247    device  cuda 0   t1 to dev   print t1     tensor   -0 2678   1 9252     print t1 is cuda    False t1   t1 to dev  print t1     tensor   -0 2678   1 9252    device  cuda 0    print t1 is cuda    True  class M nn Module       def   init   self                   super     init                     self l1   nn Linear 1 2       def forward self  x                                 x   self l1 x          return x model   M       not on cuda model to dev    is on cuda  all parameters  print next model parameters    is cuda    True  This all is tricky and understanding it once  helps you to deal fast with less debugging

User · Answer

Almost all answers here reference torch cuda is available    However  that s only one part of the coin  It tells you whether the GPU  actually CUDA  is available  not whether it s actually being used  In a typical setup  you would set your device with something like this   device   torch device  cuda   if torch cuda is available   else torch device  cpu     but in larger environments  e g  research  it is also common to give the user more options  so based on input they can disable CUDA  specify CUDA IDs  and so on  In such case  whether or not the GPU is used is not only based on whether it is available or not  After the device has been set to a torch device  you can get its type property to verify whether it s CUDA or not   if device type     cuda         do something

User · Answer

If you are here because your pytorch always gives False for torch cuda is available   that s probably because you installed your pytorch version without GPU support   Eg  you coded up in laptop then testing on server     The solution is to uninstall and install pytorch again with the right command from pytorch downloads page  Also refer this pytorch issue

User · Answer

To check if there is a GPU available   torch cuda is available     If the above function returns False     you either have no GPU   or the Nvidia drivers have not been installed so the OS does not see the GPU   or the GPU is being hidden by the environmental variable CUDA VISIBLE DEVICES   When the value of CUDA VISIBLE DEVICES is -1  then all your devices are being hidden   You can check that value in code with this line  os environ  CUDA VISIBLE DEVICES     If the above function returns True that does not necessarily mean that you are using the GPU  In Pytorch you can allocate tensors to devices when you create them  By default  tensors get allocated to the cpu   To check where your tensor is allocated do     assuming that  a  is a tensor created somewhere else a device    returns the device where the tensor is allocated   Note that you cannot operate on tensors allocated in different devices  To see how to allocate a tensor to the GPU  see here  https   pytorch org docs stable notes cuda html

User · Answer

On the office site and the get start page  check GPU for PyTorch as below    import torch torch cuda is available     Reference  PyTorch Get Start

[python] How to check if pytorch is using the GPU?

Examples related to python

Examples related to memory-management

Examples related to gpu

Examples related to nvidia

Examples related to pytorch