What is private bytes virtual bytes working set

Question

I am trying to use the perfmon windows utility to debug memory leaks in a process   This is how perfmon explains the terms   Working Set is the current size  in bytes  of the Working Set of this process  The Working Set is the set of memory pages touched recently by the threads in the process  If free memory in the computer is above a threshold  pages are left in the Working Set of a process even if they are not in use   When free memory falls below a threshold  pages are trimmed from Working Sets  If they are needed they will then be soft-faulted back into the Working Set before leaving main memory   Virtual Bytes is the current size  in bytes  of the virtual address space the process is using  Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages  Virtual space is finite  and the process can limit its ability to load libraries   Private Bytes is the current size  in bytes  of memory that this process has allocated that cannot be shared with other processes   These are the questions I have   Is it the Private Bytes which I should measure to be sure if the process is having any leaks as it does not involve any shared libraries and any leaks  if happening  will come from the process itself   What is the total memory consumed by the process  Is it the Virtual Bytes or is it the sum of Virtual Bytes and Working Set   Is there any relation between Private Bytes  Working Set and Virtual Bytes   Are there any other tools that give a better idea of the memory usage

User · Answer

The definition of the perfmon counters has been broken since the beginning and for some reason appears to be too hard to correct.

A good overview of Windows memory management is available in the video "Mysteries of Memory Management Revealed" on MSDN: It covers more topics than needed to track memory leaks (eg working set management) but gives enough detail in the relevant topics.

To give you a hint of the problem with the perfmon counter descriptions, here is the inside story about private bytes from "Private Bytes Performance Counter -- Beware!" on MSDN:

Q: When is a Private Byte not a Private Byte?

A: When it isn't resident.

The Private Bytes counter reports the commit charge of the process. That is to say, the amount of space that has been allocated in the swap file to hold the contents of the private memory in the event that it is swapped out. Note: I'm avoiding the word "reserved" because of possible confusion with virtual memory in the reserved state which is not committed.

From "Performance Planning" on MSDN:

3.3 Private Bytes

3.3.1 Description

Private memory, is defined as memory allocated for a process which cannot be shared by other processes. This memory is more expensive than shared memory when multiple such processes execute on a machine. Private memory in (traditional) unmanaged dlls usually constitutes of C++ statics and is of the order of 5% of the total working set of the dll.

User · Answer

The short answer to this question is that none of these values are a reliable indicator of how much memory an executable is actually using  and none of them are really appropriate for debugging a memory leak   Private Bytes refer to the amount of memory that the process executable has asked for - not necessarily the amount it is actually using   They are  private  because they  usually  exclude memory-mapped files  i e  shared DLLs    But - here s the catch - they don t necessarily exclude memory allocated by those files   There is no way to tell whether a change in private bytes was due to the executable itself  or due to a linked library   Private bytes are also not exclusively physical memory  they can be paged to disk or in the standby page list  i e  no longer in use  but not paged yet either    Working Set refers to the total physical memory  RAM  used by the process   However  unlike private bytes  this also includes memory-mapped files and various other resources  so it s an even less accurate measurement than the private bytes   This is the same value that gets reported in Task Manager s  Mem Usage  and has been the source of endless amounts of confusion in recent years   Memory in the Working Set is  physical  in the sense that it can be addressed without a page fault  however  the standby page list is also still physically in memory but not reported in the Working Set  and this is why you might see the  Mem Usage  suddenly drop when you minimize an application   Virtual Bytes are the total virtual address space occupied by the entire process   This is like the working set  in the sense that it includes memory-mapped files  shared DLLs   but it also includes data in the standby list and data that has already been paged out and is sitting in a pagefile on disk somewhere   The total virtual bytes used by every process on a system under heavy load will add up to significantly more memory than the machine actually has   So the relationships are    Private Bytes are what your app has actually allocated  but include pagefile usage  Working Set is the non-paged Private Bytes plus memory-mapped files  Virtual Bytes are the Working Set plus paged Private Bytes and standby list    There s another problem here  just as shared libraries can allocate memory inside your application module  leading to potential false positives reported in your app s Private Bytes  your application may also end up allocating memory inside the shared modules  leading to false negatives   That means it s actually possible for your application to have a memory leak that never manifests itself in the Private Bytes at all   Unlikely  but possible   Private Bytes are a reasonable approximation of the amount of memory your executable is using and can be used to help narrow down a list of potential candidates for a memory leak  if you see the number growing and growing constantly and endlessly  you would want to check that process for a leak   This cannot  however  prove that there is or is not a leak   One of the most effective tools for detecting correcting memory leaks in Windows is actually Visual Studio  link goes to page on using VS for memory leaks  not the product page    Rational Purify is another possibility   Microsoft also has a more general best practices document on this subject   There are more tools listed in this previous question   I hope this clears a few things up   Tracking down memory leaks is one of the most difficult things to do in debugging   Good luck

User · Answer

There is an interesting discussion here  http   social msdn microsoft com Forums en-US vcgeneral thread 307d658a-f677-40f2-bdef-e6352b0bfe9e  My understanding of this thread is that freeing small allocations are not reflected in Private Bytes or Working Set   Long story short   if I call  p malloc 1000   free p     then the Private Bytes reflect only the allocation  not the deallocation   if I call  p malloc  gt 512k   free p     then the Private Bytes correctly reflect the allocation and the deallocation

User · Answer

You should not try to use perfmon  task manager or any tool like that to determine memory leaks  They are good for identifying trends  but not much else  The numbers they report in absolute terms are too vague and aggregated to be useful for a specific task such as memory leak detection   A previous reply to this question has given a great explanation of what the various types are   You ask about a tool recommendation  I recommend Memory Validator  Capable of monitoring applications that make billions of memory allocations   http   www softwareverify com cpp memory index html  Disclaimer  I designed Memory Validator

[debugging] What is private bytes, virtual bytes, working set?

Examples related to debugging

Examples related to memory-leaks

Examples related to operating-system

Examples related to memory-management