Threads are blocked in malloc and free, virtual size

I'm running a 64-bit multi-threaded program on the windows server 2003 server (X64), It run into a case that some of the threads seem to be blocked in the malloc or free function forever. The stack trace is like follows:

ntdll.dll!NtWaitForSingleObject()  + 0xa bytes  
ntdll.dll!RtlpWaitOnCriticalSection()  - 0x1aa bytes    
ntdll.dll!RtlEnterCriticalSection()  + 0xb040 bytes 
ntdll.dll!RtlpDebugPageHeapAllocate()  + 0x2f6 bytes    
ntdll.dll!RtlDebugAllocateHeap()  + 0x40 bytes  
ntdll.dll!RtlAllocateHeapSlowly()  + 0x5e898 bytes  
ntdll.dll!RtlAllocateHeap()  - 0x1711a bytes    
MyProg.exe!malloc(unsigned __int64 size=0)  Line 168    C
MyProg.exe!operator new(unsigned __int64 size=1)  Line 59 + 0x5 bytes   C++


ntdll.dll!NtWaitForSingleObject()   
ntdll.dll!RtlpWaitOnCriticalSection()   
ntdll.dll!RtlEnterCriticalSection()     
ntdll.dll!RtlpDebugPageHeapFree()   
ntdll.dll!RtlDebugFreeHeap()    
ntdll.dll!RtlFreeHeapSlowly()   
ntdll.dll!RtlFreeHeap()     
MyProg.exe!free(void * pBlock=0x000000007e8e4fe0)   C

BTW, the param values passed to the new operator is not correct here maybe due to optimization.

Also, at the same time, I found in the process Explorer, the virtual size of this program is 10GB, but the private bytes and working set is very small (<2GB). We did have some threads using virtualalloc but i开发者_运维问答n a way that commit the memory in the call, and these threads are not blocked.

m_pBuf = VirtualAlloc(NULL, m_size, MEM_COMMIT, PAGE_READWRITE);
......
VirtualFree(m_pBuf, 0, MEM_RELEASE);

This looks strange to me, seems a lot of virtual space is reserved but not committed, and malloc/free is blocked by lock. I'm guessing if there's any corruptions in the memory/object, so plan to turn on gflag with pageheap to troubleshoot this.

Does anyone has similar experience on this before? Could you share with me so I may get more hints?

Thanks a lot!

---

Your program is using PageHeap, which is intended for debugging only and imposes a ton of memory overhead. To see which programs have PageHeap activated, do this at a command line.

% Gflags.exe /p

To disable it for your process, type this (for MyProg.exe):

% Gflags.exe /p /disable MyProg.exe

---

Pageheap.exe detects most heap-related bugs - try Pageheap

Also you should look in to "the param values passed to the new ..." - does this corruption occur in the debug mode? make sure all optimizations are disabled.

---

If your system is running out of memory, it might be the case that the OS is swapping, that means that for a single allocation, in the worst case the OS could need to locate the best candidate for swapping, write it to disk, free the memory and return it. Are you sure that it is locking or might it just be performing very slowly? Can another thread be swapping memory to disk while these two threads wait for it's call to malloc/free to complete?

---

My preferred solution for debugging leaks in native applications in to use UMDH to get consecutive snapshots of the user-mode heap(s) in the process and then run UMDH again to diff the snapshots. Any pattern of change in the snapshots is likely a leak.

You get a count and size of memory blocks bucketed by their allocating callstack so it's reasonably straightforward to see where the biggest hogs are.

The user-mode dump heap (UMDH) utility works with the operating system to analyze Windows heap allocations for a specific process.