Didier Stevens

Explaining how Private Usage Memory Monitoring in HeapLocker works is easy, so let’s start with this technique.

When a malicious document performs a heap spray, allocation of private virtual memory will skyrocket. HeapLocker allows you to set a maximum to the amount of private virtual memory a process is using. If the maximum is exceeded, HeapLocker will suspend the process and inform the user.

To configure HeapLocker to monitor Adobe Reader’s usage of private memory and set a maximum, create these registry keys:

[HKEY_LOCAL_MACHINE\SOFTWARE\DidierStevens\HeapLocker\Applications\AcroRd32.exe]
"PrivateUsageMax"=dword:00000100

This will set the maximum to 256 MB (0x100).

When HeapLocker is loaded inside Acrobat Reader (AcroRd32.exe process), it will find the PrivateUsageMax setting and start a new thread inside the Adobe Reader process. During most of its life, this thread will do nothing (sleep). Every second, it will wake-up and check the amount of private memory allocated by the process (PrivateUsage). If this amount is equal to or larger than the maximum value specified in PrivateUsageMax, the thread will suspend all other threads in the process and display a message box to the user.

The user can decide to terminate the process, or decide to continue using the program. When the user decides to continue, HeapLocker will resume all threads and disable its monitoring of private memory usage for the life of this process.

The user is offered this choice to be able to deal with false positives. If you set your PrivateUsageMax value too low and you open many different documents in the same application, you could exceed the limit without an actual heap spray taking place. For applications that (mostly) just display data, like Adobe Reader and Internet Explorer, this is not a real issue, because no data is lost when you terminate the application. But with Microsoft Word or Excel, you will loose unsaved data when the process is terminated.

If you don’t trust your users to make the right decision when presented with this dialog, you can set registry key ForceTermination to 1. When this key is set, the user will not be offered a choice: the threads are suspended, a message box is displayed with just an OK button, and the application is terminated when the OK button is clicked.

I’ve tested this feature of HeapLocker for several months with Adobe and MS Office, limiting the private usage to 1024 MB. I don’t recommend you use the other HeapLocker techniques yet, I’m still testing these features.

This feature of HeapLocker is compatible with EMET, I’ve used both tools concurrently to protect Adobe Reader.

You can use Process Explorer to get an idea of the private memory usage of your applications.