A second protection technique I implemented in HeapLocker is NOP sled detection.
When you enable NOP sled monitoring, HeapLocker will create a new thread to periodically check (every second) newly committed virtual pages that are readable and writable. When a NOP sled is detected inside these pages with a length equal to or longer than NOPSledLengthMin, HeapLocker will suspend all threads (except this monitoring thread used by HeapLocker) and warn the user that a NOP-sled was detected.
For HeapLocker, a NOP sled is a sequence of single-byte instructions; these may be different or the same instructions. For a list of all single-byte instructions recognized by HeapLocker, take a look array abNOPSledDetection in the source code.
With a classic heap spray, the NOP sled will be detected long before the vulnerability is exploited. But with more sophisticated techniques, it is possible that the NOP sled is detected too late, i.e. that the shellcode already executed. Or it’s also possible that HeapLocker is too early, i.e. that it scans the new page before the NOP sled was written to it. But to prevent this, I wait 1 second between the detection of a new page and the NOP sled scan of that page.
I’ve had some false positives with this detection, that’s why you can configure HeapLocker to ask the user for confirmation.
Didier Stevens 
[…] This post was mentioned on Twitter by xanda. xanda
said: HeapLocker: NOP Sled Detection
[…]
Pingback by Tweets that mention HeapLocker: NOP Sled Detection « Didier Stevens -- Topsy.com — Wednesday 12 January 2011 @ 0:47
[…] NOP Sled Detection
[…]
Pingback by HeapLocker: NOP Sled Detection… | Xanda's Twitter Archive — Wednesday 12 January 2011 @ 0:52
This detection technique is very cool… This methodology is a great way to be included integrity checking tool like OSSEC or some HIDS.. I hope this will be embraced with AV products..
Didier, if we want our vendor to integrate this capability, are you going to share the stable code?
Comment by zack — Wednesday 12 January 2011 @ 2:03
Check out my Fnord snort preprocessor, it was multi-arch and dealt with multibyte nop codes too, it was a response to K2’s ADMutate polymorphic nop sled generator. On snort this algorithm falsed on video streams most often.
Comment by Dragos Ruiu — Wednesday 12 January 2011 @ 15:34
@ZACK Of course, and I’ve not yet made important changes to the code. It’s stable on my machines. But like I wrote, this technique had some false positives, I’m not to happy about it. I like the next technique I’ll explain more (searching for unescape).
Comment by Didier Stevens — Thursday 13 January 2011 @ 11:28
@Dragos Ruiu Thanks, will take a look.
Comment by Didier Stevens — Thursday 13 January 2011 @ 11:28
[…] parentheses). Almost all malicious PDF documents in my collection were detected by this. But like NOP-sled detection, it’s not 100% reliable. Sometimes HeapLocker will scan a page before the string […]
Pingback by HeapLocker: String Detection « Didier Stevens — Friday 18 February 2011 @ 12:44
[…] my HeapLocker tool for almost a year now, and I’ve encountered no issues, except for the NOP sled detection. When used with Adobe Reader, HeapLocker will generate too many false positives when looking for […]
Pingback by HeapLocker: Preventing Heapsprays « Didier Stevens — Tuesday 18 October 2011 @ 8:34
[…] will attempt to detect nop sleds and strings in […]
Pingback by Exploit writing tutorial part 11 : Heap Spraying Demystified | Corelan Team — Sunday 1 January 2012 @ 6:52
is it useful ?
for example : n instructions ” add register, value” where values are random but their sum cause the chosen register to rollover.
can you detect this nop ?
Comment by Massimo Sala — Sunday 16 April 2017 @ 19:27
Is that a one byte instruction?
Comment by Didier Stevens — Monday 17 April 2017 @ 22:41