Didier Stevens

Monday 21 April 2008

“Only X Out of 32 Antivirus Products Detect This!”

Filed under: Malware — Didier Stevens @ 6:47

Ever seen a title like this before? Do you know what it means? It usually means that the author didn’t actually test the malware sample on 32 Windows machines, each protected by a different AV product, but that he uploaded the sample to the free VirusTotal service and received a report.

Testing the detection of a malware with 32 AV products and submitting the malware to the VirusTotal services are two different things. Assuming that these tests are equivalent, and implicitly supposing that the results are the same, is plain wrong.

I read enough presentations and articles talking about “tested with 32 AV products” without even mentioning VirusTotal. And that is at least misleading, if not more. To me, “32 AV products” strongly suggests “tested with VirusTotal”, and not “we really tested 32 AV products”.

Julio Canto from VirusTotal was kind enough to answer a couple of questions I had about the free service they are providing.

First of all, VirusTotal uses command-line AV scanners that require no installation, this way they can run 32 different AV products on the same Windows box. These AV scanners run in sequential order when a file is submitted. An active AV product and a command-line AV product are 2 different things, with different goals, fulfilling different needs. Take McAfee for example. McAfee VirusScan Enterprise has a feature called ScriptScan that will intercept and scan each VBScript and JavaScript before it is execute by the Microsoft script engine. The command-line version of McAfee doesn’t have this feature. So if you let VirusTotal scan an heavily obfuscated script, it’s likely that the McAfee command-line scanner used by VirusTotal will not detect it. But it’s likely that McAfee VirusScan will detect it with ScriptScan, before it gets executed.

It’s the AV vendor that decides which version of his product will be used by VirusTotal and how it has to be configured. Some vendors will even provide beta versions of their product for the VirusTotal team to use. VirusTotal has a NDA with most vendors, that’s why they don’t provide the configuration details for each AV engine. Some vendors are conservative in their settings, while others will use all options (like heuristics).

VirusTotal does not executed submitted files in a sandbox, they are just scanned by the AV engines.

If you don’t get 32 results in your report, but less, it means that an AV engine timed-out (didn’t respond in the allotted time, and the process was killed) and didn’t provide a detection report. The VirusTotal service uses a cluster of 16 machines.

Although the VirusTotal service generates a lot of data that contains a wealth of statistics, they don’t usually look for trends. The company behind VirusTotal (Hispasec), is not involved in the AV world at all, but can use some of the statistics for consulting services.

VirusTotal implemented an anti-abuse system: if one source is submitting too much samples in a too short time period, subsequent request will be refused. This is done to provide all users an equal access to the service.

To finish, Julio gave me some links to similar services:

And remember, when you’re using the VirusTotal service, you’re testing your submitted sample, you’re not testing the AV products. At most, you could say you’re testing bare AV engines with a configuration that is unknown to you.

4 Comments »

  1. Fantastic overview. I am a frequent “customer” of VT, so it is interesting to get a deeper look into their operating environment.

    Thank you Didier and Julio!

    Comment by Stephen Moore — Wednesday 23 April 2008 @ 20:19

  2. [...] VirusTotal coverage: 17/33 (Caveat emptor) [...]

    Pingback by 4th of July, Business as Usual « Didier Stevens — Friday 4 July 2008 @ 8:40

  3. Check out : Anti-Malware Testing Standards Organization

    http://www.amtso.org/

    Gr,
    eth-x

    Comment by eth-x — Tuesday 2 December 2008 @ 12:14

  4. [...] from Milw0rm and let Virustotal take a look at it. Now don’t be mislead by the 5/39 ratio, this doesn’t necessarily mean that most AV products will not protect you from this [...]

    Pingback by Quickpost: /JBIG2Decode Signatures « Didier Stevens — Sunday 1 March 2009 @ 20:17


RSS feed for comments on this post. TrackBack URI

Leave a Reply (comments are moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 231 other followers

%d bloggers like this: