Didier Stevens

Ever seen a title like this before? Do you know what it means? It usually means that the author didn’t actually test the malware sample on 32 Windows machines, each protected by a different AV product, but that he uploaded the sample to the free VirusTotal service and received a report.

Testing the detection of a malware with 32 AV products and submitting the malware to the VirusTotal services are two different things. Assuming that these tests are equivalent, and implicitly supposing that the results are the same, is plain wrong.

I read enough presentations and articles talking about “tested with 32 AV products” without even mentioning VirusTotal. And that is at least misleading, if not more. To me, “32 AV products” strongly suggests “tested with VirusTotal”, and not “we really tested 32 AV products”.

Julio Canto from VirusTotal was kind enough to answer a couple of questions I had about the free service they are providing.

First of all, VirusTotal uses command-line AV scanners that require no installation, this way they can run 32 different AV products on the same Windows box. These AV scanners run in sequential order when a file is submitted. An active AV product and a command-line AV product are 2 different things, with different goals, fulfilling different needs. Take McAfee for example. McAfee VirusScan Enterprise has a feature called ScriptScan that will intercept and scan each VBScript and JavaScript before it is execute by the Microsoft script engine. The command-line version of McAfee doesn’t have this feature. So if you let VirusTotal scan an heavily obfuscated script, it’s likely that the McAfee command-line scanner used by VirusTotal will not detect it. But it’s likely that McAfee VirusScan will detect it with ScriptScan, before it gets executed.

It’s the AV vendor that decides which version of his product will be used by VirusTotal and how it has to be configured. Some vendors will even provide beta versions of their product for the VirusTotal team to use. VirusTotal has a NDA with most vendors, that’s why they don’t provide the configuration details for each AV engine. Some vendors are conservative in their settings, while others will use all options (like heuristics).

VirusTotal does not executed submitted files in a sandbox, they are just scanned by the AV engines.

If you don’t get 32 results in your report, but less, it means that an AV engine timed-out (didn’t respond in the allotted time, and the process was killed) and didn’t provide a detection report. The VirusTotal service uses a cluster of 16 machines.

Although the VirusTotal service generates a lot of data that contains a wealth of statistics, they don’t usually look for trends. The company behind VirusTotal (Hispasec), is not involved in the AV world at all, but can use some of the statistics for consulting services.

VirusTotal implemented an anti-abuse system: if one source is submitting too much samples in a too short time period, subsequent request will be refused. This is done to provide all users an equal access to the service.

To finish, Julio gave me some links to similar services:

• http://virusscan.jotti.org/
• http://scanner.virus.org/ (dead link at time of writing)
• http://viruschief.com/
• http://www.virscan.org/

And remember, when you’re using the VirusTotal service, you’re testing your submitted sample, you’re not testing the AV products. At most, you could say you’re testing bare AV engines with a configuration that is unknown to you.