Didier Stevens

count is a simple program: it takes text files as input and counts how many times each lines appears.

A couple of years ago, I made a video:

count.py uses a Python dictionary to count items, but that requires a lot of memory to process gigabytes of data.

This new version helps with this problem by providing a count method using a database (sqlite3). By default, a dictionary is still used. But counting with a database can be selected with option -c. With option -c you can provide the name of the database to use: if the name is :memory:, the database will be created in memory. Counting with a sqlite3 database in memory requires less memory than counting with a Python dictionary, but is slower. If the name is a filename, the database will be created on disk. This is of course way slower than in memory, but can process even larger files.

count_v0_2_0.zip (https)
MD5: ACF1982045ABEF86FCDBA87A84F5F588
SHA256: 373DDA0B2C176624998B5907261477943F677855CCECCDD42D6BEB758F8E7B79

python-per-line is a tool to apply a Python expression on each line of input.

I updated it because I had to process large credential dumps (I’ll blog about this later).

This new version can process .gz files too, and includes three new predefined Python functions: IFF, RIN and SBC.

From the man page:

IFF is a predefined Python function that implements the if Function
(IFF = IF Function). It takes three arguments: expression, valueTrue,
valueFalse. If expression is true, then valueTrue is returned,
otherwise valueFalse is returned.

RIN is a predefined Python function that uses the repr function if
needed (RIN = Repr If Needed). When a string contains characters that
need to be escaped to be used in Python source code, repr(string) is
returned, otherwise the string itself is returned.

SBC is a predefined Python function that helps with selecting a value
from lines with values and separators (Separator Based Cut = SBC). SBC
takes five arguments: data, separator, columns, column, failvalue.
data is the data we want to parse (usually line), separator is the
separator character, columns is the number of columns per line, column
is the value we want to select (cut) starting from 0, and failvalue is
the value that SBC needs to return if the function fails (for example
because there are less columns in the line than specified by the
columns value).
Here is an example. We use this file with credentials (creds.txt):
username1:password
username2
username3:pass:word
username4:

And this is the command to extract the passwords:
python-per-line.py "SBC(line, ':', 2, 1, [])" creds.txt

The result:
password
pass:word

If a line contains more separators than specified by the columns
argument, then everything past the last expected separator is
considered the last value (this includes the extra separator(s)). We
can see this with line "username3:pass:word". The password is
pass:word (not pass). SBC returns pass:word.
If a line contains less separators than specified by the columns
argument, then the failvalue is returned. [] makes python-per-line
skip an output line, that is why no output is produced for user2.

python-per-line_V0_0_2.zip (https)
MD5: AB2377D366AB33992A535AF1EE489CBD
SHA256: 045F398FBCF6DDFF4A25B38007ADDF89B3256C21C8808B58FBC96855D55E6171

This new version outputs the filename for attachments:

emldump_V0_0_10.zip (https)
MD5: 34DBB3BCB1A2B04C45286C0583F11C07
SHA256: C5877E252DDB61B40BFFCC5403DB500E672DACFE96FAA7D1E0668246C5202DE5

Like I did with zipdump, this oledump version now also supports YARA rules provided via the command-line (# and #s#).

oledump_V0_0_28.zip (https)
MD5: D89C1E0DA9A95A166EF8F36165F6A873
SHA256: 58F44B68BC997C2A7F329978E13DC50E406CCCCD2017C0375AA144712F029BFB

Sometimes I just need to search for a string in the files of a ZIP container, and for that I need to create a small YARA rule.

With this new version, I can let zipdump generate the rule, I just need to provide the string. The value provided to option -y needs to start with #s# (s stands for string). Here is an example where I search for string HUBBLE:

zipdump_v0_0_11.zip (https)
MD5: E97E0191757230D2C7F9109B91636BF7
SHA256: 6640F971F61F7915D89388D3072854C00C81C47476A96CAC7BE6740DA348467B

I regularly use YARA rules with my tools. Option -y starts the YARA engine, and option –yarastrings gives an overview of the matched strings, like this:

But it’s too much information when I use regular expressions in my YARA rules to match, for example, XML elements.

I added option –yarastringsraw to zipdump to view just the matched string, and nothing else:

zipdump_v0_0_10.zip (https)
MD5: 71B2483D24C4258DD34406CC433A3AF0
SHA256: 1259ABC36FDC13A2738D9C38549AB95A83D5039190ADAF44590E07AF6785BF7A

This new version of re-search.py introduces options –script and –execute to provide your custom Python functions.

Regular expressions can contain comments, like programming languages. This is a comment for regular expressions: (?#comment).
If you use re-search with regular expression comments, nothing special happens:
re-search.py “(?#comment)[a-z]+\.com” list.txt

However, if your regular expression comment prefixes the regular expression, and the comment starts with keyword extra=, then you can use gibberish detection, whitelist/blacklist filtering and Python function matching.

Python function matching is defined via directive P (Python). If you want to validate a string with a Python function, you use the following regular expression comment: (?#extra=P:Validate). Validate is a Python function that takes a string as argument and returns a boolean: True for a match and False if there is no match. You can provide your custom Python function(s) in a file via option –script or as a commandline argument via option –execute.

Example: Bitcoin address matching. Regular expression [13][a-km-zA-HJ-NP-Z1-9]{25,34} will match Bitcoin addresses, but also other strings that look like a Bitcoin address but are not a valid Bitcoin address. A valid Bitcoin address has a particular syntax, and a valid checksum. The regular expression can check the syntax, but not validate the checksum. Python function BTCValidate can check the checksum of a Bitcoin address. The following regular expression matches Bitcoin addresses with a valid syntax and uses Python function BTCValidate to validate the checksum:
(?#extra=P:BTCValidate)[13][a-km-zA-HJ-NP-Z1-9]{25,34}

re-search_V0_0_8.zip (https)
MD5: D4895B54268683BFBE0126D02B01A4A2
SHA256: 85919EB964FF9CF0EDE7DA64E9BCE6619480DAC71D0CB65B5EE667322B18DDBB

This new version of pecvheck.py adds an overview of sections. More details here.

pecheck-v0_7_0.zip (https)
MD5: 7BE550EC71BF99FC31704C2DD4ED3C8A
SHA256: 12C03369362045DF5A9AAB83002E59A4A31050EC008DF45F777C87186D611F6E

In this new version of zipdump.py, you can provide a YARA rule directly on the command line, without having to store it inside a file.

Just start the value of option -y with # and type your rule (use quotes because of spaces):

zipdump_v0_0_9.zip (https)
MD5: 2700AF663980204075107164AA12750A
SHA256: 5686F24373AF64E1F5D866C71B29A22CE97964EC563A2219681A6268CC9A1153

This new version of base64dump.py has a new option: -z. With this option, you can ignore leading null bytes (to be used for example to handle UNICODE).

You can see this option used in this video (starting 1:28):

base64dump_V0_0_7.zip (https)
MD5: D37DE7CEFDA55ADD1822EADDD84D5FFB
SHA256: 5F676DF8B36172A1D7B29F03E2B0CCB026BB9A96DF8830FDB137E65CBB59DD63