Didier Stevens

PDF Tools

pdf-parser.py

This tool will parse a PDF document to identify the fundamental elements used in the analyzed file. It will not render a PDF document. The code of the parser is quick-and-dirty, I’m not recommending this as text book case for PDF parsers, but it gets the job done.

You can see the parser in action in this screencast.

The stats option display statistics of the objects found in the PDF document. Use this to identify PDF documents with unusual/unexpected objects, or to classify PDF documents. For example, I generated statistics for 2 malicious PDF files, and although they were very different in content and size, the statistics were identical, proving that they used the same attack vector and shared the same origin.

The search option searches for a string in indirect objects (not inside the stream of indirect objects). The search is not case-sensitive, and is susceptible to the obfuscation techniques I documented (as I’ve yet to encounter these obfuscation techniques in the wild, I decided no to resort to canonicalization).

filter option applies the filter(s) to the stream. For the moment, only FlateDecode is supported (e.g. zlib decompression).

The raw option makes pdf-parser output raw data (e.g. not the printable Python representation).

objects outputs the data of the indirect object which ID was specified. This ID is not version dependent. If more than one object have the same ID (disregarding the version), all these objects will be outputted.

reference allows you to select all objects referencing the specified indirect object. This ID is not version dependent.

type alows you to select all objects of a given type. The type is a Name and as such is case-sensitive and must start with a slash-character (/).

Download:

pdf-parser_V0_3_7.zip (https)

MD5: BDC0E5A82EB6D7C287E7360D8901023D

SHA256: C83D39F8938A00A3EB2BDE3134EFAF3A2BE11E72C2C8A92841D4E1E82366D7E1

make-pdf tools
make-pdf-javascript.py allows one to create a simple PDF document with embedded JavaScript that will execute upon opening of the PDF document. It’s essentially glue-code for the mPDF.py module which contains a class with methods to create headers, indirect objects, stream objects, trailers and XREFs.

20081109-134003

If you execute it without options, it will generate a PDF document with JavaScript to display a message box (calling app.alert).

To provide your own JavaScript, use option –javascript for a script on the command line, or –javascriptfile for a script contained in a file.

Download:

make-pdf_V0_1_1.zip (https)

MD5: 9AF2E343B78553021C989E8E22355531

SHA256: C604679ABEB0469C1463159E02E74F12487B2755A6096B416A8F4F638DEB8AA9

pdfid.py
This tool is not a PDF parser, but it will scan a file to look for certain PDF keywords, allowing you to identify PDF documents that contain (for example) JavaScript or execute an action when opened. PDFiD will also handle name obfuscation.

The idea is to use this tool first to triage PDF documents, and then analyze the suspicious ones with my pdf-parser.

An important design criterium for this program is simplicity. Parsing a PDF document completely requires a very complex program, and hence it is bound to contain many (security) bugs. To avoid the risk of getting exploited, I decided to keep this program very simple (it is even simpler than pdf-parser.py).

20090330-214223

PDFiD will scan a PDF document for a given list of strings and count the occurrences (total and obfuscated) of each word:

  • obj
  • endobj
  • stream
  • endstream
  • xref
  • trailer
  • startxref
  • /Page
  • /Encrypt
  • /ObjStm
  • /JS
  • /JavaScript
  • /AA
  • /OpenAction
  • /JBIG2Decode
  • /RichMedia

Almost every PDF documents will contain the first 7 words (obj through startxref), and to a lesser extent stream and endstream. I’ve found a couple of PDF documents without xref or trailer, but these are rare (BTW, this is not an indication of a malicious PDF document).

/Page gives an indication of the number of pages in the PDF document. Most malicious PDF document have only one page.

/Encrypt indicates that the PDF document has DRM or needs a password to be read.

/ObjStm counts the number of object streams. An object stream is a stream object that can contain other objects, and can therefor be used to obfuscate objects (by using different filters).

/JS and /JavaScript indicate that the PDF document contains JavaScript. Almost all malicious PDF documents that I’ve found in the wild contain JavaScript (to exploit a JavaScript vulnerability and/or to execute a heap spray). Of course, you can also find JavaScript in PDF documents without malicious intend.

/AA and /OpenAction indicate an automatic action to be performed when the page/document is viewed. All malicious PDF documents with JavaScript I’ve seen in the wild had an automatic action to launch the JavaScript without user interaction.

The combination of automatic action  and JavaScript makes a PDF document very suspicious.

/JBIG2Decode indicates if the PDF document uses JBIG2 compression. This is not necessarily and indication of a malicious PDF document, but requires further investigation.

/RichMedia is for embedded Flash.

A number that appears between parentheses after the counter represents the number of obfuscated occurrences. For example, /JBIG2Decode 1(1) tells you that the PDF document contains the name /JBIG2Decode and that it was obfuscated (using hexcodes, e.g. /JBIG#32Decode).

BTW, all the counters can be skewed if the PDF document is saved with incremental updates.

Because PDFiD is just a string scanner (supporting name obfuscation), it will also generate false positives. For example, a simple text file starting with %PDF-1.1 and containing words from the list will also be identified as a PDF document.

Download:

pdfid_v0_0_10.zip (https)

MD5: A06B023457DACE24FDFBF537282E1A76

SHA256: 18D88B15C90504BE6A2FF2814BD15A7B20B945337252018A0072AEFD99D5AAC8

58 Comments »

  1. [...] PDF, Quickpost — Didier Stevens @ 11:57 Per request, a more detailed post on how I use my pdf-parser stats [...]

    Pingback by Quickpost: Fingerprinting PDF Files « Didier Stevens — Saturday 1 November 2008 @ 11:57

  2. I’d like to be able to view a scanned pdf file (with handwriting in some fields) and black
    out boxes on the form whose fields contain info I don’t want published.

    Can that kind of thing be automated in a batch so that I don’t even have to open the files ?

    That would be cool …

    Can you point me in the right direction ? I’m not looking for you to code, but sending me in the right direction for this would be useful, and it looks like you’re cognizant of this kind of information.

    Comment by james — Friday 21 November 2008 @ 15:03

  3. @james

    I’ve no experience with such tools, but you can start to look in the forum of PDF Planet.

    Comment by Didier Stevens — Saturday 22 November 2008 @ 8:46

  4. [...] download my Python program to generate these PoC PDF documents here, it needs the mPDF module of my PDF-tools. Quickpost info Possibly related posts: (automatically generated)PDF Stream ObjectsUsing DLE In An [...]

    Pingback by Quickpost: /JBIG2Decode Essentials « Didier Stevens — Monday 2 March 2009 @ 23:12

  5. [...] PDF — Didier Stevens @ 7:08 I’ve developed a new tool to triage PDF documents, PDFiD. It helps you differentiate between PDF documents that could be malicious and those that are most [...]

    Pingback by PDFiD « Didier Stevens — Tuesday 31 March 2009 @ 7:08

  6. Thanks Didier for sharing _yet_again_ I appreciate it very much!

    Comment by Mitch Impey — Wednesday 1 April 2009 @ 20:17

  7. [...] developed a new tool to triage PDF documents, PDFiD. It helps you differentiate between PDF documents that could be malicious and those that are most [...]

    Pingback by Didier Stevens posted PDFid « Betterworldforus — Friday 3 April 2009 @ 18:34

  8. [...] will give you statistics of some very basic elements of the PDF language. This helps you decide if a PDF could be malicious or [...]

    Pingback by PDFiD On VirusTotal « Didier Stevens — Tuesday 21 April 2009 @ 16:59

  9. [...] We can go a number of ways with this now, but since the point of all this is the fun we can have with obfuscated scripts in Adobe PDFs we’ll run it through PDFiD from Didier Stevens. [...]

    Pingback by L1pht Offensive Labs » From Bloodhound to Acrobat JS — Saturday 25 April 2009 @ 2:59

  10. Hello — I am using pdf-parser and python for the first time so please excuse my ignorance.

    I’m using Python 3.0.1 on Windows XP. I’ve copied the pdf-parser.py file into the C:\Python30 directory which contains the python executable. Below is the error I get when attempting to execute your utility:

    C:\Python30>python.exe pdf-parser.py
    File “pdf-parser.py”, line 180
    print ‘todo 1: %s’ % (self.token[1] + self.token2[1])
    ^
    SyntaxError: invalid syntax

    ————

    Any ideas? Thanks for your time.

    Comment by Mike — Tuesday 5 May 2009 @ 19:02

  11. @rMike No problem.

    I’ve not tested my Python tools on Python 3

    You should remove Python 3.0.1 and install Python 2.6. This will probably fix your problems.

    I’ve still to decide when I upgrade my tools to Python 3

    Comment by Didier Stevens — Tuesday 5 May 2009 @ 20:59

  12. Hello — I’m now using Python 2.6.2. It appears to be working, however I am getting so much output from every pdf I examine, I wonder if I am doing something wrong.

    My syntax is

    pdf-parser.py –search javascript malware.pdf

    The utility spits out hundreds maybe thousands of lines of returned information. At the very top there appears to be useful data, however there are hundreds of lines that look like:

    todo 10: 3 ‘X1\x1e\x1b\x03\x12\x05X60B

    Am I doing something incorrectly here or is there a way to filter the rest of this data out?

    Thanks for your help.

    PS, I watched your video on pdf-parser and it doesn’t have any audio.

    Comment by Mike — Thursday 7 May 2009 @ 20:40

  13. Also, just a note I am do have two hyphens in front of search (–search)

    Comment by Mike — Thursday 7 May 2009 @ 20:41

  14. No, what you’re doing is correct. These todo 10 messages indicate that your PDF document is possibly corrupt. I’ve seen one such corrupt document before, the malware authors had inserted their payload in the PDF without respecting the PDF syntax.

    I’m sending you an e-mail to see if you can share your sample with me.

    About the video: most of my videos have no sound, it takes me much more time to produce one with audio, and I’m never satisfied with the result.

    Comment by Didier Stevens — Thursday 7 May 2009 @ 21:07

  15. [...] updated my PDF-tools to support [...]

    Pingback by PDF Filter Abbreviations « Didier Stevens — Monday 11 May 2009 @ 0:01

  16. Didier,
    Any idea whether/when Filiol, Blonce, & Frayssignes might release their ‘PDF StructAzer’ tool? My Google research doesn’t show anything from them since their presentation at Blackhat Europe last year. According to that paper, they planned to release the tool “very soon”, but maybe they’ve run into some red tape with their employer over it.
    John

    Comment by John McCash — Tuesday 12 May 2009 @ 13:03

  17. Hi John,

    No, still no news about a release, but I’ll ask him again.

    Comment by Didier Stevens — Tuesday 12 May 2009 @ 13:51

  18. [...] added some new features to my PDF tools to handle malformed PDF [...]

    Pingback by Malformed PDF Documents « Didier Stevens — Thursday 14 May 2009 @ 7:55

  21. [...] Finding and detecting Malicious PDF’s. Tyler’s Snort signature. Didier Steven’s fantastic PDF analysis tools. [...]

    Pingback by Security Justice » Blog Archive » Security Justice - Episode 13 — Saturday 6 June 2009 @ 2:30

  22. Didier,
    Does your script pdf-parser.py works on Encrypted PDF streams ?

    Comment by NeoIsOffline — Tuesday 14 July 2009 @ 12:28

  23. No, I didn’t add code to decrypt PDF streams. And I’m not sure if I ever will, because then it could also be considered as a copyright infringement tool, and I don’t want to deal with that.

    Comment by Didier Stevens — Tuesday 14 July 2009 @ 19:13

  24. Im getting errors when running the python script:

    C:\Documents and Settings\yo\Desktop\Tools\pdf>pdf-parser.py
    File “C:\Documents and Settings\yo\Desktop\Tools\pdf\pdf-parser.py”, line 198
    print ‘todo 1: %s’ % (self.token[1] + self.token2[1])
    ^
    SyntaxError: invalid syntax

    Im getting errors when trying to run the script. Im using activepyton 3.1 on windows xp. Launching it from the commandline. Was there any recent modifications which broke the script?

    Thanks

    Comment by Dave — Friday 17 July 2009 @ 16:48

  25. @Dave

    That looks the same error as in comment 10. Read comments 10 to 12 for a solution.

    Comment by Didier Stevens — Friday 17 July 2009 @ 17:12

  26. wow I feel ignorant, i just breezed through the comments fast. Thanks!

    Comment by Dave — Friday 17 July 2009 @ 18:55

  27. @Dave

    No problem.

    Comment by Didier Stevens — Friday 17 July 2009 @ 18:58

  28. Hello Didier,

    This tool is very cool, I am wondering how to integrate this to the ironport (mail filter) so that all attachments like pdf will be scanned and if found that there are openaction or javascript then probably we can filter that. Also, maybe this can be integrated in the Proxy or firewall. Do you have any link to see a topic on integrating this tool.

    Thank you very much, indeed this is very helpful

    Comment by Yaggi — Tuesday 28 July 2009 @ 0:12

  29. What interface options does ironport offer? Does it support ICAP? http://en.wikipedia.org/wiki/Internet_Content_Adaptation_Protocol

    Comment by Didier Stevens — Tuesday 28 July 2009 @ 18:55

  30. Hello Didier,

    I talked to our notes admin and this can somehow be integrated in the sendmail. He ask me some technicalities on the integration. maybe you can also provide some link. HE told me ironport will not be used.

    Where can we possibly put this script for internet filter, is it in the proxy server or possibly to content filter boxes like from 8e6 technologies? Im hoping to finish this integration so we can maximize your script.

    Comment by Yaggi — Wednesday 29 July 2009 @ 1:30

  31. @Yaggi

    How do you interface with other scanner, like AV software?

    Comment by Didier Stevens — Friday 31 July 2009 @ 12:48

  32. [...] Taking a quick look at the rest of the file it was clear that it is just a “simple” exploit using obfuscated javascript. So I extracted the scripts with the pdf-tools from Didier Stevens. [...]

    Pingback by hong10.net » Blog Archive » Analyzing malicious PDF Documents — Monday 3 August 2009 @ 21:28

  33. I was looking over your code in the pdf parser, and trying very very hard to get it to extract something from a pdf. I have tried using pdfs I’ve been given, as well as going directly into the files to try to mess things up, and I have yet to see the code go into the cPDFElementMalformed method. What kinds of data (objects?) do you expect to fall into this category?

    Thank you -ld

    Comment by Delucci — Tuesday 4 August 2009 @ 14:16

  34. @Delucci

    It’s for sometinh like this:

    %PDF-1.4

    1 0 obj
    <>
    endobj

    unexpected

    2 0 obj
    <>
    endobj

    Comment by Didier Stevens — Tuesday 4 August 2009 @ 15:33

  35. [...] the PDF analysis, I used the excellent PDF-Tools from Didier Stevens that can be located here. The main python script that was used was pdf-parser seen [...]

    Pingback by PDF Malware Analysis – Part 1 | isolated-threat — Tuesday 18 August 2009 @ 21:02

  36. [...] can download PDFiD here. Leave a [...]

    Pingback by Update: PDFiD Version 0.0.9 to Detect Another Adobe 0Day « Didier Stevens — Tuesday 13 October 2009 @ 21:24

  37. [...] Here is a set of tools that can embed Javascript into a PDF. [...]

    Pingback by Adobe Reader 0-day exploit FINALLY fixed. | Invariable Truth — Sunday 18 October 2009 @ 9:35

  38. Hi,

    Great set of tools, I have noticed while checking PDF files for embedded links (i.e. URI tags), some PDFs contain hyperlinks but do not contain the URI tag, is there an alternative method of embedding hyperlinks, should I be searching for some other keyword?

    Comment by Tye — Monday 19 October 2009 @ 15:41

  39. Check if the URLs you see are in the metadata.

    Comment by Didier Stevens — Tuesday 20 October 2009 @ 16:48

  40. Hi,

    I’ve been using your tool to decode the malicious PDF file that I have. It’s using /Filter /ASCIIHexDecode /FlateDecode.

    I used the following command:
    pdf-parser.py -f -w malpdf.1 > mal.1

    The resulting file didn’t show any JavaScript code, instead it showed “ASCIIHexDecode decompress failed”.

    Wepawet is able to decode it though (http://wepawet.cs.ucsb.edu/view.php?hash=c9aad1ecee10ddcf1985ae4961e18fbf&type=js).

    Are my parameters for the tool incorrect? Or doesn’t the tool support this?

    Thanks in advance.

    Comment by anima — Friday 30 October 2009 @ 5:40

  41. @anima

    I’ve e-mailed you a request for the sample.

    Comment by Didier Stevens — Thursday 5 November 2009 @ 17:45

  42. [...] my method: Use the tools from here. First of all pdfid can tell you if a pdf has Javascript included as well as autorun functionality [...]

    Pingback by PDF file check question - Remote Exploit Forums — Sunday 6 December 2009 @ 22:25

  43. [...] first tool we’ll be using is pdf-parser.py from the PDF Tools suite.  This script will search through a PDF file’s sections, display raw data in the sections, [...]

    Pingback by Reversing the Adobe 0-day APSA09-07 Exploit – Part 1 | Missouri S&T ACM SIG-SEC|Reversing — Wednesday 16 December 2009 @ 3:55

  44. [...] Countermeasures __________________ Either you're part of the problem or you're part of the solution or you're just part of the landscape. [...]

    Pingback by Using-an-adobe-exploit-in-a-email-attack - Remote Exploit Forums — Tuesday 22 December 2009 @ 15:46

  45. [...] pdf-parser.py http://blog.didierstevens.com/programs/pdf-tools/ Lets decompression some of the zlib compressed code inside of the PDF and send the raw output to a [...]

    Pingback by Reversing MerryChristmas.pdf - Sp8sCorp — Thursday 31 December 2009 @ 5:01

  46. [...] pdf-parser.py or PDF Structazer to analyze PDF files [...]

    Pingback by Can You Trust That File? « Aggressive Virus Defense — Thursday 31 December 2009 @ 22:40

  47. [...] [...]

    Pingback by How to encode a PDF payload in metasploit? - Remote Exploit Forums — Tuesday 5 January 2010 @ 14:10

  48. [...] thanks to Didier Stevens for his free PDF tools and for providing some [...]

    Pingback by PDF file loader to extract and analyse shellcode « c0llateral Blog — Wednesday 6 January 2010 @ 23:19

  49. Hey Didier,
    Thanks for excellent tool and great PDF analysis blog. I enjoyed every minute and in addition I have become much more paranoid when it comes to carelessly downloading tons of PDF material. Now I run all my PDFs through your “pdfid” tool, if I have downloaded anything from a suspicious site…

    But I can’t help thinking that this should be implemented as an automatic plug-in/add-on to Firefox? You know, when you click on PDFs, they usually automatically open in the browser, which is nice if it was safe. But in the cyber-war era of today it is simply very bad, at it’s best!

    Comment by E:V:A — Saturday 16 January 2010 @ 18:40

  50. I’m looking into this, but the problem is to prevent the download PDF from being opened after it’s downloaded and before it’s scanned. I talked to the developer of the Fireclam add-on and he has the same issue.

    Comment by Didier Stevens — Tuesday 19 January 2010 @ 9:29

  51. Only thing i get are syntax error!

    C:\pdfid_v0_0_10>pdfid.py
    File “C:\pdfid_v0_0_10\pdfid.py”, line 271
    print ‘/%s -> /%s’ % (HexcodeName2String(wordExact), wordExactSwapped)

    Comment by sheldor — Sunday 31 January 2010 @ 22:18

  52. Are you using Python 3? Haven’t tested PDFiD on Python 3. Use Python 2.

    Comment by Didier Stevens — Sunday 31 January 2010 @ 22:20

  53. got it!! just read the comments!

    Comment by sheldor — Sunday 31 January 2010 @ 22:35

  54. whow just noticed your quick response! thank you didier! great tool!

    Comment by sheldor — Sunday 31 January 2010 @ 22:36

  55. Is the File Size limited? Everytime i scan larger PDF files i get exceptions like this:

    ***Error occured***
    Traceback (most recent call last):
    File “C:\PDFtools\pdfid.py”, line 363, in PDFiD
    (bytesHeader, pdfHeader) = FindPDFHeaderRelaxed(oBinaryFile)
    File “C:\PDFtools\pdfid.py”, line 218, in FindPDFHeaderRelaxed
    bytes = oBinaryFile.bytes(1024)
    File “C:\PDFtools\pdfid.py”, line 70, in bytes
    inbytes = self.infile.read(size – len(self.ungetted))
    IOError: [Errno 9] Bad file descriptor

    Comment by sheldor — Monday 8 February 2010 @ 13:50

  56. @sheldor: No, I didn’t code an explicit file size limit. I tried on PDF files up to 41MB without problems. How large is your PDF file?

    Comment by Didier Stevens — Monday 8 February 2010 @ 14:55

  57. I have this issue with PDFs 20MB and up! Well,.. then there must be another reason! Still can’t figure it out!
    Anyhow, thank you!

    Comment by sheldor — Monday 8 February 2010 @ 17:27

  58. @sheldor: If you can point me to an online PDF document that causes the problem you experience, I’ll take a look at it.

    Comment by Didier Stevens — Monday 8 February 2010 @ 20:24


RSS feed for comments on this post. TrackBack URI

Leave a comment

Blog at WordPress.com.