Out-of-bounds read during processing of a password-protected PDF file

[Nils]

Out-of-bounds read during processing of a password-protected PDF file

# Description
During processsing of the attached pdf file via
```
pdftotext -upw USERPASS -opw OWNERPASS $PWD/testcase /tmp/out.txt
```
a out-of-bounds read happens. Since I was unable to reproduce this bug
on the most recent upstream commit (b3f93644de4941bdbd532a7d8f82cd652dfbeadf), I report it here.

This bug allows an attacker to perform a denial of service and possibly opens up other attack vectors.

To reproduce the crash, we provide the following script alongside the crashing input:
- ./reproduce-ubuntu.sh: Reproduce crash via a Ubuntu 20.04 docker container

If you need further details, we are happy to answer all questions.

# apt show poppler-utils
Package: poppler-utils
Version: 0.86.1-0ubuntu1
Priority: optional
Section: utils
Source: poppler
Origin: Ubuntu
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: Debian freedesktop.org maintainers <email address hidden>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 754 kB
Provides: pdftohtml, xpdf-utils
Depends: libpoppler97 (= 0.86.1-0ubuntu1), libc6 (>= 2.14), libcairo2 (>= 1.12.0), libfreetype6 (>= 2.2.1), liblcms2-2 (>= 2.2+git20110628), libstdc++6 (>= 5.2)
Conflicts: pdftohtml
Breaks: xpdf-common, xpdf-utils (<< 1:0)
Replaces: pdftohtml, xpdf-reader, xpdf-utils (<< 3.02-2~)
Homepage: http://poppler.freedesktop.org/
Task: print-server, ubuntu-desktop-minimal, ubuntu-desktop, kubuntu-desktop, xubuntu-core, xubuntu-desktop, lubuntu-desktop, ubuntustudio-desktop-core, ubuntustudio-desktop, ubuntukylin-desktop, ubuntu-mate-core, ubuntu-mate-desktop, ubuntu-budgie-desktop
Download-Size: 174 kB
APT-Manual-Installed: no
APT-Sources: http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
Description: PDF utilities (based on Poppler)

# valgrind Ubuntu
==1== Memcheck, a memory error detector
==1== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1== Command: pdftotext -upw USERPASS -opw OWNERPASS /testcase /tmp/out.txt
==1==
Syntax Error (409): Dictionary key must be a name object
Syntax Error (796): Illegal character <29> in hex string
Syntax Error (798): Illegal character <14> in hex string
Syntax Error (799): Illegal character <d3> in hex string
Syntax Error (800): Illegal character <d7> in hex string
Syntax Error (801): Illegal character <8a> in hex string
Syntax Error (860): Illegal character <58> in hex string
Unimplemented Feature: Unsupported version/revision (1/0) of Standard security handler
==1== Invalid read of size 8
==1== at 0x498F758: FilterStream::getDict() (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0B4A9: Parser::makeStream(Object&&, unsigned char const*, CryptAlgorithm, int, int, int, int, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0BE3D: Parser::getObj(bool, unsigned char const*, CryptAlgorithm, int, int, int, int, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x49EE0AE: Hints::readTables(BaseStream*, Linearization*, XRef*, SecurityHandler*) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0C949: PDFDoc::checkLinearization() (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E4A9: PDFDoc::getPage(int) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E88C: PDFDoc::displayPage(OutputDev*, int, double, double, int, bool, bool, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E9E1: PDFDoc::displayPages(OutputDev*, int, int, double, double, int, bool, bool, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x10C57F: main (pdftotext.cc:400)
==1== Address 0x600000005ad8fc0 is not stack'd, malloc'd or (recently) free'd
==1==
==1==
==1== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==1== General Protection Fault
==1== at 0x498F758: FilterStream::getDict() (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0B4A9: Parser::makeStream(Object&&, unsigned char const*, CryptAlgorithm, int, int, int, int, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0BE3D: Parser::getObj(bool, unsigned char const*, CryptAlgorithm, int, int, int, int, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x49EE0AE: Hints::readTables(BaseStream*, Linearization*, XRef*, SecurityHandler*) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0C949: PDFDoc::checkLinearization() (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E4A9: PDFDoc::getPage(int) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E88C: PDFDoc::displayPage(OutputDev*, int, double, double, int, bool, bool, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*, bool) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x4A0E9E1: PDFDoc::displayPages(OutputDev*, int, int, double, double, int, bool, bool, bool, bool (*)(void*), void*, bool (*)(Annot*, void*), void*) (in /usr/lib/x86_64-linux-gnu/libpoppler.so.97.0.0)
==1== by 0x10C57F: main (pdftotext.cc:400)
==1==
==1== HEAP SUMMARY:
==1== in use at exit: 389,676 bytes in 5,022 blocks
==1== total heap usage: 5,512 allocs, 490 frees, 904,123 bytes allocated
==1==
==1== LEAK SUMMARY:
==1== definitely lost: 72 bytes in 1 blocks
==1== indirectly lost: 0 bytes in 0 blocks
==1== possibly lost: 0 bytes in 0 blocks
==1== still reachable: 389,604 bytes in 5,021 blocks
==1== suppressed: 0 bytes in 0 blocks
==1== Rerun with --leak-check=full to see details of leaked memory
==1==
==1== For lists of detected and suppressed errors, rerun with: -s
==1== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

valgrind: the 'impossible' happened:
   main(): signal was supposed to be fatal

host stacktrace:
==1== at 0x58046FFA: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)
==1== by 0x58047127: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)
==1== by 0x58047390: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)
==1== by 0x580473C0: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)
==1== by 0x580BA566: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)
==1== by 0x580F6117: ??? (in /usr/lib/x86_64-linux-gnu/valgrind/memcheck-amd64-linux)

sched status:
  running_tid=1