apt-get fails if a package has a space in its Filename

Bug #1086997 reported by Thomas Bushnell, BSG
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned

Bug Description

If a package has a space in its Filename: entry in the Packages file, you get errors on HTTP resolution.

Normal Debian repos don't have a case like that, but the Packages file should work even if packages are not stored in the pool in the most usual way.

Related branches

CVE References

Revision history for this message
Daniel Hartwig (wigs) wrote : Re: [Bug 1086997] [NEW] apt-get fails if a package has a space in its Filename

On 6 December 2012 05:09, Thomas Bushnell, BSG
<email address hidden> wrote:
> If a package has a space in its Filename: entry in the Packages file,
> you get errors on HTTP resolution.

Why are there spaces, is it due to a service that munges filenames
with “s/+/ /g” on upload? Provide example. Issue on Amazon S3 has
already been reported; is a wontfix bug in their system; and has the
work-around of percent-encoding the filename during upload (IIRC).

> Normal Debian repos don't have a case like that, but the Packages file
> should work even if packages are not stored in the pool in the most
> usual way.

There is very little restriction on directory structure, however,
format of filenames is effectively restricted by the combination of
debian-policy and the names that dpkg-deb generates. Note that
dpkg-deb is the only supported tool for creating binary packages.
Other tools rely on the naming scheme used by dpkg-deb, especially in
contexts where Filename is not available.

Further, filenames with spaces can not appear in the Filename field as
space is used to separate multiple filenames.

Ref: debian-policy appendices B through D

Revision history for this message
Thomas Bushnell, BSG (tbushnell) wrote :

Multiple filenames? Policy? Binary packages do not have filename fields at
all. I'm talking about Filename fields in the Packages file in an archive.
I'm not talking about S3 at all.

dpkg-deb only generates the last component of a filename, the problem as I
encountered it concerned a repo with spaces in some directory names. At the
same time, there is no requirement that the package filename last component
be in the same style as dpkg-deb generates, though that's not a concern
here.

Debian Policy doesn't address Packages files at all, nor archive layout.
On Dec 5, 2012 5:50 PM, "Daniel Hartwig" <email address hidden> wrote:

> On 6 December 2012 05:09, Thomas Bushnell, BSG
> <email address hidden> wrote:
> > If a package has a space in its Filename: entry in the Packages file,
> > you get errors on HTTP resolution.
>
> Why are there spaces, is it due to a service that munges filenames
> with “s/+/ /g” on upload? Provide example. Issue on Amazon S3 has
> already been reported; is a wontfix bug in their system; and has the
> work-around of percent-encoding the filename during upload (IIRC).
>
> > Normal Debian repos don't have a case like that, but the Packages file
> > should work even if packages are not stored in the pool in the most
> > usual way.
>
> There is very little restriction on directory structure, however,
> format of filenames is effectively restricted by the combination of
> debian-policy and the names that dpkg-deb generates. Note that
> dpkg-deb is the only supported tool for creating binary packages.
> Other tools rely on the naming scheme used by dpkg-deb, especially in
> contexts where Filename is not available.
>
> Further, filenames with spaces can not appear in the Filename field as
> space is used to separate multiple filenames.
>
> Ref: debian-policy appendices B through D
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1086997
>
> Title:
> apt-get fails if a package has a space in its Filename
>
> Status in “apt” package in Ubuntu:
> New
>
> Bug description:
> If a package has a space in its Filename: entry in the Packages file,
> you get errors on HTTP resolution.
>
> Normal Debian repos don't have a case like that, but the Packages file
> should work even if packages are not stored in the pool in the most
> usual way.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1086997/+subscriptions
>

Revision history for this message
Daniel Hartwig (wigs) wrote :

On 6 December 2012 10:00, Thomas Bushnell, BSG
<email address hidden> wrote:
> the problem as I
> encountered it concerned a repo with spaces in some directory names.

> Debian Policy doesn't address Packages files at all, nor archive layout.

Appendix D, Control files and their fields?

Revision history for this message
Daniel Hartwig (wigs) wrote :

Also, there is this statement in section 5.1:

> Whitespace must not appear inside names (of packages,
> architectures, files or anything else) or version numbers, or
> between the characters of multi-character version
> relationships.

Revision history for this message
Thomas Bushnell, BSG (tbushnell) wrote :

Thanks for the reference. I apologize for missing section D; it's not
relevant here, but I had forgotten that it has those things. It doesn't
actually say much relevant here.

However, I think section 5.1 is dispositive, so this isn't a bug. It's an
unfortunate restriction, because it ends up meaning that there is no way to
use a directory name with spaces, especially given that the repository
format documentation says that the Filename: "should not make user of any
protocol-specific components, such as URL-encoded parameters", so you can't
put in URL escaping syntax either.

It's worth noting what happens when this is triggered however. It's not
that apt-get splits it into multiple words and forgets part (which 5.1
might suggest); it's that apt-get doesn't form a proper HTTP request
because it passes the spaces through in the HTTP GET where they aren't
allowed. Given that 5.1 says that such a Filename field is misformed, apt
can't be blamed for failing to interpret it correctly.

But this suggests that there isn't any good escaping going on between
Filename: and the HTTP server. Since *any *character--except whitespace--is
permitted in the Filename (as long as it isn't trying to use
"protocol-specific components") it seems that this might run afoul (for
example, semicolons might well cause problems as well given the language of
RFC 2396).

I think apt-get would be very well served to escape Filename:'s when passed
to http or https handlers, and carefully not when passed to file: handlers,
etc. I suspect there are other bugs here which 5.1 is not relevant for,
since it only addresses the specific case of whitespace.

Thomas

On Wed, Dec 5, 2012 at 6:42 PM, Daniel Hartwig <email address hidden> wrote:

> Also, there is this statement in section 5.1:
>
> > Whitespace must not appear inside names (of packages,
> > architectures, files or anything else) or version numbers, or
> > between the characters of multi-character version
> > relationships.
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1086997
>
> Title:
> apt-get fails if a package has a space in its Filename
>
> Status in “apt” package in Ubuntu:
> New
>
> Bug description:
> If a package has a space in its Filename: entry in the Packages file,
> you get errors on HTTP resolution.
>
> Normal Debian repos don't have a case like that, but the Packages file
> should work even if packages are not stored in the pool in the most
> usual way.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1086997/+subscriptions
>

Revision history for this message
Daniel Hartwig (wigs) wrote :

On 6 December 2012 12:05, Thomas Bushnell, BSG
<email address hidden> wrote:
> Thanks for the reference.

Yes, who doesn't enjoy a good read of policy documents?

> I apologize for missing section D; it's not
> relevant here, but I had forgotten that it has those things. It doesn't
> actually say much relevant here.

I'd say it's still relevant:
- although not included in debian-policy, Packages is still a control file;
- section 5 defines the syntax of control files, including valid
values for a filename;
- appendix D defines the syntax of Filename fields in control files.

So yes, unless the apendices are intended to be non-normative :-)

Anyway, 5.1 does include the no-whitespace-in-filenames restriction.
Unfortunate for your situation, which would require updating
debian-policy to permit including whitespace some how. Given how many
utilities and parsers using control files there are, I don't know how
manageable such a change would be.

>
> However, I think section 5.1 is dispositive, so this isn't a bug. It's an
> unfortunate restriction, because it ends up meaning that there is no way to
> use a directory name with spaces,

>
> It's worth noting what happens when this is triggered however. It's not
> that apt-get splits it into multiple words and forgets part (which 5.1
> might suggest);

[It is appendix D that indicates a Filename /field/ is split on
whitespace. Section 5.1 only states a filename /value/ with spaces is
not allowed in control files.]

> it's that apt-get doesn't form a proper HTTP request
> because it passes the spaces through in the HTTP GET where they aren't
> allowed. Given that 5.1 says that such a Filename field is misformed, apt
> can't be blamed for failing to interpret it correctly.

Regardless of how debian-policy defines valid filenames /in control
files/, the APT Acquire system and HTTP method can be used outside the
context of debian control files, and should still issue
correctly-formed HTTP requests and save the files properly.

So I'd say that is a bug, and superficially related to bug #1003633
(although there is no requirement to percent-encode “+” in a URI).

There is a further bug here: APT is incorrectly parsing the Filename
field in Packages (according to appendix D). Filed as bug #1087129.

> I think apt-get would be very well served to escape Filename:'s when passed
> to http or https handlers, and carefully not when passed to file: handlers,
> etc. I suspect there are other bugs here which 5.1 is not relevant for,
> since it only addresses the specific case of whitespace.

Yes, I agree. And fixing the HTTP headers would probably resolve your
problem although the Packages file would still be malformed for
containing such filenames. That issue would not be fixed without a
corresponding update to debian-policy.

Anyway, I wish you well in wrestling with these server paths.

Revision history for this message
Daniel Hartwig (wigs) wrote :

On 6 December 2012 13:27, Daniel Hartwig <email address hidden> wrote:
> Regardless of how debian-policy defines valid filenames /in control
> files/, the APT Acquire system and HTTP method can be used outside the
> context of debian control files, and should still issue
> correctly-formed HTTP requests and save the files properly.

Acquire system should be ok. Attaching trivial patch for
methods/http.cc. Rationale: space (#x20) is a delimiter in the
context of a HTTP Request-Line [1] and therefore must be
percent-encoded in the Request-URI.

[1] <http://tools.ietf.org/html/rfc2616#section-5.1>

Revision history for this message
Thomas Bushnell BSG (tb-becket) wrote :

What I meant by saying that appendix D wasn't relevant was only that it doesn't say anything particularly helpful about the syntax of a Filename: field.

I agree that any attempt to modify 5.1 is pointless, and it's not any trouble in my particular case to simply conform more closely to the typical way of construction repos.

You should feel free to deal with this issue and related ones as you see fit. Obviously apt-get is only particularly well tested on Filename: values that are "pool/f/foo/foo..." in form, for the obvious reason: that's what everyone writes. And in my case, there was an independent bug *producing* the directories with spaces in them which I did not intend. So solving this bug is not necessary to addressing any operational difficulty I'm encountering; I reported it only because it seemed a risk for other cases more important than spaces in filenames.

Revision history for this message
Daniel Hartwig (wigs) wrote : Re: [Bug 1086997] Re: apt-get fails if a package has a space in its Filename

On 6 December 2012 14:06, Thomas Bushnell BSG
<email address hidden> wrote:
> And in my case, there was an independent
> bug *producing* the directories with spaces in them which I did not
> intend.

Ah, I had thought you were stuck with those directories!

Revision history for this message
Thomas Bushnell, BSG (tbushnell) wrote :

No, certainly not. The details are embarrassing for a public bug so I was
being cagey. :)

I was parsing Source: lines incorrectly. The policy manual says that
version numbers on a Source line are allowed in .changes and .deb controls,
but not .dsc. This makes sense; for .debs in particular they are necessary
for binary NMUs.

But I misread that, as if the Source: line was allowed only for .changes
files. And so when I encountered a .deb that had a Source: line with a
version in it, I created a pool directory with the "wrong" name. So if
there was:

Package: foo
Version: 1.9+1
Source: foo (1.9)

I would create "pool/main/f/foo (1.9)/foo_1.9+1_amd64.deb" and create a
Filename: argument to match.

The problem with apt-get handling of this was actually a happy accident,
because it highlighted this case so I could fix it!

Thomas

On Wed, Dec 5, 2012 at 10:25 PM, Daniel Hartwig <email address hidden> wrote:

> On 6 December 2012 14:06, Thomas Bushnell BSG
> <email address hidden> wrote:
> > And in my case, there was an independent
> > bug *producing* the directories with spaces in them which I did not
> > intend.
>
> Ah, I had thought you were stuck with those directories!
>
> --
> You received this bug notification because you are a member of Goobuntu
> Team, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1086997
>
> Title:
> apt-get fails if a package has a space in its Filename
>
> Status in “apt” package in Ubuntu:
> New
>
> Bug description:
> If a package has a space in its Filename: entry in the Packages file,
> you get errors on HTTP resolution.
>
> Normal Debian repos don't have a case like that, but the Packages file
> should work even if packages are not stored in the pool in the most
> usual way.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1086997/+subscriptions
>

Revision history for this message
Michael Vogt (mvo) wrote :

Thanks for the bugreport and the patch.

Given the trivial nature of the patch and that indeed http is used outside of a strict packages context sometimes (e.g. changelogs and in the future potentially more) I applied it to the bzr tree and it will be part of the next upload.

Changed in apt (Ubuntu):
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (13.3 KiB)

This bug was fixed in the package apt - 0.9.9.1~ubuntu1

---------------
apt (0.9.9.1~ubuntu1) saucy; urgency=low

  * merged from the debian/sid branch:
    - debian/gbp.conf: change build branch to ubuntu/master
    - use ubuntu keyring and ubuntu archive keyring in apt-key
    - run update-apt-xapian-index in apt.cron
    - run apt-key net-update in cron.daily
    - different example sources.list
    - APT::pkgPackageManager::MaxLoopCount set to 5000
    - apport pkgfailure handling
    - ubuntu changelog download handling
    - patch for apt cross-building, see http://bugs.debian.org/666772
    - debian/apt.auto-removal.sh
      + make kernels auto-removable

apt (0.9.9.1) UNRELEASED; urgency=low

  * debian/rules:
    - call dh_clean in clean (closes: #714980)

apt (0.9.9) unstable; urgency=low

  [ Michael Vogt ]
  * improve debug output for the Debug::pkgProblemResolver and
    Debug::pkgDepCache::AutoInstall
  * improve apt-cdrom output when no CD-ROM can be auto-detected
  * document --no-auto-detect in apt-cdrom

  [ David Kalnischkies ]
  * build the en manpages in subdirectory doc/en
  * remove -ldl from cdrom and -lutil from apt-get linkage
  * rewrite pkgOrderList::DepRemove to stop incorrect immediate setting
    (Closes: 645713)
  * prefer Essentials over Removals in ordering score
  * fix priority sorting by prefering higher in MarkInstall
  * try all providers in order if uninstallable in MarkInstall
  * do unpacks before configures in SmartConfigure (Closes: #707578)
  * fix support for multiple patterns in apt-cache search (Closes: #691453)
  * set Fail flag in FileFd on all errors consistently
  * don't explicitly init ExtractTar InFd with invalid fd
  * OpenDescriptor should autoclose fd always on error (Closes: #704608)
  * fail in CopyFile if the FileFds have error flag set
  * ensure state-dir exists before coyping cdrom files
  * fix file location for configure-index.gz in apt.conf(5) (Closes: #711921)
  * handle missing "Description" in apt-cache show (Closes: #712435)
  * try defaults if auto-detection failed in apt-cdrom (Closes: #712433)
  * support \n and \r\n line endings in ReadMessages
  * do not redownload unchanged InRelease files
  * trigger NODATA error for invalid InRelease files (Closes: #712486)

apt (0.9.8.2) unstable; urgency=low

  [ Programs translations ]
  * French translation : typo fix. Closes: #677272

  [ Guillem Jover ]
  * Update Vcs fields (Closes: #708562)

  [ Michael Vogt ]
  * buildlib/apti18n.h.in:
    - fix build failure when building without NLS (closes: #671587)

  [ Gregoire Menuel ]
  * Fix double free (closes: #711045)

  [ Raphael Geissert ]
  * Fix crash when the "mirror" method does not find any entry
    (closes: #699303)

  [ Johan Kiviniemi ]
  * cmdline/apt-key:
    - Create new keyrings with mode 0644 instead of 0600.
    - Accept a nonexistent --keyring file with the adv subcommand as well.

apt (0.9.8.1) unstable; urgency=low

  [ David Kalnischkies ]
  * apt-pkg/indexcopy.cc:
    - non-inline RunGPGV methods to restore ABI compatibility with previous
      versions to fix partial upgrades (Closes: #707771)

  [ Michael Vogt ]
  * moved source to http://git.debian.org/...

Changed in apt (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.