[MIR] xdelta3

[Thomi Richards]

MIR for xdelta3

This is a request to include the xdelta3 package in Ubuntu main.

See below for point-for-point discussion of the items listed at:
https://wiki.ubuntu.com/UbuntuMainInclusionRequirements

[Availability]

Ubuntu Zesty contains xdelta 3.0.11-dfsg-1 in universe.

[Rationale]

xdelta3 is required for the 'download delta' feature in snapd. This allows
users to save a considerable amount of bandwidth when downloading updates for
installed snap packages. The code has all landed in snapd behind a feature flag,
but cannot be turned on by default until xdelta3 is in main, so snapd can depend
on xdelta3.

[Security]

There was one CVE files against xdelta3 that I could find:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-9765

The xdelta3 package installs a single binary (/usr/bin/xdelta3) which is not
suid or sgid.

[Quality assurance]

- The xdelta3 package requires no configuration after installation.

- As far as I can tell, the package asks no debconf questions of any priority.

- There are 90 open issues in the upstream bugtracker:
  https://github.com/jmacd/xdelta/issues

- I've scanned the issue list, and while a few issues may impact Ubuntu users
  using xdelta3, none of them seem serious enough to warrant exclusion from main
  in my opinion (but what do I know - that's for someone else to determine).

- The debian bug tracker contains security bug:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814067

  However this is fixed in the upstream release that's in zesty, and I can see a
  distropatch in the version that's in Xenial (I'm assuming it's been fixed in
  yakkety as well).

- The debian package is maintained by 'A Mennucc1', see:
  https://packages.qa.debian.org/x/xdelta3.html

- The xdelta3 packages does not require any exotic hardware.

- I'm honestly not sure if the upstream test suite is run during the package
  build. I see no explicit test runs in debian/rules, but there is a 'check'
  make target, so perhaps that's invoked by default?

- The package contains a debain/watch file.

[UI Standards]

The xdelta3 package ships command line utilities, so I think it's except from
the requirements of this section.

[Dependencies]

The two dependencies of xdelta3 (libc6 and liblzma5) are both already in main.

[Standards Compliance]

Since xdelta3 is already in debian, I can only assume that it conforms to the
related standards.

[Maintenance]

I think xdelta3 is relatively stable software, and the debian maintenance seems
adequate to me to minimise the amount of work we need to do to keep this package
in main.

[Background Information]

The xdelta3 package description contains a basic useful description of the
purpose of the package. The motivation behind this MIR is described in the
'rationale' section of this bug report.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthias Klose (doko)]

is "desktop packages" the correct bug subscriber for snap related packages?

[Matthias Klose (doko)]

 - please run the tests during the build

[William Grant (wgrant)]

I've fixed xdelta3 in zesty-proposed to run the test suite. It's currently stuck behind a few thousand autopkgtests, but should migrate soonish.

[Thomi Richards (thomir-deactivatedaccount)]

Hi,

After speaking with Gustavo, we agreed that the snappy-dev team can be the bug subscribers for this package.

[Matthias Klose (doko)]

then somebody (the team admin) has to subscribe the team; I can't do that myself.

[Thomi Richards (thomir-deactivatedaccount)]

Hi Matthias,

snappy-dev has been subscribed now.

Thanks,

[Matthias Klose (doko)]

Override component to main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty: universe/utils -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty amd64: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty arm64: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty armhf: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty i386: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty powerpc: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty ppc64el: universe/utils/optional/100% -> main
xdelta3 3.0.11-dfsg-1ubuntu1 in zesty s390x: universe/utils/optional/100% -> main
8 publications overridden.

[Thomi Richards (thomir-deactivatedaccount)]

Thanks Matthias!

On Tue, Jan 10, 2017 at 1:43 AM, Matthias Klose <email address hidden> wrote:

> Override component to main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty: universe/utils -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty amd64: universe/utils/optional/100%
> -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty arm64: universe/utils/optional/100%
> -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty armhf: universe/utils/optional/100%
> -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty i386: universe/utils/optional/100%
> -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty powerpc:
> universe/utils/optional/100% -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty ppc64el:
> universe/utils/optional/100% -> main
> xdelta3 3.0.11-dfsg-1ubuntu1 in zesty s390x: universe/utils/optional/100%
> -> main
> 8 publications overridden.
>
>
> ** Changed in: xdelta3 (Ubuntu)
> Status: New => Fix Released
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1647222
>
> Title:
> [MIR] xdelta3
>
> Status in xdelta3 package in Ubuntu:
> Fix Released
>
> Bug description:
> MIR for xdelta3
>
>
> This is a request to include the xdelta3 package in Ubuntu main.
>
> See below for point-for-point discussion of the items listed at:
> https://wiki.ubuntu.com/UbuntuMainInclusionRequirements
>
> [Availability]
>
> Ubuntu Zesty contains xdelta 3.0.11-dfsg-1 in universe.
>
> [Rationale]
>
> xdelta3 is required for the 'download delta' feature in snapd. This
> allows
> users to save a considerable amount of bandwidth when downloading
> updates for
> installed snap packages. The code has all landed in snapd behind a
> feature flag,
> but cannot be turned on by default until xdelta3 is in main, so snapd
> can depend
> on xdelta3.
>
> [Security]
>
> There was one CVE files against xdelta3 that I could find:
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-9765
>
> The xdelta3 package installs a single binary (/usr/bin/xdelta3) which is
> not
> suid or sgid.
>
> [Quality assurance]
>
> - The xdelta3 package requires no configuration after installation.
>
> - As far as I can tell, the package asks no debconf questions of any
> priority.
>
> - There are 90 open issues in the upstream bugtracker:
> https://github.com/jmacd/xdelta/issues
>
> - I've scanned the issue list, and while a few issues may impact Ubuntu
> users
> using xdelta3, none of them seem serious enough to warrant exclusion
> from main
> in my opinion (but what do I know - that's for someone else to
> determine).
>
> - The debian bug tracker contains security bug:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=814067
>
> However this is fixed in the upstream release that's in zesty, and I
> can see a
> distropatch in the version that's in Xenial (I'm assuming it's been
> fixed in
> yakkety as well).
>
> - The debian package is maintained by 'A Mennucc1', see:
> https://packages.qa.debian.org/x/xdelta3.html
>
> - The xdelta3 packages does not require any exotic hardware.
>
> - I'm honestly not sure if the upstream test suite is run during the
> packa...

[Seth Arnold (seth-arnold)]

I reviewed xdelta3 version 3.0.11-dfsg-1ubuntu1 as checked into zesty.
This should not be considered a full security audit but rather a quick
gauge of maintainability.

There is one CVE in our database; the patch mingled together the
functional change with twenty times more whitespace changes and a test
for the functional change. This isn't ideal but at least the patch was
labeled to fix a buffer overflow.

xdelta3 provides a binary-patch style of interface that is immensely
flexible. It can either be used as very manual style of compression
tool -- one of the inputs can be a dictionary of 'phrases' that would
be discovered automatically by other compression tools -- or it could be
used as a way to distribute a small set of changes against a fixed file.
It does so by building a program to interpret to emit the final results.

No dependencies.

xdelta3 is written in C, and heavily exploits pointer arithmetic, memcpy()
(but oddly enough not memmove() despite comments indicating that there may
be overlaps), and array indexing operations. When a native Rust version is
available please consider switching to it. (The existing VCDIFF crates on
crates.io are bindings for Google's VCDIFF implementation open-vcdiff.)

- xdelta3 provides a command line interface and header files that can be
  used to bake xdelta3 into other software (not via library use but by
  including the headers directly).
- No build deps
- No daemons
- No pre/post inst/rm scripts
- No init scripts
- No dbus services
- No setuid executables
- /usr/bin/xdelta3 binary
- No sudo fragments
- No udev rules
- The test suite is built into the executable and distributed in the
  binary. We should DISABLE these tests as they are very low quality.
- No cron jobs
- One common warning in the build logs "invalid suffix on literal"

- Subprocesses are spawned to support optional compression; execlp() is
  used directly, and while the execution itself looks safe, the
  compressors it executes can have their execution modified via
  environment variables. This is probably not a security issue but could
  be a cause of unreliable behaviour or support trouble. I strongly
  recommend executing xdelta3 with a whitelist of a handful of needed
  environment variables.

- Subprocesses are also spawned in the test suite. This code may be
  suitable for tests but is unacceptable in the executable or in the
  address space of anything that may choose to embed xdelta3. Ideally we'd
  build two executables in the package: one to run tests, and one to ship
  to users in binary packages, and the test binary would not be packaged
  in any binary packages. If this is too much work please just disable
  the tests at build time.

- Memory management has some potential integer multiplication overflows.

- Memory management is very complicated. Bugs have been found before and
  probably bugs still exist. We should fuzz xdelta3 extensively before we
  rely upon it. I'd love to see a re-write in Rust.

- Files are written to, under control of the command line, using simple
  fopen(3) calls.

- Logging is extensive, looked safe.

- The XDELTA environment variable handling is very complicated; I don't
  understand what t...