[SRU] version in repository is outdated and has vulnerabilities

[enkore]

[ Test description ]
* upstream has a really big testsuite, and coverage tools that helps covering all the code paths, e.g. by running borg save, crypt, decrypt, create, restore, with various files (binary, text and so on).
We run such testsuite on every architecture, and for stuff that requires
root access or different accesses there is a custom autopkgtestsuite that covers that border line cases.

============================= 55 tests deselected ==============================
 511 passed, 60 skipped, 55 deselected, 2 xpassed, 1 pytest-warnings in 169.40 seconds

and some of the skipped tests are run in autopkgtestsuite.

[Impact]

The current version in 16.10 universe is 1.0.7 which has two known vulnerabilities (CVE-2016-10099 and CVE-2016-10100) fixed in upstream version 1.0.9 (released ~6 months ago). The current upstream version is 1.0.10 (released ~3 months ago) and contains various other bugfixes.

[CHANGELOG]
Version 1.0.10 (2017-02-13)
---------------------------

Bug fixes:

- Manifest timestamps are now monotonically increasing,
  this fixes issues when the system clock jumps backwards
  or is set inconsistently across computers accessing the same repository, #2115
- Fixed testing regression in 1.0.10rc1 that lead to a hard dependency on
  py.test >= 3.0, #2112

New features:

- "key export" can now generate a printable HTML page with both a QR code and
  a human-readable "paperkey" representation (and custom text) through the
  ``--qr-html`` option.

  The same functionality is also available through `paperkey.html <paperkey.html>`_,
  which is the same HTML page generated by ``--qr-html``. It works with existing
  "key export" files and key files.

Other changes:

- docs:

  - language clarification - "borg create --one-file-system" option does not respect
    mount points, but considers different file systems instead, #2141
- setup.py: build_api: sort file list for determinism

Version 1.0.10rc1 (2017-01-29)
------------------------------

Bug fixes:

- borg serve: fix transmission data loss of pipe writes, #1268
  This affects only the cygwin platform (not Linux, BSD, OS X).
- Avoid triggering an ObjectiveFS bug in xattr retrieval, #1992
- When running out of buffer memory when reading xattrs, only skip the
  current file, #1993
- Fixed "borg upgrade --tam" crashing with unencrypted repositories. Since
  :ref:`the issue <tam_vuln>` is not relevant for unencrypted repositories,
  it now does nothing and prints an error, #1981.
- Fixed change-passphrase crashing with unencrypted repositories, #1978
- Fixed "borg check repo::archive" indicating success if "archive" does not exist, #1997
- borg check: print non-exit-code warning if --last or --prefix aren't fulfilled
- fix bad parsing of wrong repo location syntax
- create: don't create hard link refs to failed files,
  mount: handle invalid hard link refs, #2092
- detect mingw byte order, #2073
- creating a new segment: use "xb" mode, #2099
- mount: umount on SIGINT/^C when in foreground, #2082

Other changes:

- binary: use fixed AND freshly compiled pyinstaller bootloader, #2002
- xattr: ignore empty names returned by llistxattr(2) et al
- Enable the fault handler: install handlers for the SIGSEGV, SIGFPE, SIGABRT,
  SIGBUS and SIGILL signals to dump the Python traceback.
- Also print a traceback on SIGUSR2.
- borg change-passphrase: print key location (simplify making a backup of it)
- officially support Python 3.6 (setup.py: add Python 3.6 qualifier)
- tests:

  - vagrant / travis / tox: add Python 3.6 based testing
  - vagrant: fix openbsd repo, #2042
  - vagrant: fix the freebsd64 machine, #2037 #2067
  - vagrant: use python 3.5.3 to build binaries, #2078
  - vagrant: use osxfuse 3.5.4 for tests / to build binaries
    vagrant: improve darwin64 VM settings
  - travis: fix osxfuse install (fixes OS X testing on Travis CI)
  - travis: require succeeding OS X tests, #2028
  - travis: use latest pythons for OS X based testing
  - use pytest-xdist to parallelize testing
  - fix xattr test race condition, #2047
  - setup.cfg: fix pytest deprecation warning, #2050
- docs:

  - language clarification - VM backup FAQ
  - borg create: document how to backup stdin, #2013
  - borg upgrade: fix incorrect title levels
  - add CVE numbers for issues fixed in 1.0.9, #2106
- fix typos (taken from Debian package patch)
- remote: include data hexdump in "unexpected RPC data" error message
- remote: log SSH command line at debug level
- API_VERSION: use numberspaces, #2023
- remove .github from pypi package, #2051
- add pip and setuptools to requirements file, #2030
- SyncFile: fix use of fd object after close (cosmetic)
- Manifest.in: simplify, exclude \*.{so,dll,orig}, #2066
- ignore posix_fadvise errors in repository.py, #2095
  (works around issues with docker on ARM)
- make LoggedIO.close_segment reentrant, avoid reentrance

Version 1.0.9 (2016-12-20)
--------------------------

Security fixes:

- A flaw in the cryptographic authentication scheme in Borg allowed an attacker
  to spoof the manifest. See :ref:`tam_vuln` above for the steps you should
  take.

  CVE-2016-10099 was assigned to this vulnerability.
- borg check: When rebuilding the manifest (which should only be needed very rarely)
  duplicate archive names would be handled on a "first come first serve" basis, allowing
  an attacker to apparently replace archives.

  CVE-2016-10100 was assigned to this vulnerability.

Bug fixes:

- borg check:

  - rebuild manifest if it's corrupted
  - skip corrupted chunks during manifest rebuild
- fix TypeError in integrity error handler, #1903, #1894
- fix location parser for archives with @ char (regression introduced in 1.0.8), #1930
- fix wrong duration/timestamps if system clock jumped during a create
- fix progress display not updating if system clock jumps backwards
- fix checkpoint interval being incorrect if system clock jumps

Other changes:

- docs:

  - add python3-devel as a dependency for cygwin-based installation
  - clarify extract is relative to current directory
  - FAQ: fix link to changelog
  - markup fixes
- tests:

  - test_get\_(cache|keys)_dir: clean env state, #1897
  - get back pytest's pretty assertion failures, #1938
- setup.py build_usage:

  - fixed build_usage not processing all commands
  - fixed build_usage not generating includes for debug commands

Version 1.0.9rc1 (2016-11-27)
-----------------------------

Bug fixes:

- files cache: fix determination of newest mtime in backup set (which is
  used in cache cleanup and led to wrong "A" [added] status for unchanged
  files in next backup), #1860.

- borg check:

  - fix incorrectly reporting attic 0.13 and earlier archives as corrupt
  - handle repo w/o objects gracefully and also bail out early if repo is
    *completely* empty, #1815.
- fix tox/pybuild in 1.0-maint
- at xattr module import time, loggers are not initialized yet

New features:

- borg umount <mountpoint>
  exposed already existing umount code via the CLI api, so users can use it,
  which is more consistent than using borg to mount and fusermount -u (or
  umount) to un-mount, #1855.
- implement borg create --noatime --noctime, fixes #1853

Other changes:

- docs:

  - display README correctly on PyPI
  - improve cache / index docs, esp. files cache docs, fixes #1825
  - different pattern matching for --exclude, #1779
  - datetime formatting examples for {now} placeholder, #1822
  - clarify passphrase mode attic repo upgrade, #1854
  - clarify --umask usage, #1859
  - clarify how to choose PR target branch
  - clarify prune behavior for different archive contents, #1824
  - fix PDF issues, add logo, fix authors, headings, TOC
  - move security verification to support section
  - fix links in standalone README (:ref: tags)
  - add link to security contact in README
  - add FAQ about security
  - move fork differences to FAQ
  - add more details about resource usage
- tests: skip remote tests on cygwin, #1268
- travis:

  - allow OS X failures until the brew cask osxfuse issue is fixed
  - caskroom osxfuse-beta gone, it's osxfuse now (3.5.3)
- vagrant:

  - upgrade OSXfuse / FUSE for macOS to 3.5.3
  - remove llfuse from tox.ini at a central place
  - do not try to install llfuse on centos6
  - fix fuse test for darwin, #1546
  - add windows virtual machine with cygwin
  - Vagrantfile cleanup / code deduplication

Version 1.0.8 (2016-10-29)
--------------------------

Bug fixes:

- RemoteRepository: Fix busy wait in call_many, #940

New features:

- implement borgmajor/borgminor/borgpatch placeholders, #1694
  {borgversion} was already there (full version string). With the new
  placeholders you can now also get e.g. 1 or 1.0 or 1.0.8.

Other changes:

- avoid previous_location mismatch, #1741

  due to the changed canonicalization for relative pathes in PR #1711 / #1655
  (implement /./ relpath hack), there would be a changed repo location warning
  and the user would be asked if this is ok. this would break automation and
  require manual intervention, which is unwanted.

  thus, we automatically fix the previous_location config entry, if it only
  changed in the expected way, but still means the same location.

- docs:

  - deployment.rst: do not use bare variables in ansible snippet
  - add clarification about append-only mode, #1689
  - setup.py: add comment about requiring llfuse, #1726
  - update usage.rst / api.rst
  - repo url / archive location docs + typo fix
  - quickstart: add a comment about other (remote) filesystems

- vagrant / tests:

  - no chown when rsyncing (fixes boxes w/o vagrant group)
  - fix fuse permission issues on linux/freebsd, #1544
  - skip fuse test for borg binary + fakeroot
  - ignore security.selinux xattrs, fixes tests on centos, #1735

Version 1.0.8rc1 (2016-10-17)
-----------------------------

Bug fixes:

- fix signal handling (SIGINT, SIGTERM, SIGHUP), #1620 #1593
  Fixes e.g. leftover lock files for quickly repeated signals (e.g. Ctrl-C
  Ctrl-C) or lost connections or systemd sending SIGHUP.
- progress display: adapt formatting to narrow screens, do not crash, #1628
- borg create --read-special - fix crash on broken symlink, #1584.
  also correctly processes broken symlinks. before this regressed to a crash
  (5b45385) a broken symlink would've been skipped.
- process_symlink: fix missing backup_io()
  Fixes a chmod/chown/chgrp/unlink/rename/... crash race between getting
  dirents and dispatching to process_symlink.
- yes(): abort on wrong answers, saying so, #1622
- fixed exception borg serve raised when connection was closed before reposiory
  was openend. add an error message for this.
- fix read-from-closed-FD issue, #1551
  (this seems not to get triggered in 1.0.x, but was discovered in master)
- hashindex: fix iterators (always raise StopIteration when exhausted)
  (this seems not to get triggered in 1.0.x, but was discovered in master)
- enable relative pathes in ssh:// repo URLs, via /./relpath hack, #1655
- allow repo pathes with colons, #1705
- update changed repo location immediately after acceptance, #1524
- fix debug get-obj / delete-obj crash if object not found and remote repo,
  #1684
- pyinstaller: use a spec file to build borg.exe binary, exclude osxfuse dylib
  on Mac OS X (avoids mismatch lib <-> driver), #1619

New features:

- add "borg key export" / "borg key import" commands, #1555, so users are able
  to backup / restore their encryption keys more easily.

  Supported formats are the keyfile format used by borg internally and a
  special "paper" format with by line checksums for printed backups. For the
  paper format, the import is an interactive process which checks each line as
  soon as it is input.
- add "borg debug-refcount-obj" to determine a repo objects' referrer counts,
  #1352

Other changes:

- add "borg debug ..." subcommands
  (borg debug-* still works, but will be removed in borg 1.1)
- setup.py: Add subcommand support to build_usage.
- remote: change exception message for unexpected RPC data format to indicate
  dataflow direction.
- improved messages / error reporting:

  - IntegrityError: add placeholder for message, so that the message we give
    appears not only in the traceback, but also in the (short) error message,
    #1572
  - borg.key: include chunk id in exception msgs, #1571
  - better messages for cache newer than repo, #1700
- vagrant (testing/build VMs):

  - upgrade OSXfuse / FUSE for macOS to 3.5.2
  - update Debian Wheezy boxes, #1686
  - openbsd / netbsd: use own boxes, fixes misc rsync installation and
    fuse/llfuse related testing issues, #1695 #1696 #1670 #1671 #1728
- docs:

  - add docs for "key export" and "key import" commands, #1641
  - fix inconsistency in FAQ (pv-wrapper).
  - fix second block in "Easy to use" section not showing on GitHub, #1576
  - add bestpractices badge
  - link reference docs and faq about BORG_FILES_CACHE_TTL, #1561
  - improve borg info --help, explain size infos, #1532
  - add release signing key / security contact to README, #1560
  - add contribution guidelines for developers
  - development.rst: add sphinx_rtd_theme to the sphinx install command
  - adjust border color in borg.css
  - add debug-info usage help file
  - internals.rst: fix typos
  - setup.py: fix build_usage to always process all commands
  - added docs explaining multiple --restrict-to-path flags, #1602
  - add more specific warning about write-access debug commands, #1587
  - clarify FAQ regarding backup of virtual machines, #1672
- tests:

  - work around fuse xattr test issue with recent fakeroot
  - simplify repo/hashindex tests
  - travis: test fuse-enabled borg, use trusty to have a recent FUSE
  - re-enable fuse tests for RemoteArchiver (no deadlocks any more)
  - clean env for pytest based tests, #1714
  - fuse_mount contextmanager: accept any options

[Regression Potential]
* borgbackup has a really huge testsuite, and we run it during build/autopkgtest

[Gianfranco Costamagna (costamagnagianfranco)]

Subscribing security team because of the CVEs

[Gianfranco Costamagna (costamagnagianfranco)]

This is affecting only xenial and yakkety

[Brian Murray (brian-murray)]

There is no information which meets the "Test Case" criteria of the Ubuntu SRU process. Given the large laundry list of changes, how do plan on testing that this works in Yakkety and Xenial and that there are no regressions?

[enkore (enkore)]

Borg has, broadly speaking, relatively few regressions. In the months since 1.0.9/10 I believe only one regression was reported (related to UNC paths on the not officially supported Cygwin platform, w/ workaround) and one or two minor bugs in some of the new functionality.

That does, of course, technically not imply that it works on Ubuntu.

[Gianfranco Costamagna (costamagnagianfranco)]

@Brian Murray, do you want me to check if the CVEs are fixed? Otherwise the testsuite creates directory files, backups them on various different filesystems, and recovers them after checking checksums of files and a lot of more stuff.

I can do manual testing, of course, but it won't really be better than the really nice automatic testsuite :) (plus autopkgtests for the stuff that requires root/llfuse)

So, if you want me to test, just ask.
BTW a lot of people are using my ppa (because officially mentioned on the website IIRC), and I mostly never got a complain, at least for the new releases I'm trying to SRU.

1.0.10 is the safe bet WRT vulnerabilities and stability (I would oppose right now an sru to 1.1 e.g.)

[Brian Murray (brian-murray)]

Do you think this qualifies as a "New upstream microrelease", https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases, if so then I'd expect the bug description to contain information about the upstream development process and how they test releases per the wiki page.

As it currently stands having a bunch of changelog entries saying "New upstream release" doesn't provide much justification for why it should be SRU'ed.

[Gianfranco Costamagna (costamagnagianfranco)]

(reuploaded the same file)

[Gianfranco Costamagna (costamagnagianfranco)]

Hello, ping?

[Brian Murray (brian-murray)]

Could you respond to my previous comment?

https://bugs.launchpad.net/ubuntu/+source/borgbackup/+bug/1690846/comments/6

[Chris Halse Rogers (raof)]

It's good that this has a test-suite, and autopkgtests running it. Does it test the compatibility against archives created by previous versions?

The SRU microrelease guidelines say to point at the upstream documentation of their test proceedures; is there one?