start-stop-daemon fails debsums check

[Rafał Roncoszek]

[ SRU Justification ]
Currently, systems installed from live media have several critical files (like initctl and start-stop-daemon) failing debsums checks because they are incorrectly using the files from the release pocket versions of their packages, rather than the -updates version that's actually installed.

[ Test Case ]
Build a test CD, run through an install, and verify that, post-install, all the files referenced in the patches are the expected content. Those files are:

For trusty:
/sbin/initctl (should be from the -updates version of upstart)
/sbin/start-stop-daemon (should be from the -updates version of dpkg)
/bin/hostname (should be from the release pocket version of hostname)
/usr/sbin/policy-rc.d (should not exist on the filesystem, post-install)

For vivid:
/sbin/initctl (should be from the -updates version of upstart)
None of the other packages have had an SRU in vivid.

[ Regression Potential ]
Assuming the above test case passes, pretty minimal. The change isn't invasive, except for swapping silly mv && do_stuff && mv semantics with more correct dpkg-divert add/remove.

[ Original Report ]
After fresh install from 14.04.1-amd64-dvd - debsums -c results in: /sbin/start-stop-daemon. I checked dpkg package with dpkg -s dpkg and got: Status: install ok installed, Version: 1.17.5ubuntu5.3 (the latest version) but when I did md5sum /sbin/start-stop-daemon, I got b1b8894ae2e3b547dca0e288634cce4a, which is md5sum for start-stop-daemon from dpkg version 1.17.5ubuntu5!
I confirmed the problem in Ubuntu/Xubuntu 14.04.1-i386 (VirtualBox) - the same case - latest version of dpkg package contains older version of start-stop-daemon. I guess this is the same problem I reported here: https://bugs.launchpad.net/ubuntu/+source/live-build/+bug/1150737. I fixed the problem with sudo apt-get install --reinstall dpkg - simple if You know about it, otherwise You have outdated/buggy/ package/system...
Rgds!

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: dpkg 1.17.5ubuntu5.3
ProcVersionSignature: Ubuntu 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
CurrentDesktop: Unity
Date: Sat Aug 30 23:47:41 2014
InstallationDate: Installed on 2014-08-07 (23 days ago)
InstallationMedia: Ubuntu 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.2)
SourcePackage: dpkg
UpgradeStatus: No upgrade log present (probably fresh install)

[Rafał Roncoszek (r-roncoszek)]

The above mentioned system also reports problem (debsums -c) with /boot/vmlinuz-3.13.0-32-generic.
My md5sum is: 144bf4beed11fb77e5ad629452741310 vmlinuz-3.13.0-32-generic. I couldn't find that hash anywhere online.
Hash of vmlinuz-3.13.0-32-generic downloaded from http://packages.ubuntu.com/trusty/amd64/linux-image-3.13.0-32-generic/download is: 70f26e2f62b9dc8ee05309fe96170ee7 vmlinuz-3.13.0-32-generic.
There was no problem with debsums in the system when I installed it from 14.04-amd64.iso right after Trusty was released.

[Seth Arnold (seth-arnold)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

[Seth Arnold (seth-arnold)]

BTW, please check dmesg output for error messages that may indicate failing hardware -- most similar reports are filed when user's hard drives are near death.

Thanks

[Rafał Roncoszek (r-roncoszek)]

Thanks for Your reply! Of course I'll try dmesg, but:
1) Maybe I didn't describe it precisely enough - the systems (Ubuntu/Xubuntu i386) I checked in VirtualBox were installed on another machine and gave the same results as on my laptop;
2) it's hard to imagine that a corrupted hard disk changes md5sum of a file to md5sum of the file's previous version;
3) I reported similar problem a year ago with 12.04 https://bugs.launchpad.net/ubuntu/+source/live-build/+bug/1150737
4) there was no problem with debsums after fresh install of 14.04.0...
that's why I thought it's rather packaging problem during building the release iso.
Why I marked is as a security issue? If my theory is OK - we have files in our systems that are not updated. Not updated means for me potentially buggy. If this is random situation, out of anyone's control, the problem seems to be serious.
Rgds!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Grocock (mgrocock)]

I'm seeing the exact same thing. On a fresh Ubuntu Server 14.04.1 install:

root@server1:~# md5sum /sbin/start-stop-daemon
b1b8894ae2e3b547dca0e288634cce4a /sbin/start-stop-daemon
root@server1:~# apt-get install --reinstall dpkg
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 to upgrade, 0 to newly install, 1 reinstalled, 0 to remove and 0 not to upgrade.
Need to get 1,954 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu/ trusty-updates/main dpkg amd64 1.17.5ubuntu5.3 [1,954 kB]
Fetched 1,954 kB in 0s (6,556 kB/s)
(Reading database ... 71661 files and directories currently installed.)
Preparing to unpack .../dpkg_1.17.5ubuntu5.3_amd64.deb ...
Unpacking dpkg (1.17.5ubuntu5.3) over (1.17.5ubuntu5.3) ...
Processing triggers for man-db (2.6.7.1-1) ...
Setting up dpkg (1.17.5ubuntu5.3) ...
root@server1:~# md5sum /sbin/start-stop-daemon
6a1ece8cbbe7d816dcdd83e79e837605 /sbin/start-stop-daemon

[Seth Arnold (seth-arnold)]

Just a quick note that I've seen several bug reports filed since this one that included some Dependencies.txt files with lines like "modified: sbin/start-stop-daemon" -- I think this is more widespread than this bug indicates. See 1374670 for an example.

[Brian Murray (brian-murray)]

Could you please include the log file from your install of Ubuntu? It can be found in /var/log/installer/syslog. Thanks!

[Rafał Roncoszek (r-roncoszek)]

Tomorrow at work I'll try to get the syslogs of the VBox installations of Ubuntu/Xubuntu i386.
Rgds!

[Samuel Leslie (sdl)]

Just chipping in that I have just encountered this exact bug as reported by Rafał Roncoszek on an Ubuntu Server 14.04.1 x64 server I admin. Was initially very concerned to see a critical system binary failing a debsums check but the md5sum's of the failing binary and the updated binary via a 'apt-get install dpkg --reinstall' exactly match those provided in this bug report.

While I agree this isn't a security issue, I do consider it a fairly severe bug when incorrect/unexpected versions of system binaries are being installed and in turn causing simple auditing tools to report issues with the installation. Thanks!

[Thomas Mayer (thomas303)]

I am very concerned about this issue. I installed from media 119cb63b48c9a18f31f417f09655efbd ubuntu-14.04.1-desktop-amd64.iso. I double-checked the hash comes from a SSL-trusted page and checked the md5 sum of the file which was correct.

However, I also get
 md5sum /boot/vmlinuz-3.13.0-32-generic
144bf4beed11fb77e5ad629452741310 /boot/vmlinuz-3.13.0-32-generic
md5sum /sbin/start-stop-daemon
b1b8894ae2e3b547dca0e288634cce4a /sbin/start-stop-daemon

Could you please re-check that this is trusted software?

Next, I'm thinking about the question, how the community can technically make sure that we never install untrusted software via apt-get. What if a man-in-the-middle-Attack happens? This can easyly happen, if your router is fishy!

Besides that, I think about third-party-repositories, and there's two scenarios I could imagine.
- What if I get untrusted keys via a man-in-the-middle-attack
- What about software which is downloaded by (third-party-)repo-software? First, this is out of focus for debsums. Second, I'm never sure if the downloaded software is rechecked against trustworth (SSL-transmitted) hashes. Many of us have to use oracle-java for some reason, which is provided by webupd8 repos, for example. Not that I say, I don't trust them. But if EVER they provide/download untrusted software(e.g. occasionally via man-in-the-middle), a lot of servers and desktops would be affected via regular update. For example, oracle-java download (done by the apt-package of webupd8) needs to be checked against a trusted hash. I hope, that is already the case, I did not recheck this.
- Think about wine software which is downloaded before you can use it
- Think about extensions in Mozilla's Firefox/Thunderbird or in OpenOffice/LibreOffice

In general:
- There's a bunch of software you need to download from sources which are not covered by apt-get (but possibly updated via apt-get). You don't get around that for some reason (e.g. oracle-java).
- If a man-in-the-middle-attack happens, the attacker could install any piece of software he wants via a regular update via third-party-repos and other software like wine.

The longer I think about it, the more I am concerned about ubuntu's security concerning software rollout. Still, I think that ubuntu's universe repo is the best way to provide lots of software which can be trusted (if the maintainer can be trusted). I would like to have the same affirmation for all software I ever need to install.

[Thomas Mayer (thomas303)]

To track it down, I took the 14.04.1 media 119cb63b48c9a18f31f417f09655efbd again and installed it without any updates.

Doing that, I got the two
144bf4beed11fb77e5ad629452741310 /boot/vmlinuz-3.13.0-32-generic
b1b8894ae2e3b547dca0e288634cce4a /sbin/start-stop-daemon

again. So, the problem should be reprocudable by anyone. I would appreciate if someone could double check this.

This kind of problem should be avoided in future by making sure that debsums can safely check integrity of installed packages again. We totally rely on such instruments when it comes to security. Especially when core components are affected.

Best regards
Thomas

[Mitar (mitar)]

Is there any workaround?

[Tuomas Heino (iheino+ub)]

Spotted this again on a fresh 14.04.1 LTS server (amd64) installation.

Applied workaround from https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1363519/comments/11 and added it as another post-inst thing to do on all my installations, along with a verification ("dpkg -V"/debsums/tripwire/etc).

[Thomas Mayer (thomas303)]

As of today, debsums -c reports me

debsums -c
/usr/lib/x86_64-linux-gnu/gbm/old.gbm_gallium_drm.so
/usr/lib/x86_64-linux-gnu/old.libgbm.so.1.0.0

Please, please, clean up all the wrong entries. There should be a test which any package must pass before it can be rolled out.

The test should also include updates and corresponding scripts/triggers. It always drives me crazy when debsums -c is reporting something. Especially, I'm not sure if I can rely on virustotal to check detected files against malware (the virus scanners are somehow windows related).

In germany, we have a "federal trojan" created by our government which claims to also infect linux boxes. Not that I'm afraid of being a target. I'm rather afraid someone even worse than our government could reuse that malware (with some improvements, as our government has no idea about computing...).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu19

---------------
live-build (3.0~a57-1ubuntu19) wily; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)
  * debian/patches/upstart-user-sessions: Dropped, obsoleted by the above.

 -- Adam Conrad <email address hidden> Tue, 21 Jul 2015 13:51:28 -0600

[Brian Murray (brian-murray)]

Hello Rafał, or anyone else affected,

Accepted live-build into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Rafał, or anyone else affected,

Accepted live-build into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu16.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Rafał, or anyone else affected,

Accepted live-build into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu11.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Rafał, or anyone else affected,

Accepted live-build into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu16.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

SRU verified per the test case on trusty, vivid, and wily.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu11.2

---------------
live-build (3.0~a57-1ubuntu11.2) trusty; urgency=medium

  * debian/patches/upstart-not-in-bootstrap: Rewrite the old user-sessions
    patch to handle when upstart isn't in the bootstrap set (LP: #1477051)

live-build (3.0~a57-1ubuntu11.1) trusty; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)

 -- Adam Conrad <email address hidden> Wed, 22 Jul 2015 09:40:28 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for live-build has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu16.3

---------------
live-build (3.0~a57-1ubuntu16.3) vivid; urgency=medium

  * debian/patches/upstart-not-in-bootstrap: Rewrite the old user-sessions
    patch to handle when upstart isn't in the bootstrap set (LP: #1477051)

live-build (3.0~a57-1ubuntu16.2) vivid; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)
  * debian/patches/upstart-user-sessions: Dropped, obsoleted by the above.

 -- Adam Conrad <email address hidden> Wed, 22 Jul 2015 09:02:59 -0600