live-build causes installation old /sbin/initctl and start-stop-daemon to be installed in Cloud Images

[Kevin Blackham]

[ SRU Info ]
See bug #1363519

[ Original Report ]
I've encountered at least five different Amazon AMIs which all fail debsums right out of the box. The offending binaries are upstart: /sbin/initctl and dpkg: /sbin/start-stop-daemon. Both are handy locations to drop a rootkit. Most prominently, the banner-choice for the GUI AWS console wizard in us-east-1 is ami-3fec7956, which seems to be created by Canonical (ami-3fec7956 099720109477/ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20130124).

us-east-1: ami-3fec7956, ami-de0d9eb7
us-west-1: ami-b81230fd
us-west-2: ami-da1810ae
ap-northeast-1: ami-77cf4976

I analyzed only initctl, as it was the first to trip my alarms. In the following examples, the left side is the one from the upstart package version dpkg claims to have installed, the right is the one actually on the system.

* A side-by-side hex/ascii of the diff : http://pastebin.com/raw.php?i=JGN1rMC5
* The same, with some color: http://cl.ly/image/2x2l3S1j1f38
* A side-by-side comparison of objdump --disassemble-all: http://pastebin.com/raw.php?i=5ppcJG2H
* The same, with some color: http://cl.ly/image/2d0w1I3W083x

The file size is the same, but clearly it is not the one from the package. objdump interpreted this delta as having a mov instruction, then executing a conditional jump. This "looks" scary, but I'm no expert. This may be only a side-effect of trying to disassemble code.

This is either malicious, or a totally benign mistake, but in either case, Canonical should not be distributing AMIs which cannot pass rudimentary integrity checks.

To reproduce, simply start an AWS instance with one of these AMIs, install debsums, and run something like:

for PKG in `dpkg --get-selections | awk '{print $1}'`; do echo ${PKG}: >> failsums; debsums $PKG | grep FAILED >> failsums; done

[Kevin Blackham (thekev.)]

/sbin/initctl (md5 a08543b3a5d7f2221358f9f160c3b09f) which is not the same as in upstart 1.5-0ubuntu7.2 (md5 bae534f4f29d22f3fda948e8a8157745): http://user.xmission.com/~kevin/initctl

/sbin/start-stop-daemon (md5 668f331a1ee2a34b049bcca5c5516322) not matching dpkg 1.16.1.2ubuntu7.1 (md5 733bf57a6e070bb6541d7e688b3c85d1): http://user.xmission.com/~kevin/start-stop-daemon

[Ante Karamatić (ivoks)]

ATM, investigation shows that failed md5sums are actually the same as those from 'original' packages. For instance, package upstart, version 1.5-0ubuntu5, is shipped with Ubuntu 12.04. /sbin/initctl from that package has a MD5 a08543b3a5d7f2221358f9f160c3b09f. MD5 for /sbin/initctl from current upstart package is bae534f4f29d22f3fda948e8a8157745. Installs on hardware machines do have correct MD5, but cloud images have an older MD5.

I suspect this is caused by live-build build process, where updates are done after the initial install and where build process mangles with /sbin/initctl and /sbin/start-stop-daemon so that those tools don't interfere with system on which image is built. In other words, this is a bug, this is a problem, it has potential of being serious bug, but it's not security issue. I'll leave it to the security team to make final decision.

[Ben Howard (darkmuggle-deactivatedaccount)]

This is a live-build artifact. During installation, livebuild does this little gem to keep things from starting:

        if [ -f chroot/sbin/initctl ]
        then
            # Save initctl file
            mv chroot/sbin/initctl chroot/sbin/initctl.orig
        fi

And then restores it via:
        if [ -f chroot/sbin/initctl.orig ]
        then
            # Restore initctl file
            mv chroot/sbin/initctl.orig chroot/sbin/initctl
        else
            # Remove initctl file
            rm -f chroot/sbin/initctl
        fi

[Ben Howard (darkmuggle-deactivatedaccount)]

Since this is a bug with the build system, I am taking ownership and will resolve this ASAP.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ben Howard (darkmuggle-deactivatedaccount)]

Confirmed the same logic for start-stop-daemon.

[Ben Howard (darkmuggle-deactivatedaccount)]

Fixed by switching to "dpkg-divert" replacing the files. Test build complete and passed debsum check.

Currently building and publisihing a 12.04 daily, which will be confirmed tomorrow.

[Ben Howard (darkmuggle-deactivatedaccount)]

Submitted fix to Debian live-build.

[Rafał Roncoszek (r-roncoszek)]

Confirm the problem in Ubuntu 12.04 64bit. I installed the system from pendrive prepared using UNetbootin and ubuntu-12.04.2-desktop-amd64.iso. Sudo debsums -c results in what follows:
/sbin/start-stop-daemon
/sbin/initctl
My md5sums of the files are:
a08543b3a5d7f2221358f9f160c3b09f initctl
668f331a1ee2a34b049bcca5c5516322 start-stop-daemon
Rgds!

[Timothy R. Chavez (timrchavez)]

Just wondering why this hasn't made it into Ubuntu's version of live-build yet?

[David Medberry (med)]

It doesn't look like the Ubuntu version has synced with Debian in some time. I can't tell by examining https://launchpad.net/ubuntu/+source/live-build why that is the case. I think Ben marked it as fix-committed for the build system (but not for the actual ubuntu package.)

[Ben Howard (darkmuggle-deactivatedaccount)]

med: Indeed I did.

I'll run the SRU for this. Scheduling the work to be done shortly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu19

---------------
live-build (3.0~a57-1ubuntu19) wily; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)
  * debian/patches/upstart-user-sessions: Dropped, obsoleted by the above.

 -- Adam Conrad <email address hidden> Tue, 21 Jul 2015 13:51:28 -0600

[Brian Murray (brian-murray)]

Hello Kevin, or anyone else affected,

Accepted live-build into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kevin, or anyone else affected,

Accepted live-build into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu16.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kevin, or anyone else affected,

Accepted live-build into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu11.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kevin, or anyone else affected,

Accepted live-build into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/live-build/3.0~a57-1ubuntu16.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Adam Conrad (adconrad)]

SRU verified per the test case in bug #1363519 on trusty, vivid, and wily.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu11.2

---------------
live-build (3.0~a57-1ubuntu11.2) trusty; urgency=medium

  * debian/patches/upstart-not-in-bootstrap: Rewrite the old user-sessions
    patch to handle when upstart isn't in the bootstrap set (LP: #1477051)

live-build (3.0~a57-1ubuntu11.1) trusty; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)

 -- Adam Conrad <email address hidden> Wed, 22 Jul 2015 09:40:28 -0600

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for live-build has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu16.3

---------------
live-build (3.0~a57-1ubuntu16.3) vivid; urgency=medium

  * debian/patches/upstart-not-in-bootstrap: Rewrite the old user-sessions
    patch to handle when upstart isn't in the bootstrap set (LP: #1477051)

live-build (3.0~a57-1ubuntu16.2) vivid; urgency=medium

  * debian/patches/dpkg-divert-{initctl,others}: Pull two commits from git
    upstream to resolve issues where we would end up with files from their
    release version packages rather than -updates (LP: #1363519, #1150737)
  * debian/patches/upstart-user-sessions: Dropped, obsoleted by the above.

 -- Adam Conrad <email address hidden> Wed, 22 Jul 2015 09:02:59 -0600