source.list for armhf includes trusty-security which does not exist for arm

Bug #1711735 reported by ruffsl on 2017-08-18
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
cloud-images
Undecided
Unassigned
livecd-rootfs (Ubuntu)
High
Unassigned
Trusty
High
Michael Hudson-Doyle

Bug Description

[Impact]
The armhf cloud image fails during apt-update as the source.list includes a reference to trusty-security, a source that does not appear to host support for that architecture:
http://security.ubuntu.com/ubuntu/dists/trusty-security/

This issue is blocking the use of the armhf version of ubuntu on docker hub's official library:
https://github.com/tianon/docker-brew-ubuntu-core/issues/94#issuecomment-323401913

The latest image possesses this issue,
but might want to check other image architecture and suites as well:

Image: ubuntu-trusty-core-cloudimg-armhf-root.tar.gz
SHA256SUM: c264a88662bbb1d5cdf0deb888720b4b508ad3e5054707d017a1f84d31827bc7
Link: https://partner-images.canonical.com/core/trusty/20170817/

[Test Case]
Unpack the armhf chroot and attempt to run apt-get update in it. If it succeeds, this bug is fixed.

[Regression Potential]
Part of the verification should be to diff the trees produced by versions of livecd-rootfs in -updates and -proposed. The only non-trivial difference should be in /etc/apt/sources.list (there will be some differences timestamps in log files and obviously different indexes will be present in /var/lib/apt/lists) and if that is the only change then there is no regression potential.

Related branches

ruffsl (roxfoxpox) wrote :
ruffsl (roxfoxpox) on 2017-08-18
description: updated
ruffsl (roxfoxpox) on 2017-08-18
tags: added: docker partner-images
Dan Watkins (daniel-thewatkins) wrote :

I can confirm that this is happening, and it's in the output that comes from the buildds, so it's a livecd-rootfs/live-build bug.

(It's probably the former, so adding that package to this bug.)

Dan Watkins (daniel-thewatkins) wrote :

This doesn't exhibit in xenial.

Changed in cloud-images:
status: New → Confirmed
Dan Watkins (daniel-thewatkins) wrote :

This was fixed for xenial in bug 1513529 and bug 1679252. (Specifically, the former refactored sources.list generation significantly and in a way which broke non-x86 architectures, and the latter remedied that.)

Unfortunately, the changes to sources.list generation were fairly major, so I don't think it would be appropriate to pull them all the way back to trusty; we should find a less invasive fix.

Dan Watkins (daniel-thewatkins) wrote :

The fix in bug 1679252 uses LB_PARENT_MIRROR_BINARY_SECURITY for security lines (where currently trusty uses LB_PARENT_MIRROR_BINARY; I don't _think_ this should work because LB_PARENT_MIRROR_BINARY is used for the other sources.list lines which are fine, but it's worth a try.

Commit with that change pushed up to https://code.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/+git/livecd-rootfs/+ref/bug/1711735, and I've kicked off a build of livecd-rootfs to attempt a build from.

ruffsl (roxfoxpox) wrote :

Ya! thanks for following up @daniel-thewatkins , I was just about to message you as I found you helped to resolve another previous docker image related bug 1645463 with @tianon.

> we should find a less invasive fix.

What would you have in mind. Is there another way we could resolve this upstream, or should we turn to the user to make a patch to the source.list before use, e.g. modifying the downstream debootstrap additionally for docker to make the fix per trusty?
https://github.com/moby/moby/blob/9a9fc01af8fb5d98b8eec0740716226fadb3735c/contrib/mkimage/debootstrap#L67

Dan Watkins (daniel-thewatkins) wrote :

I still think we should fix this in Ubuntu, just not with the full fix from the aforementioned bugs. :)

ruffsl (roxfoxpox) wrote :

Ok Dan, let me know if you'd like me to test anything once the build comes back.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in livecd-rootfs (Ubuntu Trusty):
status: New → Confirmed
Changed in livecd-rootfs (Ubuntu):
status: New → Confirmed
ruffsl (roxfoxpox) wrote :

Hello Dan, was your latest commit successful in building a cloud image with a patched source list?
https://git.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/commit/?id=1bc0b2eec50726f46510df4a340d2f27f62756bb

Dimitri John Ledkov (xnox) wrote :

We need to SRU this into Trusty right?

Changed in livecd-rootfs (Ubuntu Trusty):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in livecd-rootfs (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in livecd-rootfs (Ubuntu Trusty):
importance: Undecided → High
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → High

On Wed, Aug 23, 2017 at 07:10:16PM -0000, ruffsl wrote:
> Hello Dan, was your latest commit successful in building a cloud image with a patched source list?
> https://git.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/commit/?id=1bc0b2eec50726f46510df4a340d2f27f62756bb

The package itself failed to build, which I've addressed now; will
attempt an image build once it publishes out to my PPA.

Unfortunately my change didn't fix the issue; I'll revisit this soon.

ruffsl (roxfoxpox) wrote :

Hello Dan, just following up. Any new ideas? I suppose I can modify our official docker hub manifest to omit trusty for armhf to at least get xenial images out of our farm.

Dimitri John Ledkov (xnox) wrote :

On 10 September 2017 at 08:55, ruffsl <email address hidden> wrote:
> Hello Dan, just following up. Any new ideas? I suppose I can modify our
> official docker hub manifest to omit trusty for armhf to at least get
> xenial images out of our farm.

Do not omit trusty for armhf, there are production systems that depend on it.

Apart from not having correct security stanzas are there any other
side-effects? Note it is mostly harmless as -updates is enabled by
default, and all security updates are published in both -security and
-updates. Thus "wrong" security stanza, is mostly harmless.

--
Regards,

Dimitri.

Tianon Gravi (tianon) wrote :

He means simply removing "trusty + armhf" for the "ros" and/or
"gazebo" images, I believe (not for the "ubuntu" image).

The harm this causes is that "apt-get update" fails, which causes the
whole "docker build" to fail.

Here's an example from ppc64el:

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/restricted/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/universe/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/multiverse/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

E: Some index files failed to download. They have been ignored, or old
ones used instead.

ruffsl (roxfoxpox) wrote :

Just checking in to ask if there is anything I could assist for this issue.
Are there any docs on how build a cloud image using the source in livecd-rootfs?

Tianon Gravi (tianon) wrote :

This is still causing quite a _lot_ of downstream pain -- any chance it could be re-prioritized?

Andrew Hsu (andrewhsu) wrote :

It looks to me the ports.ubuntu.com endpoint is receiving updates for security fixes for non-amd64 architectures: http://ports.ubuntu.com/ubuntu-ports/dists/trusty-security/multiverse/

As a workaround until this problem is fixed, I've changed the apt repo definition to ports.ubuntu.com:

$ sed -i 's|security.ubuntu.com/ubuntu|ports.ubuntu.com/ubuntu-ports|' /etc/apt/sources.list

And then `apt-get update` work for me on trusty:

$apt-get update
...
Get:1 http://ports.ubuntu.com trusty-security/universe Sources [76.3 kB]
Get:2 http://ports.ubuntu.com trusty-security/main armhf Packages [688 kB]
Get:3 http://ports.ubuntu.com trusty-security/restricted armhf Packages [10.3 kB]
Get:4 http://ports.ubuntu.com trusty-security/universe armhf Packages [239 kB]
Hit http://ports.ubuntu.com trusty-security/multiverse armhf Packages
Hit http://ports.ubuntu.com trusty/universe Sources
Hit http://ports.ubuntu.com trusty/main armhf Packages
Hit http://ports.ubuntu.com trusty/restricted armhf Packages
Hit http://ports.ubuntu.com trusty/universe armhf Packages
Hit http://ports.ubuntu.com trusty/multiverse armhf Packages
Fetched 1013 kB in 17s (56.5 kB/s)
Reading package lists... Done

This affects the trusty image for arm64 as well, noted at https://github.com/docker-library/official-images/issues/3583 (which referred me back here).

tags: added: id-59e4f7322915e8534b646c46

------- Comment From <email address hidden> 2017-10-18 14:00 EDT-------
.

tags: added: architecture-ppc64le bugnameltc-159986 severity-high targetmilestone-inin---
Dimitri John Ledkov (xnox) wrote :

@bugproxy

Reverse-proxying this bug report bears no meaning. Please do not track this launchpad issue. If you are experiencing this or a similar issue on POWER9 please open a new LTC bug report, and proxy it to launchpad afresh, explaining the issue you have.

Changed in livecd-rootfs (Ubuntu):
assignee: Dimitri John Ledkov (xnox) → Michael Hudson-Doyle (mwhudson)
assignee: Michael Hudson-Doyle (mwhudson) → nobody
status: Confirmed → Fix Released
Changed in livecd-rootfs (Ubuntu Trusty):
status: Confirmed → In Progress
assignee: Dimitri John Ledkov (xnox) → Michael Hudson-Doyle (mwhudson)
Changed in livecd-rootfs (Ubuntu Trusty):
status: In Progress → Triaged
description: updated
Changed in livecd-rootfs (Ubuntu Trusty):
status: Triaged → In Progress

Hello ruffsl, or anyone else affected,

Accepted livecd-rootfs into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.208.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in livecd-rootfs (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-trusty

I built rootfses for amd64 and armhf out of trusty-proposed:

https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/trusty/docker-ubuntu-core/+build/113861
https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/trusty/docker-ubuntu-core/+build/113862

I downloaded these and the builds from a few hours previously that were built out of updates. I confirmed with the help of qemu-arm-static that "apt update" failed in the armhf rootfs from updates and succeeded in the one from proposed.

I also unpacked all the tarballs and diffed the proposed and updates rootfses for each architecture and saw no unexpected changes.

tags: added: verification-done verification-done-trusty
removed: verification-needed verification-needed-trusty
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.208.15

---------------
livecd-rootfs (2.208.15) trusty; urgency=medium

  * Fix security mirror sources.list entries for non-x86 architectures by
    backporting trunk revision 1408. (LP: #1711735)

 -- Michael Hudson-Doyle <email address hidden> Tue, 31 Oct 2017 10:33:13 +1300

Changed in livecd-rootfs (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

ruffsl (roxfoxpox) wrote :

Could the "current" folder in Index of /core/trusty at https://partner-images.canonical.com/core/trusty/ be updated to redirect to 20171102 instead of 20170817, as it is presently? The docker library ubuntu image build is I think pointed to "current", and I would like to see the fix released into the official docker hub arm32v7 ubuntu trusty image.

On Mon, Nov 06, 2017 at 11:19:10PM -0000, ruffsl wrote:
> Could the "current" folder in Index of /core/trusty at https://partner-
> images.canonical.com/core/trusty/ be updated to redirect to 20171102
> instead of 20170817, as it is presently? The docker library ubuntu image
> build is I think pointed to "current", and I would like to see the fix
> released into the official docker hub arm32v7 ubuntu trusty image.

At the moment, we aren't successfully building images for all
architectures; we'll only move the symlink once we are. That said, we
know what the problem with the failing architectures is (it's a bug
related to moving the Launchpad image build infrastructure from chroots
to lxd containers) and the fix for it is currently working its way
through validation to production, so it shouldn't be a long wait.

The infrastructure got fixed and https://partner-images.canonical.com/core/trusty/current now points to builds that have this bug fixed, hooray.

Changed in cloud-images:
status: Confirmed → Fix Released
bugproxy (bugproxy) on 2017-11-20
tags: added: targetmilestone-inin14044
removed: targetmilestone-inin---
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.