source.list for armhf includes trusty-security which does not exist for arm

[ruffsl]

[Impact]
The armhf cloud image fails during apt-update as the source.list includes a reference to trusty-security, a source that does not appear to host support for that architecture:
http://security.ubuntu.com/ubuntu/dists/trusty-security/

This issue is blocking the use of the armhf version of ubuntu on docker hub's official library:
https://github.com/tianon/docker-brew-ubuntu-core/issues/94#issuecomment-323401913

The latest image possesses this issue,
but might want to check other image architecture and suites as well:

Image: ubuntu-trusty-core-cloudimg-armhf-root.tar.gz
SHA256SUM: c264a88662bbb1d5cdf0deb888720b4b508ad3e5054707d017a1f84d31827bc7
Link: https://partner-images.canonical.com/core/trusty/20170817/

[Test Case]
Unpack the armhf chroot and attempt to run apt-get update in it. If it succeeds, this bug is fixed.

[Regression Potential]
Part of the verification should be to diff the trees produced by versions of livecd-rootfs in -updates and -proposed. The only non-trivial difference should be in /etc/apt/sources.list (there will be some differences timestamps in log files and obviously different indexes will be present in /var/lib/apt/lists) and if that is the only change then there is no regression potential.

[Dan Watkins (oddbloke)]

I can confirm that this is happening, and it's in the output that comes from the buildds, so it's a livecd-rootfs/live-build bug.

(It's probably the former, so adding that package to this bug.)

[Dan Watkins (oddbloke)]

This doesn't exhibit in xenial.

[Dan Watkins (oddbloke)]

This was fixed for xenial in bug 1513529 and bug 1679252. (Specifically, the former refactored sources.list generation significantly and in a way which broke non-x86 architectures, and the latter remedied that.)

Unfortunately, the changes to sources.list generation were fairly major, so I don't think it would be appropriate to pull them all the way back to trusty; we should find a less invasive fix.

[Dan Watkins (oddbloke)]

The fix in bug 1679252 uses LB_PARENT_MIRROR_BINARY_SECURITY for security lines (where currently trusty uses LB_PARENT_MIRROR_BINARY; I don't _think_ this should work because LB_PARENT_MIRROR_BINARY is used for the other sources.list lines which are fine, but it's worth a try.

Commit with that change pushed up to https://code.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/+git/livecd-rootfs/+ref/bug/1711735, and I've kicked off a build of livecd-rootfs to attempt a build from.

[ruffsl (roxfoxpox)]

Ya! thanks for following up @daniel-thewatkins , I was just about to message you as I found you helped to resolve another previous docker image related bug 1645463 with @tianon.

> we should find a less invasive fix.

What would you have in mind. Is there another way we could resolve this upstream, or should we turn to the user to make a patch to the source.list before use, e.g. modifying the downstream debootstrap additionally for docker to make the fix per trusty?
https://github.com/moby/moby/blob/9a9fc01af8fb5d98b8eec0740716226fadb3735c/contrib/mkimage/debootstrap#L67

[Dan Watkins (oddbloke)]

I still think we should fix this in Ubuntu, just not with the full fix from the aforementioned bugs. :)

[ruffsl (roxfoxpox)]

Ok Dan, let me know if you'd like me to test anything once the build comes back.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[ruffsl (roxfoxpox)]

Hello Dan, was your latest commit successful in building a cloud image with a patched source list?
https://git.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/commit/?id=1bc0b2eec50726f46510df4a340d2f27f62756bb

[Dimitri John Ledkov (xnox)]

We need to SRU this into Trusty right?

[Dan Watkins (oddbloke)]

On Wed, Aug 23, 2017 at 07:10:16PM -0000, ruffsl wrote:
> Hello Dan, was your latest commit successful in building a cloud image with a patched source list?
> https://git.launchpad.net/~daniel-thewatkins/ubuntu/+source/livecd-rootfs/commit/?id=1bc0b2eec50726f46510df4a340d2f27f62756bb

The package itself failed to build, which I've addressed now; will
attempt an image build once it publishes out to my PPA.

[Dan Watkins (oddbloke)]

Unfortunately my change didn't fix the issue; I'll revisit this soon.

[ruffsl (roxfoxpox)]

Hello Dan, just following up. Any new ideas? I suppose I can modify our official docker hub manifest to omit trusty for armhf to at least get xenial images out of our farm.

[Dimitri John Ledkov (xnox)]

On 10 September 2017 at 08:55, ruffsl <email address hidden> wrote:
> Hello Dan, just following up. Any new ideas? I suppose I can modify our
> official docker hub manifest to omit trusty for armhf to at least get
> xenial images out of our farm.

Do not omit trusty for armhf, there are production systems that depend on it.

Apart from not having correct security stanzas are there any other
side-effects? Note it is mostly harmless as -updates is enabled by
default, and all security updates are published in both -security and
-updates. Thus "wrong" security stanza, is mostly harmless.

--
Regards,

Dimitri.

[Tianon Gravi (tianon)]

He means simply removing "trusty + armhf" for the "ros" and/or
"gazebo" images, I believe (not for the "ubuntu" image).

The harm this causes is that "apt-get update" fails, which causes the
whole "docker build" to fail.

Here's an example from ppc64el:

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/main/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/restricted/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/universe/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

W: Failed to fetch
http://security.ubuntu.com/ubuntu/dists/trusty-security/multiverse/binary-ppc64el/Packages
 404 Not Found [IP: 91.189.88.152 80]

E: Some index files failed to download. They have been ignored, or old
ones used instead.

[ruffsl (roxfoxpox)]

Just checking in to ask if there is anything I could assist for this issue.
Are there any docs on how build a cloud image using the source in livecd-rootfs?

[Tianon Gravi (tianon)]

This is still causing quite a _lot_ of downstream pain -- any chance it could be re-prioritized?

[Andrew Hsu (andrewhsu)]

It looks to me the ports.ubuntu.com endpoint is receiving updates for security fixes for non-amd64 architectures: http://ports.ubuntu.com/ubuntu-ports/dists/trusty-security/multiverse/

As a workaround until this problem is fixed, I've changed the apt repo definition to ports.ubuntu.com:

$ sed -i 's|security.ubuntu.com/ubuntu|ports.ubuntu.com/ubuntu-ports|' /etc/apt/sources.list

And then `apt-get update` work for me on trusty:

$apt-get update
...
Get:1 http://ports.ubuntu.com trusty-security/universe Sources [76.3 kB]
Get:2 http://ports.ubuntu.com trusty-security/main armhf Packages [688 kB]
Get:3 http://ports.ubuntu.com trusty-security/restricted armhf Packages [10.3 kB]
Get:4 http://ports.ubuntu.com trusty-security/universe armhf Packages [239 kB]
Hit http://ports.ubuntu.com trusty-security/multiverse armhf Packages
Hit http://ports.ubuntu.com trusty/universe Sources
Hit http://ports.ubuntu.com trusty/main armhf Packages
Hit http://ports.ubuntu.com trusty/restricted armhf Packages
Hit http://ports.ubuntu.com trusty/universe armhf Packages
Hit http://ports.ubuntu.com trusty/multiverse armhf Packages
Fetched 1013 kB in 17s (56.5 kB/s)
Reading package lists... Done

[Edward Vielmetti (edward-vielmetti)]

This affects the trusty image for arm64 as well, noted at https://github.com/docker-library/official-images/issues/3583 (which referred me back here).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-10-18 14:00 EDT-------
.

[Dimitri John Ledkov (xnox)]

@bugproxy

Reverse-proxying this bug report bears no meaning. Please do not track this launchpad issue. If you are experiencing this or a similar issue on POWER9 please open a new LTC bug report, and proxy it to launchpad afresh, explaining the issue you have.

[Adam Conrad (adconrad)]

Hello ruffsl, or anyone else affected,

Accepted livecd-rootfs into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.208.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Michael Hudson-Doyle (mwhudson)]

I built rootfses for amd64 and armhf out of trusty-proposed:

https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/trusty/docker-ubuntu-core/+build/113861
https://launchpad.net/~cloud-images-release-managers/+livefs/ubuntu/trusty/docker-ubuntu-core/+build/113862

I downloaded these and the builds from a few hours previously that were built out of updates. I confirmed with the help of qemu-arm-static that "apt update" failed in the armhf rootfs from updates and succeeded in the one from proposed.

I also unpacked all the tarballs and diffed the proposed and updates rootfses for each architecture and saw no unexpected changes.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.208.15

---------------
livecd-rootfs (2.208.15) trusty; urgency=medium

  * Fix security mirror sources.list entries for non-x86 architectures by
    backporting trunk revision 1408. (LP: #1711735)

 -- Michael Hudson-Doyle <email address hidden> Tue, 31 Oct 2017 10:33:13 +1300

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[ruffsl (roxfoxpox)]

Could the "current" folder in Index of /core/trusty at https://partner-images.canonical.com/core/trusty/ be updated to redirect to 20171102 instead of 20170817, as it is presently? The docker library ubuntu image build is I think pointed to "current", and I would like to see the fix released into the official docker hub arm32v7 ubuntu trusty image.

[Dan Watkins (oddbloke)]

On Mon, Nov 06, 2017 at 11:19:10PM -0000, ruffsl wrote:
> Could the "current" folder in Index of /core/trusty at https://partner-
> images.canonical.com/core/trusty/ be updated to redirect to 20171102
> instead of 20170817, as it is presently? The docker library ubuntu image
> build is I think pointed to "current", and I would like to see the fix
> released into the official docker hub arm32v7 ubuntu trusty image.

At the moment, we aren't successfully building images for all
architectures; we'll only move the symlink once we are. That said, we
know what the problem with the failing architectures is (it's a bug
related to moving the Launchpad image build infrastructure from chroots
to lxd containers) and the fix for it is currently working its way
through validation to production, so it shouldn't be a long wait.

[Michael Hudson-Doyle (mwhudson)]

The infrastructure got fixed and https://partner-images.canonical.com/core/trusty/current now points to builds that have this bug fixed, hooray.