Precise i386 deployment failed with root-tgz file not found error

[Po-Hsu Lin]

The Precise deployment for the kernel SRU test will fail with:

curtin: Installation started. (18.1-17-gae48e86f-0ubuntu1~18.04.1)

third party drivers not installed or necessary.

--2018-12-11 06:29:35-- http://10.246.72.3:5248/images/ubuntu/i386/generic/precise/daily/root-tgz

Connecting to 10.246.72.3:5248... connected.

HTTP request sent, awaiting response... 404 Not Found

2018-12-11 06:29:35 ERROR 404: Not Found.

tar: This does not look like a tar archive

tar: Exiting with failure status due to previous errors

Unexpected error while running command.

Command: ['sh', '-cf', 'wget "$1" --progress=dot:mega -O - |smtar -C "$2" --xattrs --xattrs-include=* -Sxpf - --numeric-owner', '--', 'http://10.246.72.3:5248/images/ubuntu/i386/generic/precise/daily/root-tgz', '/tmp/tmpz4anrjio/target']

Exit code: 2

Reason: -

Stdout: ''

Stderr: ''

curtin: Installation failed with exception: Unexpected error while running command.

Command: ['curtin', 'extract']

Exit code: 3

Reason: -

Stdout: --2018-12-11 06:29:35-- http://10.246.72.3:5248/images/ubuntu/i386/generic/precise/daily/root-tgz

        Connecting to 10.246.72.3:5248... connected.

        HTTP request sent, awaiting response... 404 Not Found

        2018-12-11 06:29:35 ERROR 404: Not Found.

        tar: This does not look like a tar archive

        tar: Exiting with failure status due to previous errors

        Unexpected error while running command.

        Command: ['sh', '-cf', 'wget "$1" --progress=dot:mega -O - |smtar -C "$2" --xattrs --xattrs-include=* -Sxpf - --numeric-owner', '--', 'http://10.246.72.3:5248/images/ubuntu/i386/generic/precise/daily/root-tgz', '/tmp/tmpz4anrjio/target']

        Exit code: 2

        Reason: -

        Stdout: ''

        Stderr: ''

Stderr: ''

Here is what we have in http://10.246.72.3:5248/images/ubuntu/i386/generic/precise/daily/
Filename Size Content type Content encoding
boot-initrd 17M [text/html]
boot-kernel 4M [text/html]
root-image.gz 250M [text/html] [gzip]

And from the server side
/var/lib/maas/boot-resources/current/ubuntu/i386/generic/precise/daily$ ls -lh
total 253M
-rw-r--r-- 3 maas maas 18M Dec 6 17:03 boot-initrd
-rw-r--r-- 3 maas maas 4.7M Dec 6 17:03 boot-kernel
-rw-r--r-- 7 maas maas 251M Dec 6 17:43 root-image.gz

The strange part is that AMD64 Precise deployment works.

/var/lib/maas/boot-resources/current/ubuntu/amd64/generic/precise/daily$ ls -lh
total 227M
-rw-r--r-- 3 maas maas 19M Dec 6 16:43 boot-initrd
-rw-r--r-- 3 maas maas 4.8M Dec 6 16:43 boot-kernel
-rw-r--r-- 3 maas maas 204M Dec 6 16:43 squashfs

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: maas 2.4.2-7034-g2f5deb8b8-0ubuntu1
ProcVersionSignature: Ubuntu 4.15.0-39.42-generic 4.15.18
Uname: Linux 4.15.0-39-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Tue Dec 11 06:31:03 2018
PackageArchitecture: all
SourcePackage: maas
UpgradeStatus: Upgraded to bionic on 2018-12-06 (4 days ago)

[Po-Hsu Lin (cypressyew)]

BTW Precise i386 deployment works fine before we upgrade the MaaS from 2.3 something to 2.4.2

[Andres Rodriguez (andreserl)]

So MAAS is correctly trying to download the root-tgz but the files avialable do not include a root-tgz. I also looked in the code and it seems that root-image.gz should be converted into root-tgz:

    #: Root Image (gets converted to root-image root-tgz, on the rack)
    ROOT_IMAGE = 'root-image.gz'

But:

root@maas-bionic:/var/lib/maas/boot-resources/current/ubuntu/i386/generic# ls -l /var/lib/maas/boot-resources/current/ubuntu/i386/generic/precise/daily/
total 278696
-rw-r--r-- 3 maas maas 18318877 Dec 12 23:44 boot-initrd
-rw-r--r-- 3 maas maas 4910272 Dec 12 23:44 boot-kernel
-rw-r--r-- 7 maas maas 262145114 Dec 12 23:44 root-image.gz

[Andres Rodriguez (andreserl)]

So it seems MAAS is no longer creating the root-tgz, and that's the bug.

[Andres Rodriguez (andreserl)]

or maybe, this is a regression on the image streams side because seems at a certain point we started building precise images: https://git.launchpad.net/maas-images/commit/?id=49fb1e42572873184104d77c027e0b823d726f8a

[Andres Rodriguez (andreserl)]

squashfs images that is.

[Andres Rodriguez (andreserl)]

Actually it seems this is because we may have dropped support for i386 in ubuntu altogether because the squashfs image exists for amd64:

root@maas-bionic:/var/lib/maas/boot-resources/current/ubuntu/i386/generic# ls /var/lib/maas/boot-resources/current/ubuntu/amd64/generic/precise/daily/
boot-initrd boot-kernel squashfs

[Po-Hsu Lin (cypressyew)]

Hi Andres,

thanks for the analysis, do you know who should I talk to for this issue?
We need this image for the Precise ESM SRU testing.

Thanks!

[Sean Feole (sfeole)]

Thanks Sam for filing this, the kernel team relies on this to be working in MAAS, so if there is any assistance we can offer, please let us know.

[Terry Rudd (terrykrudd)]

Hello Andres,

To further Po-Hsu Lin's point, with extended support for some Ubuntu kernels going out to 10 years now, it seems extra care will need to be taken for regression testing our older kernels since this bug would also potentially impact those that still deploy extended-life kernels in their legacy environments.

It is expensive and time-consuming to work around this bug.

Thanks!
Terry

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team