maas-import-pxe-files breaks on 404

[Gavin Panella]

Fresh MAAS install from the dailybuilds PPA. Clicking "Import boot images" in the UI was not having the desired effect: most boot resources (kernel, initrd) were being imported, but ephemeral and install images were not. Running by hand showed the problem:

# maas-import-pxe-files
Downloading to temporary location /tmp/tmp.EKKx3SdU2s.
/tmp/tmp.EKKx3SdU2s ~
2013-08-14 21:52:33 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//linux [4965840/4965840] -> "linux" [1]
2013-08-14 21:53:02 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//initrd.gz [17446386/17446386] -> "initrd.gz" [1]
2013-08-14 21:53:16 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//linux [5130968/5130968] -> "linux" [1]
2013-08-14 21:53:54 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//initrd.gz [18668122/18668122] -> "initrd.gz" [1]
2013-08-14 21:54:05 URL:http://archive.ubuntu.com/ubuntu//dists/raring/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//linux [5357848/5357848] -> "linux" [1]
2013-08-14 21:54:43 URL:http://archive.ubuntu.com/ubuntu//dists/raring/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64//initrd.gz [19119185/19119185] -> "initrd.gz" [1]
2013-08-14 21:54:55 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386//linux [5015840/5015840] -> "linux" [1]
2013-08-14 21:55:22 URL:http://archive.ubuntu.com/ubuntu//dists/precise/main/installer-i386/current/images/netboot/ubuntu-installer/i386//initrd.gz [15977428/15977428] -> "initrd.gz" [1]
2013-08-14 21:55:31 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-i386/current/images/netboot/ubuntu-installer/i386//linux [5171760/5171760] -> "linux" [1]
2013-08-14 21:56:05 URL:http://archive.ubuntu.com/ubuntu//dists/quantal/main/installer-i386/current/images/netboot/ubuntu-installer/i386//initrd.gz [17086667/17086667] -> "initrd.gz" [1]
2013-08-14 21:56:13 URL:http://archive.ubuntu.com/ubuntu//dists/raring/main/installer-i386/current/images/netboot/ubuntu-installer/i386//linux [5367344/5367344] -> "linux" [1]
2013-08-14 21:56:41 URL:http://archive.ubuntu.com/ubuntu//dists/raring/main/installer-i386/current/images/netboot/ubuntu-installer/i386//initrd.gz [17454164/17454164] -> "initrd.gz" [1]
2013-08-14 21:56:45 URL:http://ports.ubuntu.com/ubuntu-ports//dists/precise-updates/main/installer-armhf/current/images/highbank/netboot//vmlinuz [2978672/2978672] -> "vmlinuz" [1]
2013-08-14 21:56:56 URL:http://ports.ubuntu.com/ubuntu-ports//dists/precise-updates/main/installer-armhf/current/images/highbank/netboot//initrd.gz [4951617/4951617] -> "initrd.gz" [1]
2013-08-14 21:57:04 URL:http://ports.ubuntu.com/ubuntu-ports//dists/quantal/main/installer-armhf/current/images/highbank/netboot//vmlinuz [3738504/3738504] -> "vmlinuz" [1]
2013-08-14 21:57:20 URL:http://ports.ubuntu.com/ubuntu-ports//dists/quantal/main/installer-armhf/current/images/highbank/netboot//initrd.gz [6213909/6213909] -> "initrd.gz" [1]
http://ports.ubuntu.com/ubuntu-ports//dists/raring/main/installer-armhf/current/images/highbank/netboot//vmlinuz:
2013-08-14 21:57:21 ERROR 404: Not Found.

I would expect more feedback on this error, and would also expect MAAS to continue to download other resources, perhaps disabling installations of Raring on armhf/highbank.

[Julian Edwards (julian-edwards)]

Linked private blueprint which has two work items relating to this bug (UI progress indicator and better handling of errors)

[Raphaël Badin (rvb)]

As discussed a while ago, rewritting the current import scripts in python is likely to drastically decrease their complexity and improve their testability by a lot. This would require further analysis but I wouldn't be surprised if doing the rewrite turns out to be the quickest solution to fix all the problems/feature requests related to these scripts.

[Julian Edwards (julian-edwards)]

Yes I would be expecting a re-write on here simply because I don't think supporting the other features will be viable in bash.

[Raphaël Badin (rvb)]

Raising this to critical as it breaks MAAS completely.

[Raphaël Badin (rvb)]

Note that even though the fact that the script breaks when an image is missing is the real bug here, this is related to bug 1166994.

[Scott Moser (smoser)]

FWIW, I'd much rather us get real data from a sane server side, then scrape around on mulitple different urls.

My goal, which I've told lots of people, but it seems has not stuck anywhere was to get maas-import-pxe-files replaced with a sipmlestreams client and get simplestreams server side data.

The code at: https://code.launchpad.net/~smoser/maas/netinst2ss
basically scrapes the archives and represents the data in a simplestreams format that can then be mirrored.

[Scott Moser (smoser)]

See https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1223033

[Gavin Panella (allenap)]

> My goal, which I've told lots of people, but it seems has not stuck
> anywhere was to get maas-import-pxe-files replaced with a
> sipmlestreams client and get simplestreams server side data.

It's being worked on already, but it's not finished.

Once it's done, are we going to backport it, or is 12.04 MAAS going to
keep using the current method of obtaining boot resources?

[Scott Moser (smoser)]

Gavin,
  Your response shows misunderstanding.
  There are 2 things that maas downloads:
    maas-import-pxe-files:
      This downloads d-i (debian-installer) initramfs and kernels from ports.archive.ubuntu.com and archive.ubuntu.com.
    maas-import-ephemerals
      This downloads ephemeral images (which include a kernel and initramfs) from maas.ubuntu.com/images.

The thing that is broken here is 'maas-import-pxe-files'.
The thing that Tycho's work fixes/improves is 'maas-import-ephemerals'.

My goal was to make 'maas-import-pxe-files' use simplestreams data. Already "In progress" is making maas-import-ephemerals use simplestreams data.

[Jeroen T. Vermeulen (jtv)]

Scott: it was our understanding that the simplestreams-enabled version of the ephemerals script, once operational, would be easy to extend to the install images, and indeed that this would be the logical next step. Is that not the case? We were expecting to move entirely to simplestreams very soon.

One other thing that would be needed for that, of course, is to have the install images represented in a stream as well. Assuming this approach makes sense, is that something you could provide?

[Julian Edwards (julian-edwards)]

On 27/09/13 03:08, Gavin Panella wrote:
> Once it's done, are we going to backport it, or is 12.04 MAAS going to
> keep using the current method of obtaining boot resources?

I would hope so - it's trivial to backport/SRU provided the
simplestreams feed gives the necessary data for older releases.

[Jeroen T. Vermeulen (jtv)]

The simplestreams data does include older releases. The changes to the code are fairly extensive, but merge relatively cleanly into 1.2/1.3 so far.

Meanwhile, the fix to the bug we're discussing here (maas-import-pxe-files breaking when an install image fails to download) has already been backported to 1.2 and 1.3.