[1.9.0] 400 Error juju bootstrapping missing images

[Adam Collard]

2016-01-22 08:38:05 [-] 127.0.0.1 - - [22/Jan/2016:14:38:04 +0000] "POST /MAAS/api/1.0/nodes/node-d6072d54-4b4c-11e4-ad24-a0b3cce4ecca/?op=start HTTP/1.1" 400 169 "-" "Go 1.1 package http"

juju ended with exit code 1 (out='', err='Bootstrapping environment "4"
Starting new instance for initial state server
Launching instance
Bootstrap failed, destroying environment
ERROR failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"osystem": ["'ubuntu' is not a valid osystem. It should be one of: ''."], "distro_series": ["'ubuntu/trusty' is not a valid distro_series. It should be one of: ''."]})
')

[Adam Collard (adam-collard)]

(yes, images were synced)

[Blake Rouse (blake-rouse)]

I looks like the cluster was disconnected when this operation was being performed. Can you in the clusterd.log that the cluster was connected to the region during that time frame. If it was then there should also be a stacktrace in clusterd.log on why it was unable to provide the available images back to the region.

[Adam Collard (adam-collard)]

2016-01-22 07:11:21-0600 [-] Stopping protocol <tftp.bootstrap.RemoteOriginReadSession instance at 0x7f881dca2170>
2016-01-22 09:18:30-0600 [TFTP (UDP)] Datagram received from ('10.1.101.143', 2070): <RRQDatagram(filename=pxelinux.0, mode=octet, options={'tsize': '0'})>

these were consecutive log messages, the local time when I tried this was 08:37

[Adam Collard (adam-collard)]

2016-01-22 08:24:29 [HTTPChannel,57,10.0.7.8] Closing connection: <STATUSES=GOING_AWAY>

this was in regiond.log - looks a bit odd (10.0.7.8 is the local ip hosting both the region and cluster controllers and is the MAAS_URL)

[Blake Rouse (blake-rouse)]

Can you please provide a cross section of all the maas logs around that time frame like 10 minutes before and 10 minutes after?

[Adam Collard (adam-collard)]

/var/log/maas $ rgrep '22 08:[234]' *.log

http://paste.ubuntu.com/14598638/

[Andreas Hasenack (ahasenack)]

The same happened again with 1.9.0. And I think an image update was happening in the background. At least, the logs show one started one minute before juju bootstrap failed saying that trusty is not a valid ubuntu release.

The error we got is this from juju:
01:11:39 2016-03-15 01:11:39 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})

I'm going to attach several logs, including the jenkins console where the above error appeared.

[Andreas Hasenack (ahasenack)]

Jenkins console snippet showing the bootstrap failure (--debug), with timestamps

[Andreas Hasenack (ahasenack)]

clusterd. I cut it a bit, since it had data since the 8th.

[Andreas Hasenack (ahasenack)]

I'm looking through our CI and I have other failed bootstraps, same error, in different runs in that timeframe:

WORKED:
#1381: 01:17:01 2016-03-15 01:17:00 INFO juju.cmd cmd.go:129 Starting new instance for initial state server

FAILED:
#1380: 01:11:39 2016-03-15 01:11:39 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1379: 01:07:37 2016-03-15 01:07:37 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1378: 01:04:13 2016-03-15 01:04:13 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1377: 01:00:04 2016-03-15 01:00:04 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1376: 00:56:44 2016-03-15 00:56:44 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1375: 00:53:27 2016-03-15 00:53:27 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})
#1374: 00:49:52 2016-03-15 00:49:52 ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'trusty' is not a valid distro_series. It should be one of: '', 'ubuntu/xenial'."]})

WORKED:
#1373: 23:07:52 2016-03-14 23:07:52 INFO juju.cmd cmd.go:129 Starting new instance for initial state server

[Larry Michel (lmic)]

bug 1558188 looks like a duplicate of this one.

[[e-mail] (skiba1122)]

When bootstrapping i get
ERROR juju.cmd supercommand.go:429 failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'xenial' is not a valid distro_series. It should be one of: '', 'ubuntu/trusty'."]})

[Paul Larson (pwlars)]

I am seeing something that looks like this too. I'm on 1.8.0+bzr4001-0ubuntu2~trusty1 from the stable ppa right now, but I'm reluctant to try upgrading just to fix this, since it seems that people on newer versions are experiencing this problem as well.

I noticed something strange today though, xenial was listed in maas boot-resources read when I checked on Friday, but today I don't see it there.

[Gavin Panella (allenap)]

Upgrading to 1.9.x for pwlars seems to have helped. Conversation noted
here for reference:

<allenap> plars: To what extent can we play around with your environment
  to try and diagnose https://bugs.launchpad.net/maas/+bug/1537095?
<allenap> I got the impression it was "not a lot".
<plars> allenap: well unfortunately it's no longer reproducible since I
  upgraded to 1.9.2
<plars> allenap: it seemed like it was never fully importing, but I
  couldn't find any log entries telling me that. On the UI it still had
  it checked, but when I looked at maas boot-resources read, it didn't
  list it
<plars> allenap: the other weird thing was that it called it xenial in
  the ui, not 16.04
<plars> allenap: don't know if that helps, but just in case

[Gavin Panella (allenap)]

I can't reproduce this bug in 2.0, and is infrequent enough in 1.8 and 1.9 that we can't progress it further right now.

[JuanJo Ciarlante (jjo)]

I'm hitting exactly this, with:
- trusty maas node
- maas 1.9.3+bzr4577-0ubuntu1 (upgraded from 1.9.0)
- only 14.04 LTS images at maas
- juju 1.24.7 (preferred vs 1.25 as it's less race-y)
, when doing:

juju bootstrap --upload-tools --series trusty
-> http://paste.ubuntu.com/21029935/

If I let 16.04 images at maas, above cmd succeeds, but
deploys a xenial unit.

[JuanJo Ciarlante (jjo)]

(#18 cont), error:

ERROR failed to bootstrap environment: cannot start bootstrap instance: gomaasapi: got error back from server: 400 BAD REQUEST ({"distro_series": ["'xenial' is not a valid distro_series. It should be one of: '', 'ubuntu/trusty'."]})

[JuanJo Ciarlante (jjo)]

FTR tried without success:
- juju 1.25.5
- at maas images UI: select 16.04 only+apply, wait for logs to settle, repeat for 14.04 only

[JuanJo Ciarlante (jjo)]

Workaround that WfM: downgrade distro-info-data to pre-xenial, with:

apt-get install distro-info-data=0.18

, so that this trusty install gives instead:

$ distro-info --lts
trusty

Just FTR/FAOD below now works as expected:

juju bootstrap --upload-tools --series trusty

[Miguel Meneses (miguel.meneses)]

Hi there,

Currently I am working with 4 VMs
I successfully commission the first 3 VM, but on the 4 VM It tries to load the xenial image ( its kernel ), the one doesn't exist or has been downloaded

Regards

[Miguel Meneses (miguel.meneses)]

I mean has not been downloaded

[Andres Rodriguez (andreserl)]

Hi All,

Does this continue to be an issue? it doens't seem to be the case. AS such I'm marking this as invalid. If you find the issue again, please file a new bug or re-open this one.