Error creating an instance

[Chris MacNaughton]

OSCI failed a run of 'specs/full_stack/next_deploy zesty-ocata' with the following error:

#############################################################################
Create 1 * tiny cirros and 1 * small precise instances on the overcloud
#############################################################################

2017-10-17 08:00:15 [INFO] Pulling secrets from /tmp/tmp.M9zse4KV0C/LOCAL/mojo-openstack-specs/specs/full_stack/next_deploy/ocata to /tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/local
2017-10-17 08:00:15 [WARNING] Automatic secrets phase ran but secrets directory /tmp/tmp.M9zse4KV0C/LOCAL/mojo-openstack-specs/specs/full_stack/next_deploy/ocata does not exist!
2017-10-17 08:00:15 [INFO] Running script simple_os_checks.py
2017-10-17 08:00:52 [WARNING] No debug log matching debug-logs found. Using default.
2017-10-17 08:00:54 [ERROR] 2017-10-17 08:00:18 [INFO] Using keystone API V2 for overcloud auth
2017-10-17 08:00:21 [INFO] Creating key mojo
2017-10-17 08:00:23 [INFO] Adding ssh security group rule
2017-10-17 08:00:23 [INFO] Adding ping security group rule
2017-10-17 08:00:23 [INFO] Using config /tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/spec/specs/full_stack/next_deploy/ocata/images.yaml
2017-10-17 08:00:25 [INFO] Creating m1.tiny cirros [{'net-id': u'0268f0ed-3859-442b-99a6-a7779fb5f056'}]instance 20171017080025
2017-10-17 08:00:33 [INFO] Issued boot
2017-10-17 08:00:33 [INFO] Launched <Server: 20171017080025>
2017-10-17 08:00:33 [INFO] Waiting for boot
2017-10-17 08:00:33 [INFO] Waiting 180s for 20171017080025 to reach ACTIVE state
2017-10-17 08:00:52 [ERROR] instance 20171017080025 in unknown state ERROR
Traceback (most recent call last):
  File "/tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/spec/specs/full_stack/next_deploy/ocata/simple_os_checks.py", line 89, in <module>
    sys.exit(main(sys.argv))
  File "/tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/spec/specs/full_stack/next_deploy/ocata/simple_os_checks.py", line 85, in main
    ping_wait=ping_wait)
  File "/tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/spec/helper/tests/utils/mojo_os_utils.py", line 790, in boot_and_test
    cloudinit_wait)
  File "/tmp/tmp.M9zse4KV0C/mojo-openstack-specs/zesty/osci-mojo/spec/helper/tests/utils/mojo_os_utils.py", line 645, in wait_for_boot
    raise Exception('Error initialising %s' % vm_name)
Exception: Error initialising 20171017080025

[Aurelien Lourot (aurelien-lourot)]

Happened twice recently:
http://osci:8080/view/MojoMatrix/job/mojo_runner/22587
http://osci:8080/view/MojoMatrix/job/mojo_runner/22605

[Aurelien Lourot (aurelien-lourot)]

Seems to now be always (3 times in a row) happening on bionic-train to next. Last occurrence:
http://osci:8080/view/MojoMatrix/job/mojo_runner/22848/console

[Ryan Beisner (1chb1n)]

To clarify, #3 is an OpenStack upgrade of Train to Ussuri.

[Alex Kavanagh (ajkavanagh)]

So this is the issue: not all of the apps are being upgraded to ussuri:

app: ceilometer origin: cloud:bionic-ussuri
app: ceilometer-agent origin: <not known>
app: ceph-mon origin: cloud:bionic-train
app: ceph-osd origin: cloud:bionic-train
app: cinder origin: cloud:bionic-ussuri
app: cinder-ceph origin: <not known>
app: glance origin: cloud:bionic-ussuri
app: gnocchi origin: cloud:bionic-train
app: keystone origin: cloud:bionic-ussuri
app: memcached origin: <not known>
app: mysql origin: cloud:bionic-train
app: neutron-api origin: cloud:bionic-ussuri
app: neutron-gateway origin: cloud:bionic-ussuri
app: neutron-openvswitch origin: <not known>
app: nova-cloud-controller origin: cloud:bionic-ussuri
app: nova-compute origin: cloud:bionic-ussuri
app: openstack-dashboard origin: cloud:bionic-ussuri
app: placement origin: cloud:bionic-train
app: rabbitmq-server origin: cloud:bionic-train
app: swift-proxy origin: cloud:bionic-train
app: swift-storage-z1 origin: cloud:bionic-train
app: swift-storage-z2 origin: cloud:bionic-train
app: swift-storage-z3 origin: cloud:bionic-train

(I wrote a script to extract that.)

So when nova talks to placement it uses a trait (COMPUTE_NODE) that doesn't exist in train; hence the failure. Now to work out why the upgrade isn't doing all the applications before moving on.

[Alex Kavanagh (ajkavanagh)]

So #5 issue is that placement isn't upgraded to train and nova-cc at ussuri tries to talk to it; hence the problem. This was solved by: https://github.com/openstack-charmers/zaza-openstack-tests/pull/455

[Alex Kavanagh (ajkavanagh)]

This has been overtaken by events. The error is no longer taking place.

[Aurelien Lourot (aurelien-lourot)]

FWIW it's still happening on OSCI:
http://osci:8080/job/mojo_runner/24882/console