os_collect_config.ec2 [...] Max retries exceeded with url: /latest/meta-data/ [...] Network is unreachable

[Therese McHale]

Got build failure

2014-03-28 13:56:48.351 | ++ OS_NO_CACHE=True
2014-03-28 13:56:48.351 | ++ export OS_CLOUDNAME=seed
2014-03-28 13:56:48.351 | ++ OS_CLOUDNAME=seed
2014-03-28 13:56:48.352 | + nova list
2014-03-28 13:56:48.785 | +----+------+--------+------------+-------------+----------+
2014-03-28 13:56:48.785 | | ID | Name | Status | Task State | Power State | Networks |
2014-03-28 13:56:48.785 | +----+------+--------+------------+-------------+----------+
2014-03-28 13:56:48.822 | +----+------+--------+------------+-------------+----------+
2014-03-28 13:56:48.822 | ++ grep -oE '([0-9]+\.){3}[0-9]+'
2014-03-28 13:56:48.823 | ++ nova list
2014-03-28 13:56:49.300 | ERROR: the main setup script run by this job failed - exit code: 1
2014-03-28 13:56:49.301 | please look at the relevant log files to determine the root cause
2014-03-28 13:56:49.301 | Cleaning up host
2014-03-28 13:56:49.301 | ... this takes 3 - 4 minutes (logs at logs/devstack-gate-cleanup-host.txt.gz)
2014-03-28 13:56:53.058 | Build step 'Execute shell' marked build as failure

seems not the same as 1298946 which has a specific ssl failure.

On failure I can see in the logs is :devstack-gate-setup-workspace-new.txt :

2014-03-28 13:14:36.000 | + git fetch http://zm01.openstack.org/p/openstack-dev/devstack refs/zuul/master/Z490e5e53e05b433f966c68241cb64301
2014-03-28 13:14:36.449 | fatal: Couldn't find remote ref refs/zuul/master/Z490e5e53e05b433f966c68241cb64301
2014-03-28 13:14:36.455 | Unexpected end of command stream

not clear if the above is expected

logs are here: http://logs.openstack.org/70/80570/6/check-tripleo/check-tripleo-ironic-undercloud-precise/f2824ea/

[Jeremy Stanley (fungi)]

The "ERROR: the main setup script run by this job failed - exit code: 1" message is referring to a failure encountered in the toci_gate_test.sh script, which was actually reporting its progress to the console. I haven't looked too closely, but it appears nova list may have returned an empty list, failing to match the patterns for which a grep was looking.

Those "fatal: Couldn't find remote ref refs/zuul/master/Z.*" messages in the workspace setup job are normal, and simply indicate that for one or more projects there was no zuul ref needed (and thus non served) when the script tried retrieving them (Git lacks a sane look-before-you-leap capability to check for the existence of a remote ref without emitting an error message, and there's no good way to silence those without losing other important error messages at the same time).

[Therese McHale (therese-mchale)]

The seed logs show

os-collect-config.log shows the script associated with the fix completed ok:

...
dib-run-parts Fri Mar 28 13:46:22 UTC 2014 60-mysql-secure completed
...

though it shows this at the start:

2014-03-28 13:46:08.818 1155 WARNING os_collect_config.ec2 [-] HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/ (Caused by <class 'socket.'>: [Errno 101] Network is unreachable)

however these logs show:

/var/log/upstart/keystone.log:

2014-03-28 13:46:43.058 3800 WARNING keystone.common.wsgi [-] Could not find credential, 419df94216a922746fee972826c9573f2758bfd4b432760769fe424bfca5a6e4.
2014-03-28 13:46:46.043 3800 WARNING keystone.common.wsgi [-] Could not find user, neutron.
2014-03-28 13:46:49.441 3800 WARNING keystone.common.wsgi [-] Could not find user, glance.
2014-03-28 13:46:52.649 3800 WARNING keystone.common.wsgi [-] Could not find user, ec2.
2014-03-28 13:46:57.364 3800 WARNING keystone.common.wsgi [-] Could not find user, nova.
2014-03-28 13:47:00.567 3800 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 127.0.0.1
2014-03-28 13:47:01.790 3800 WARNING keystone.common.wsgi [-] Could not find user, nova.
2014-03-28 13:47:03.393 3800 WARNING keystone.common.wsgi [-] Could not find user, ironic.
2014-03-28 13:47:09.749 3800 WARNING keystone.common.wsgi [-] Could not find credential, ab992fd10b8013ec7bb125212adcb299087cf3984ef487bde0357b50fb434664.
2014-03-28 13:47:09.788 3800 WARNING keystone.common.wsgi [-] Could not find credential, 1a62a41b7c3e50cc7de74538a291cb8a6818d7c89cb613bb2f2b7d069ee4b82f.
2014-03-28 13:47:13.243 3800 WARNING keystone.common.wsgi [-] Could not find credential, 419df94216a922746fee972826c9573f2758bfd4b432760769fe424bfca5a6e4.
...

nova-api.log shows:
2014-03-28 13:47:09.736 4446 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.4
2014-03-28 13:47:09.777 4446 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.5
2014-03-28 13:47:13.231 4446 ERROR nova.api.metadata.handler [-] Failed to get metadata for ip: 192.0.2.3
...

ironic-conductor.log shows:

stevedore.extension): 2014-03-28 13:46:28,401 ERROR Could not load 'pxe_seamicro': PXEAndSeaMicroDriver
(stevedore.extension): 2014-03-28 13:46:28,402 ERROR PXEAndSeaMicroDriver
Traceback (most recent call last):
  File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/stevedore/extension.py", line 162, in _load_plugins
    verify_requirements,
  File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/stevedore/enabled.py", line 66, in _load_one_plugin
    verify_requirements,
  File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/stevedore/extension.py", line 180, in _load_one_plugin
    obj = plugin(*invoke_args, **invoke_kwds)
  File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/drivers/pxe.py", line 102, in __init__
    raise exception.DriverNotFound('PXEAndSeaMicroDriver')
DriverNotFound: PXEAndSeaMicroDriver
(stevedore.extension): 2014-03-28 13:46:28,410 ERROR Could not load 'fake_seamicro': FakeSeaMicroDriver
(stevedore.extension): 2014-03-...

[Lance Bragstad (lbragstad)]

e-r query:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRVJST1I6IHRoZSBtYWluIHNldHVwIHNjcmlwdCBydW4gYnkgdGhpcyBqb2IgZmFpbGVkIC0gZXhpdCBjb2RlOiAxXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDE4MDIyMzY5ODR9

Hit this in Keystone on an .rst change:

http://logs.openstack.org/66/96966/3/check/check-tempest-dsvm-postgres-full/6987b19/console.html