timeouts on wait_for nova list in tripleo-ci

[Derek Higgins]

Tripleo-ci seems to be having a lot of timeouts while waiting for the seed to initialize.

devdest_seed contains
 wait_for 30 10 ping -c 1 192.0.2.1
 <keystone initialization ....>
 wait_for 30 10 nova list

The first wait_for is satisfied by configure.d/80-neutron-openvswitch-agent, the second wait_for doesn't pass until nova is initialized (in post-configure.d/80-nova-api)

So it looks like something between these two scripts has gotten slower or the keystone initialization has gotten faster

dib-run-parts Mon Feb 24 04:52:16 UTC 2014 Running /opt/stack/os-config-refresh/configure.d/80-neutron-openvswitch-agent
dib-run-parts Mon Feb 24 04:52:17 UTC 2014 Running /opt/stack/os-config-refresh/configure.d/80-nova-baremetal
dib-run-parts Mon Feb 24 04:52:17 UTC 2014 Running /opt/stack/os-config-refresh/configure.d/81-nat-metadata
dib-run-parts Mon Feb 24 04:52:18 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/40-mysql
dib-run-parts Mon Feb 24 04:52:21 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/40-rabbitmq
dib-run-parts Mon Feb 24 04:52:21 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/50-mysql-users
dib-run-parts Mon Feb 24 04:52:21 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/50-rabbitmq-passwords
dib-run-parts Mon Feb 24 04:52:21 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/70-keystone
dib-run-parts Mon Feb 24 04:53:40 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/75-glance
dib-run-parts Mon Feb 24 04:54:39 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/79-heat-api
dib-run-parts Mon Feb 24 04:54:42 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/79-heat-engine
dib-run-parts Mon Feb 24 04:55:29 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-neutron-dhcp-agent
dib-run-parts Mon Feb 24 04:55:30 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-neutron-openvswitch-agent
dib-run-parts Mon Feb 24 04:55:32 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-neutron-server
dib-run-parts Mon Feb 24 04:55:33 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-nova-api
dib-run-parts Mon Feb 24 04:58:24 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-nova-baremetal

[Derek Higgins (derekh)]

I compared how long it takes to run some of these scripts on the test env,

70-keystone (ci testenv 79s , locally 19s)
75-glance (ci testenv 59s , locally 16s)
80-nova-api (ci testenv 171s , locally 49s)

Do we just need to increase the timeout to allow for slower environments?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/76212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/76212
Committed: https://git.openstack.org/cgit/openstack/tripleo-incubator/commit/?id=61d216f5b3d1b294e9d03cc4d3150b0b8b52bfb0
Submitter: Jenkins
Branch: master

commit 61d216f5b3d1b294e9d03cc4d3150b0b8b52bfb0
Author: Derek Higgins <email address hidden>
Date: Tue Feb 25 14:12:27 2014 +0000

    Increase time waiting for nova to initialize

    Various parts of the boot process take longer on the ci test
    environments when running multiple tests on the same host
    simultaneously, specifically the various db syncs seem to take upto 4
    times as long(when comparing to a local test). To allow for this we
    give nova some more time to be initialized.

    Long term we should spend some time investigating what the bottleneck is
    to improve the performance.

    Partial-Bug: #1284054
    Change-Id: I3df6f24fd906e0fd74c36e2ba96e2b731e60416b

[Robert Collins (lifeless)]

Marked F-R, we wouldn't get to 'its slow' otherwise...