multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""

[James Slagle]

Failed job:
http://logs.openstack.org/45/354345/1/check/gate-tripleo-ci-centos-7-nonha-multinode/fba5263/

Not sure if it's affecting all jobs yet or not.

[James Slagle (james-slagle)]

cinder scheduler error:

scheduler.log:353:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
scheduler.log:354:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
scheduler.log:355:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
scheduler.log:356:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
scheduler.log:357:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
scheduler.log:358:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
scheduler.log:359:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
scheduler.log:360:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/scheduler/manager.py", line 158, in create_volume
scheduler.log:361:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server flow_engine.run()
scheduler.log:362:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
scheduler.log:363:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
scheduler.log:364:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
scheduler.log:365:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
scheduler.log:366:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
scheduler.log:367:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server failures[0].reraise()
scheduler.log:368:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
scheduler.log:369:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
scheduler.log:370:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
scheduler.log:371:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
scheduler.log:372:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 146, in execute
scheduler.log:373:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.server reason=e)
scheduler.log:374:2016-08-11 18:53:25.492 26267 ERROR oslo_messaging.rpc.serve...

[James Slagle (james-slagle)]

it seems current-tripleo was promoted over the last few hours, and I dont know how that could have happened. it seems it promoted a bad repo with the cinder bug.

[James Slagle (james-slagle)]

this is the repo that got promoted:
http://trunk.rdoproject.org/centos7/c6/bd/c6bd3cb95b9819c03345f50bf2812227e81314ab_4e6dfa3c

[James Slagle (james-slagle)]

i believe it was this patch gone awry that may have accidentally promoted:
https://review.openstack.org/#/c/346949

i tried to promote the previous repo, but the script on the rdo side might not let us promote an old repo because it does not seem to be having any effect.

Anyway, I think I've fixed the patch to not falsely promote and if it passes it will promote a working repo.

[James Slagle (james-slagle)]

actually it looks like it was a combination of my patch and sagi's patch:
https://review.openstack.org/354002

i tested the ha job, which falsely reported as passed when the mitaka job passed
he tested the nonha job, which falsely reported as passed when the mitaka job passed

together, those 2 jobs triggered the promote script to promote the repo.

[James Slagle (james-slagle)]

that job did not pass due to new issues with current repos...

i've submitted a temporary pin to tripleo-ci to pin us back to the previous promoted repo:
https://review.openstack.org/354481

we can also check with the rdo admins to see if they can restore the old symlink for us.

the previous working current-tripleo was https://trunk.rdoproject.org/centos7/8e/b0/8eb0c893074f6ba912114ce50995ee0118930775_fc22fbaf/

[James Slagle (james-slagle)]

dmsimard rolled back the symlink for us

[Juan Antonio Osorio Robles (juan-osorio-robles)]

This is what I see in the logs:

Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager Traceback (most recent call last):
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager result = task.execute(**arguments)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 146, in execute
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager reason=e)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager self.force_reraise()
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager six.reraise(self.type_, self.value, self.tb)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 126, in execute
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager filter_properties)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 86, in schedule_create_volume
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager filter_properties)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 416, in _schedule
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager filter_properties)
Aug 12 01:36:52 centos-7-2-node-rax-ord-3415467-148595 cinder-scheduler: 2016-08-12 01:36:52.957 25038 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/filter_schedule...

[James Slagle (james-slagle)]

CI is passing again after the symlink was restored to the previous repo