DIB fails to build images, UnboundLocalError: local variable 'call_order' referenced before assignment

[wes hayutin]

https://logs.rdoproject.org/openstack-periodic-4hr/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike/5a7120f/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2017-09-21_16_45_14

2017-09-21 16:45:09 | Calculating image size (this may take a minute)...
2017-09-21 16:45:12 | 2017-09-21 16:45:12.871 INFO diskimage_builder.block_device.blockdevice [-] Wrote final block device config to [/tmp/dib_build.YHVbNXat/states/block-device/config.json]
2017-09-21 16:45:14 | 2017-09-21 16:45:14.267 INFO diskimage_builder.block_device.blockdevice [-] create() called
2017-09-21 16:45:14 | 2017-09-21 16:45:14.268 ERROR diskimage_builder.block_device.blockdevice [-] Create failed; rollback initiated
2017-09-21 16:45:14 | Traceback (most recent call last):
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py", line 383, in cmd_create
2017-09-21 16:45:14 | dg, call_order = create_graph(self.config, self.params, state)
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/config.py", line 193, in create_graph
2017-09-21 16:45:14 | if node.name in dg.nodes:
2017-09-21 16:45:14 | TypeError: argument of type 'instancemethod' is not iterable
2017-09-21 16:45:14 | Traceback (most recent call last):
2017-09-21 16:45:14 | File "/usr/bin/dib-block-device", line 10, in <module>
2017-09-21 16:45:14 | sys.exit(main())
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 120, in main
2017-09-21 16:45:14 | return bdc.main()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 115, in main
2017-09-21 16:45:14 | self.args.func()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/cmd.py", line 36, in cmd_create
2017-09-21 16:45:14 | self.bd.cmd_create()
2017-09-21 16:45:14 | File "/usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py", line 388, in cmd_create
2017-09-21 16:45:14 | reverse_order = reversed(call_order)
2017-09-21 16:45:14 | UnboundLocalError: local variable 'call_order' referenced before assignment
2017-09-21 16:45:15 | 2017-09-21 16:45:15.681 INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
2017-09-21 16:45:15 | *** /tmp/dib_build.YHVbNXat/mnt is not a directory

Probably caused by:

https://github.com/openstack/diskimage-builder/commit/a47ff0dd4abbceae786d9a5a4d1ce8042a9eb4f7

[Ben Nemec (bnemec)]

This isn't just a promotion blocker, it's also going to break all jobs that build images. See http://logs.openstack.org/88/504588/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/8d35ffb/logs/undercloud/home/jenkins/overcloud_image_build.log.txt.gz

Although it looks like we have a serious mirror problem that's breaking things before we even get to that point now. Yay?

[Ben Nemec (bnemec)]

Proposed a revert at https://review.openstack.org/#/c/506376, although if that can't pass the gate because of networkx 2.0 then we're still out of luck. Worth a shot though.

[Ian Wienand (iwienand)]

Sorry, it looks like we're working in parallel a bit here.

There has been some confusion because we have put in a global pin of networkx<2.0, but *also* merged our 2.0 fix

I have put in https://review.openstack.org/506398 which reverts the change AND takes the networkx pin

In the changelog there I have outlined the plan. We have other changes in flight to make our testing correctly use constraints support. This way, we can restore the 2.0 support with a depends-on that bumps the requirements and test it properly. when other projects have updated in a similar way, we can do a global networkx bump in a controlled fashion

[Ian Wienand (iwienand)]

This should be fixed, and 2.9.0 is released with the pin.

constraints support is in [1], and a corresponding project-config change for the functional tests is in [2]

we can roll this out in a more controlled manner with [3] after those are in

[1] https://review.openstack.org/412747
[2] https://review.openstack.org/506388
[3] https://review.openstack.org/506524

[wes hayutin (weshayutin)]

looks resolved to me, thanks Ian!