[container build] SystemError: The following jobs were incomplete: state=finished

[Rafael Folco]

tripleo-build-containers-centos-7-buildah fails but process state = finished and no errors found in logs for that container build.

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_742/7
21577/1/gate/tripleo-build-containers-centos-7-buildah/7420f2f/logs/build.log

2020-05-18 11:54:07 | Building nova-serialproxy image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-builds/2a70e865-197a-4808-ba61-94e3efd8f47b/docker/nova/nova-serialproxy/nova-serialproxy-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-nova-serialproxy:8150aba8a14157223deb460c848c1a86f7c76e61_Exception occured while running the command
2020-05-18 11:54:07 | Traceback (most recent call last):
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
2020-05-18 11:54:07 | super(Command, self).run(parsed_args)
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2020-05-18 11:54:07 | return super(Command, self).run(parsed_args)
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/cliff/command.py", line 185, in run
2020-05-18 11:54:07 | return_code = self.take_action(parsed_args) or 0
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 233, in take_action
2020-05-18 11:54:07 | bb.build_all()
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 221, in build_all
2020-05-18 11:54:07 | self.build_all(deps.get(container))
2020-05-18 11:54:07 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 215, in build_all
2020-05-18 11:54:07 | not_done
2020-05-18 11:54:07 | SystemError: The following jobs were incomplete: set([<Future at 0x7f0430f4bf50 state=finished returned NoneType>])
2020-05-18 11:54:07 | The following jobs were incomplete: set([<Future at 0x7f0430f4bf50 state=finished returned NoneType>])
2020-05-18 11:54:07 | d9788c97 /tmp/kolla-builds/2a70e865-197a-4808-ba61-94e3efd8f47b/docker/nova/nova-serialproxy

No errors seen at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_742/721577/1/gate/tripleo-build-containers-centos-7-buildah/7420f2f/logs/buildah-builds/kolla-builds/2a70e865-197a-4808-ba61-94e3efd8f47b/docker/nova/nova-serialproxy/nova-serialproxy-build.log

[wes hayutin (weshayutin)]

https://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah&branch=stable%2Ftrain

1 failure w/ this.. let's wait for another at least.
This kind of error is best logged in the hackmd first until you see a repeat.

[Cédric Jeanneret (cjeanner)]

apparently there's a trend building here:
2020-05-19 08:47:50 | Building gnocchi-metricd image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-metricd/gnocchi-metricd-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-gnocchi-metricd:f21a5de89aa9230b80577f9e9654bfc2c4f45aee_33622c19 /tException occured while running the command
2020-05-19 08:47:50 | Traceback (most recent call last):
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
2020-05-19 08:47:50 | super(Command, self).run(parsed_args)
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2020-05-19 08:47:50 | return super(Command, self).run(parsed_args)
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/cliff/command.py", line 185, in run
2020-05-19 08:47:50 | return_code = self.take_action(parsed_args) or 0
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 233, in take_action
2020-05-19 08:47:50 | bb.build_all()
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 221, in build_all
2020-05-19 08:47:50 | self.build_all(deps.get(container))
2020-05-19 08:47:50 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 215, in build_all
2020-05-19 08:47:50 | not_done
2020-05-19 08:47:50 | SystemError: The following jobs were incomplete: set([<Future at 0x7fbb27359890 state=finished returned NoneType>, <Future at 0x7fbb27337550 state=finished raised SystemError>])
2020-05-19 08:47:50 | The following jobs were incomplete: set([<Future at 0x7fbb27359890 state=finished returned NoneType>, <Future at 0x7fbb27337550 state=finished raised SystemError>])
2020-05-19 08:47:50 | mp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-metricd
2020-05-19 08:47:50 | Building gnocchi-statsd image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-statsd/gnocchi-statsd-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-gnocchi-statsd:f21a5de89aa9230b80577f9e9654bfc2c4f45aee_33622c19 /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-statsd
2020-05-19 08:47:50 | Building gnocchi-api image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-api/gnocchi-api-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-gnocchi-api:f21a5de89aa9230b80577f9e9654bfc2c4f45aee_33622c19 /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-api
2020-05-19 08:47:50 | Building neutron-server-ovn image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6...

[Cédric Jeanneret (cjeanner)]

Fun fact: no error either in the container build log itself:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c53/710977/4/check/tripleo-build-containers-centos-7-buildah/c535cdf/logs/buildah-builds/kolla-builds/8b76ad8e-b2a6-4f07-8458-3af7c4cb66d6/docker/gnocchi/gnocchi-metricd/gnocchi-metricd-build.log

this sounds like some race condition. Really. Meh.

[Marios Andreou (marios-b)]

getting this quite a bit in the last few days... i have some noted examples copy/pasting here:

 https://review.opendev.org/#/c/735810/1
        * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e89/735810/1/check/tripleo-build-containers-centos-8/e89b6af/logs/build.log

        * 2020-06-17 07:28:04 | SystemError: The following jobs were incomplete: ['nova-libvirt', {'openstack-base': [{'manila-base': ['manila-api', 'manila-share', 'manila-scheduler']}, {'ceilometer-base': ['ceilometer-central', 'ceilometer-compute', 'ceilometer-notification', 'ceilometer-ipmi']}, {'gnocchi-base': ['gnocchi-metricd', 'gnocchi-statsd', 'gnocchi-api']}, {'zaqar-base': ['zaqar-wsgi']}, 'tempest', {'nova-base': ['nova-scheduler', 'nova-conductor', 'nova-novncproxy', 'nova-compute', 'nova-api', 'nova-compute-ironic']}, 'collectd', {'mistral-base': ['mistral-engine', 'mistral-api', 'mistral-event-engine', 'mistral-executor']}, {'novajoin-base': ['novajoin-server', 'novajoin-notifier']}, {'barbican-base': ['barbican-keystone-listener', 'barbican-worker', 'barbican-api']}, {'placement-base': ['placement-api']}, 'horizon', {'aodh-base': ['aodh-api', 'aodh-notifier', 'aodh-listener', 'aodh-evaluator']}, {'octavia-base': ['octavia-worker', 'octavia-health-manager', 'octavia-api', 'octavia-housekeeping']}, 'ironic-inspector', {'keystone-base': ['keystone']}, {'heat-base': ['heat-engine', 'heat-api-cfn', 'heat-all', 'heat-api']}, {'glance-base': ['glance-api']}, {'swift-base': ['swift-proxy-server', 'swift-container', 'swift-object', 'swift-account']}, {'cinder-base': ['cinder-backup', 'cinder-scheduler', 'cinder-api', 'cinder-volume']}, {'neutron-base': ['ironic-neutron-agent', 'neutron-metadata-agent-ovn', 'neutron-openvswitch-agent', 'neutron-mlnx-agent', 'neutron-dhcp-agent', 'neutron-sriov-agent', 'neutron-metadata-agent', 'neutron-l3-agent', {'neutron-server': []}]}, {'ironic-base': ['ironic-api', 'ironic-conductor', 'ironic-pxe']}, {'designate-base': ['designate-producer', 'designate-central', 'designate-sink', 'designate-mdns', 'designate-worker', 'designate-backend-bind9', 'designate-api']}, {'sahara-base': ['sahara-api', 'sahara-engine']}]}]

 https://review.opendev.org/#/c/735810/ https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_62a/735810/1/check/tripleo-build-containers-centos-8-ussuri/62a5eb2/logs/build.log
        * 2020-06-16 13:40:16 | SystemError: The following jobs were incomplete: ['nova-libvirt', 'tripleoclient', {'openstack-base': [{'ceilometer-base': ['ceilometer-compute', 'ceilometer-central', 'ceilometer-notification', 'ceilometer-ipmi']}, {'nova-base': ['nova-scheduler', 'nova-compute', 'nova-novncproxy', 'nova-conductor', 'nova-api', 'nova-compute-ironic']}, {'sahara-base': ['sahara-engine', 'sahara-api']}, {'octavia-base': ['octavia-worker', 'octavia-api', 'octavia-health-manager', 'octavia-housekeeping']}, {'designate-base': ['designate-sink', 'designate-producer', 'designate-backend-bind9', 'designate-mdns', 'designate-central', 'designate-worker', 'designate-api']}, {'neutron-base': ['neutron-sriov-agent', 'neutron-l3-agent', 'neutron-dhcp-agent', 'n...

[Marios Andreou (marios-b)]

closing this out as stale/no update in a long time. please re-open if you disagree

[Marios Andreou (marios-b)]

well, it is already marked incomplete, so just removing the milestone

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]