FS01 master overcloud image prepare time out due to 'module' object has no attribute 'abc' in dhcp agent

[chandan kumar]

Master fs01 periodic job got timed out at Prepare the overcloud images for deploy.

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/ad90818/job-output.txt

2019-12-08 14:21:10.294879 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] *********
2019-12-08 14:21:10.321794 | primary | Sunday 08 December 2019 14:21:10 +0000 (0:00:02.509) 0:00:20.351 *******
2019-12-08 17:23:25.814969 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2019-12-08 17:23:25.815282 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]

While checking on neutron dhcp agent log, we found this:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/ad90818/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz

2019-12-08 17:29:19.904 5816 DEBUG oslo_concurrency.lockutils [-] Lock "privileged-ip-lib" released by "neutron.privileged.agent.linux.ip_lib._sync" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:339
2019-12-08 17:29:19.909 5816 CRITICAL neutron [-] Unhandled error: AttributeError: 'module' object has no attribute 'abc'
2019-12-08 17:29:19.909 5816 ERROR neutron Traceback (most recent call last):
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/bin/neutron-dhcp-agent", line 10, in <module>
2019-12-08 17:29:19.909 5816 ERROR neutron sys.exit(main())
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/agents/dhcp.py", line 17, in main
2019-12-08 17:29:19.909 5816 ERROR neutron dhcp_agent.main()
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp_agent.py", line 49, in main
2019-12-08 17:29:19.909 5816 ERROR neutron manager='neutron.agent.dhcp.agent.DhcpAgentWithStateReport')
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 414, in create
2019-12-08 17:29:19.909 5816 ERROR neutron periodic_fuzzy_delay=periodic_fuzzy_delay)
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 345, in __init__
2019-12-08 17:29:19.909 5816 ERROR neutron manager_class = importutils.import_class(self.manager_class_name)
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 30, in import_class
2019-12-08 17:29:19.909 5816 ERROR neutron __import__(mod_str)
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 39, in <module>
2019-12-08 17:29:19.909 5816 ERROR neutron from neutron.agent.linux import dhcp
2019-12-08 17:29:19.909 5816 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 68, in <module>
2019-12-08 17:29:19.909 5816 ERROR neutron class DictModel(collections.abc.MutableMapping):
2019-12-08 17:29:19.909 5816 ERROR neutron AttributeError: 'module' object has no attribute 'abc'
2019-12-08 17:29:19.909 5816 ERROR neutron

On further investigation, we found that neutron and networking-generic-switch has dropped support
of python2 with [1] and [2],
[1] https://review.opendev.org/#/c/691874/
[2] https://review.opendev.org/#/c/697119/

Pinning both of them might fix the issue.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/#/c/24022/ pins both neutron and networking-generic-switch might fix the issue.

[chandan kumar (chkumar246)]

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/4b59aaa/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/aaf2cdf/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/103cea9/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz

[chandan kumar (chkumar246)]

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/939a78e/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz

[Marios Andreou (marios-b)]

that pin merged 20 mins ago so we need logs newer than those in comment #3 to confirm it.

we (talking with chkumar now in irc) will post a test in a bit and add here

[chandan kumar (chkumar246)]

Some more fixes
* Dummy commit to build package in neutron -> https://review.rdoproject.org/r/#/c/24028/
* Skip test_cmd_get_version_of_installed_package_mapped in octavia -> https://review.rdoproject.org/r/#/c/24001/

Both the patches are merged, waiting for rdo current to become consistent and next promotion.

[Marios Andreou (marios-b)]

chkumar got some green runs there https://review.rdoproject.org/r/#/c/24050/ (and e.g. no more AttributeError in http://logs.rdoproject.org/50/24050/1/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/98da2f8/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz )

BUT

don't think we can close this? I guess we keep this until we unpin? Or will we file a new bug for that?

[wes hayutin (weshayutin)]

fixed
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train%09