[queens] fs01 noop job failed with Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)

[chandan kumar]

FS001 noop job on queens branch is failing at overcloud prepare image with following errors:
https://logs.rdoproject.org/68/599068/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/5f821b5/logs/undercloud/var/log/ironic-inspector/ironic-inspector.log.txt.gz#_2018-09-18_00_49_16_331

2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base [-] The PXE filter DnsmasqFilter, state=initialized encountered an exception: Unexpected error while running command.
Command: sudo ironic-inspector-rootwrap /etc/ironic-inspector/rootwrap.conf systemctl start openstack-ironic-inspector-dnsmasq.service
Exit code: 99
Stdout: u''
Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)\n'; resetting the filter: ProcessExecutionError: Unexpected error while running command.
Command: sudo ironic-inspector-rootwrap /etc/ironic-inspector/rootwrap.conf systemctl start openstack-ironic-inspector-dnsmasq.service
Exit code: 99
Stdout: u''
Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)\n'
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base Traceback (most recent call last):
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base File "/usr/lib/python2.7/site-packages/ironic_inspector/pxe_filter/base.py", line 139, in fsm_reset_on_error
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base yield self.fsm
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base File "/usr/lib/python2.7/site-packages/ironic_inspector/pxe_filter/base.py", line 81, in inner
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base return method(self, *args, **kwargs)
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base File "/usr/lib/python2.7/site-packages/ironic_inspector/pxe_filter/dnsmasq.py", line 141, in init_filter
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base _execute(CONF.dnsmasq_pxe_filter.dnsmasq_start_command)
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base File "/usr/lib/python2.7/site-packages/ironic_inspector/pxe_filter/dnsmasq.py", line 313, in _execute
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base check_exit_code=not ignore_errors)
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base cmd=sanitized_cmd)
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base ProcessExecutionError: Unexpected error while running command.
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base Command: sudo ironic-inspector-rootwrap /etc/ironic-inspector/rootwrap.conf systemctl start openstack-ironic-inspector-dnsmasq.service
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base Exit code: 99
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base Stdout: u''
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)\n'
2018-09-18 00:49:16.331 2545 ERROR ironic_inspector.pxe_filter.base

rpm version used:
http://instack-undercloud-8.4.5-0.20180917212744.40ede62.el7.noarch/ -> pointing to this review

https://review.openstack.org/#/c/598229/ -> Add start/stop command for ironic-inspector-dnsmasq

and this review https://review.openstack.org/#/c/598433/ in ironic-inspector

We are seeing continuous failure related to this in the same jobs
https://logs.rdoproject.org/67/599067/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/af4d3a9/logs/undercloud/var/log/ironic-inspector/ironic-inspector.log.txt.gz#_2018-09-18_00_41_40_615

https://logs.rdoproject.org/24/567224/108/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/eb39d51/logs/undercloud/var/log/ironic-inspector/ironic-inspector.log.txt.gz#_2018-09-18_00_59_53_040

[Martin Schuppert (mschuppert)]

I see the same in this queens backport

http://logs.openstack.org/45/594145/2/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5fc5fb9/logs/undercloud/var/log/ironic-inspector/ironic-inspector.log.txt.gz#_2018-09-18_19_33_43_756

[Martin Schuppert (mschuppert)]

The rootwrap filter was changed 2 days ago in https://review.openstack.org/#/c/598433/1/rootwrap.d/ironic-inspector.filters

This is missing from:
http://logs.openstack.org/45/594145/2/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5fc5fb9/logs/undercloud/etc/ironic-inspector/rootwrap.d/

http://logs.rdoproject.org/45/594145/2/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/585a096/logs/undercloud/etc/ironic-inspector/rootwrap.d/

[Quique Llorente (quiquell)]

Then we need a promotion to get the latest rpm

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Is this still an issue?

[Quique Llorente (quiquell)]

OVB passing at noop https://review.openstack.org/#/c/567224 so I close this