'ovs_neutron_agent' get errors calling neutron-rootwrap : Stderr: '... sudo: 3 incorrect password attempts'

[Dennis Dmitriev]

Starting 'p_neutron-plugin-openvswitch-agent' failed on primary controller (node-4) of the second cluster.

 Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
     p_neutron-plugin-openvswitch-agent (ocf::mirantis:neutron-agent-ovs): Started node-4 (unmanaged) FAILED
     Started: [ node-7 node-9 ]

/var/log/neutron/openvswitch-agent.log on node-4: http://paste.openstack.org/show/153917/

These errors started after neutron components were installed. dpkg.log:
...
2014-12-22 02:11:30 status installed neutron-plugin-openvswitch-agent 1:2014.1.3-fuel5.1.1~mira1
...

{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

[Dennis Dmitriev (ddmitriev)]

CI job: http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.ubuntu.ceph_multinode_compact.os_patching/11/

[Eugene Nikanorov (enikanorov)]

Could you please attach neutron.conf file from node-4?

[Miroslav Anashkin (manashkin)]

Please attach /etc/neutron/* from the failed node.
Additionally, please attach /var/log/puppet-*.log from the same failed node.

[Dennis Dmitriev (ddmitriev)]

Eugene Nikanorov: it is in the diagnostic snapshot attached in the #1: fuel-snapshot-2014-12-22_03-10-07/node-4.test.domain.local/etc/neutron/neutron.conf

[Eugene Nikanorov (enikanorov)]

Looking at ovs agent logs at node-4, I see that it got incorrect rotwrapper configuration ("root_helper = sudo") while neutron.conf on node-4 has correct conf.

That means that probably neutron.conf was not passed as a parameter at ovs agent startup

[Eugene Nikanorov (enikanorov)]

Apparently, correct neutron.conf is passed to ovs agent, but still rootwrap configuration is incorrect.

[Eugene Nikanorov (enikanorov)]

Update: looks like I've been mistaken. ovs agent has proper rootwrapper configuration.
We need /etc/sudoers.d/neutron file from node-4 to analyze.

[Alexander Ignatov (aignatov)]

Since the are traces in neutron in 5.1.1 code and there are no places to fix this issue in 6.0 neutron code I'm moving this bug to the 5.1.2 milestone. Also it should be mentioned that there are too many errors in puppet during provisioning heat and swift also. Also I'm leaving this bug in Incomplete state since we need more debug information.

[Bogdan Dobrelya (bogdando)]

Please elaborate all the additional debug information required to make thus issue confirmed

[Vladimir Kuklin (vkuklin)]

 This bug was incomplete for more than 4 weeks. We cannot investigate it further so we are setting the status to Invalid. If you think it is not correct, please feel free to provide requested information and reopen the bug, and we will look into it further.