[library] Network configuration task reports success even when config is not applied

[Artem Panchenko]

Sometimes network configuration task reports success and deployment continues but fail later on another task, because no interfaces were actually configured:

root@node-5:~# grep -E 'MODULAR|err\):' /var/log/puppet.log
2015-09-09 11:16:17 +0000 Scope(Class[main]) (notice): MODULAR: hiera.pp
2015-09-09 11:16:21 +0000 Scope(Class[main]) (notice): MODULAR: globals.pp
2015-09-09 11:16:26 +0000 Scope(Class[main]) (notice): MODULAR: logging.pp
2015-09-09 11:16:30 +0000 Scope(Class[main]) (notice): MODULAR: tools.pp
2015-09-09 11:16:35 +0000 Scope(Class[main]) (notice): MODULAR: netconfig.pp
2015-09-09 11:16:40 +0000 Puppet (err): Could not apply complete catalog: Found 1 dependency cycle:
2015-09-09 11:16:42 +0000 Scope(Class[main]) (notice): MODULAR: connectivity_tests.pp
2015-09-09 11:16:46 +0000 Scope(Class[main]) (notice): MODULAR: firewall.pp
2015-09-09 11:16:51 +0000 Scope(Class[main]) (notice): MODULAR: ssl_add_trust_chain.pp
2015-09-09 11:16:55 +0000 Scope(Class[main]) (notice): MODULAR: hosts.pp
2015-09-09 11:17:00 +0000 Scope(Class[main]) (notice): MODULAR: zabbix.pp
2015-09-09 11:17:05 +0000 Scope(Class[main]) (notice): MODULAR: ceph-osd.pp
2015-09-09 11:20:14 +0000 Puppet (err): ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]
2015-09-09 11:20:14 +0000 /Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns (err): change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]
2015-09-09 11:20:17 +0000 Scope(Class[main]) (notice): MODULAR: ceph-osd.pp
2015-09-09 11:25:06 +0000 Puppet (err): ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]
2015-09-09 11:25:06 +0000 /Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns (err): change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]
2015-09-09 11:25:11 +0000 Scope(Class[main]) (notice): MODULAR: ceph-osd.pp
2015-09-09 11:27:47 +0000 Puppet (err): ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]
2015-09-09 11:27:47 +0000 /Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns (err): change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-9 returned 1 instead of one of [0]

root@node-5:~# ping -q -c 3 -W 1 node-9
PING node-9.mirantis.com (192.168.0.7) 56(84) bytes of data.

--- node-9.mirantis.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2014ms

root@node-5:~# ip r g 192.168.0.7
192.168.0.7 via 172.16.162.36 dev eth0 src 172.16.162.51
    cache

Such issues happen because we use old Puppet version on slaves which have the following bug: https://tickets.puppetlabs.com/browse/PUP-735

So if networking template has incorrect transformations order then 'dependency cycle' error occurs, but `puppet apply` still returns 0 exit code:

root@node-5:~# puppet apply -d --detailed-exitcodes /etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp
...
Error: Could not apply complete catalog: Found 1 dependency cycle:
(L23_stored_config[br-ceph] => L2_bridge[br-ceph] => L2_port[eth0.364] => L23network::L2::Port[eth0.364] => L23network::L2::Bridge[br-ex] => L2_bridge[br-ex] => L2_port[eth1] => L23network::L2::Port[eth1] => L23network::L2::Port[eth1.360] => L2_port[eth1.360] => L2_bond[lnxbond0] => L23network::L2::Bond[lnxbond0] => L23network::L2::Bridge[br-storage] => L2_bridge[br-storage] => L2_port[eth1.361] => L23network::L2::Port[eth1.361] => L23network::L2::Bridge[br-ceph] => L23_stored_config[br-ceph])
Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
...
root@node-5:~# echo $?
0
root@node-5:~# puppet -V
3.4.3

In order to fix this bug we need to upgrade Puppet on slaves to >=4.1.0 or >=3.8.3 version.

[Sergey Vasilenko (xenolog)]

please provide network template code
and full puppet logs

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/222927

[Artem Panchenko (apanchenko-8)]

@Sergey,

this bug is not about network configuration failure due to incorrect transformations order in networking template. It's about the `puppet apply` command behaviour - it returns 0 even when task failed due to 'dependency cycle' error. Environment deployment continues and fails later on another task, so for cloud administrator it's not obvious what went wrong.
I mentioned an example with networking template, because it's the easiest way for end users to cause such issue - just provide incorrect order of transformations.

[Stanislav Makar (smakar)]

@Sergey this is puppet bug https://tickets.puppetlabs.com/browse/PUP-735
@Artem we should just update puppet to 3.8.3 or backport the patch to puppet we are using now
This is not fuel-library responsibility

[Matthew Mosesohn (raytrac3r)]

This will be addressed in https://bugs.launchpad.net/bugs/1500414
Marking as duplicate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/222927
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.