Deployment failed with network template on netconfig.pp task: 'Could not apply complete catalog: Found 1 dependency cycle'

[Sergey Galkin]

Steps to reproduce
1. Install Fuel
2. Login to Fuel UI and create env with 5 nodes on VLAN with Ceph (env 2 in my case)
3. ssh to Fuel
4. upload to Fuel node attached env-14-template.yaml as /tmp/2/network_template_2.yaml
5. Execute fuel network-template --env 2 -u --dir /tmp/2/ (And get JSONDecodeError from https://bugs.launchpad.net/fuel/+bug/1490540)
6. Deploy cluster

Deploy failed with error
Deployment has failed. Method granular_deploy. Deployment failed on nodes 6.
Inspect Astute logs for the details

Please check the template. All nodes in cluster should have
- eth0 - Public
- eth2 - Admin (PXE), Storage (VLAN 141), Managment (VLAN 120)
- eth3 - Private (VLAN 441-459)

api: '1.0'
astute_sha: 53c86cba593ddbac776ce5a3360240274c20738c
auth_required: true
build_id: 2015-08-23_15-01-12
build_number: '219'
feature_groups:
- mirantis
fuel-agent_sha: 4c2ab9d6c623d345086c6e2874d1df81fd96a942
fuel-library_sha: 3a3ea6d9849bc1ba35c1bd882f0a0678b20d2e51
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: 16839cbf471b7142b04c0d2c2d94786bc486fefe
fuelmain_sha: a494e6628319abfef57e1754f6453cf8f1a4bc65
nailgun_sha: 7790ce872512ecdf21689e6a5f970dd7119febdb
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: fc7b63aa6900fe3b2c183108ba6a13e868bc0472
release: '7.0'

[Artem Panchenko (apanchenko-8)]

Deployment failed on node-6:

2015-08-31T15:40:38 err: [585] Task '{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/pupp
et/modules/osnailyfacter/modular/cluster/cluster.pp", "timeout"=>3600, "cwd"=>"/"}}' failed on node 6
2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"242311c4-06df-41cd-803c-3baf8f56e73d", "nodes"=>[{"uid"=>"6", "st
atus"=>"error", "error_type"=>"deploy", "role"=>"primary-controller", "task"=>{"priority"=>1100, "type"=>"puppet", "uids"=>["6"], "parameters"=>{"puppet_modules"=>"/etc/puppe
t/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/cluster/cluster.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}
2015-08-31T15:40:38 info: [585] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"242311c4-06df-41cd-803c-3baf8f56e73d", "status"=>"error", "error"

Here are errors from puppet logs:

node-6 2015-08-31T14:50:31.437391 notice: (Scope(Class[main])) MODULAR: netconfig.pp
node-6 2015-08-31T14:50:33.002479 notice: Compiled catalog for node-6.domain.tld in environment production in 1.59 seconds
node-6 2015-08-31T14:50:36.420798 err: Could not apply complete catalog: Found 1 dependency cycle:
node-6 2015-08-31T14:50:36.420798 err: (L23_stored_config[br-fw-admin] => L2_bridge[br-fw-admin] => L2_port[eth2] => L23network::L2::Port[eth2] => L23network::L2::Port[eth2.1
41] => L23_stored_config[eth2.141] => L23network::L2::Port[eth2.141] => L23network::L2::Bridge[br-fw-admin] => L23_stored_config[br-fw-admin])
node-6 2015-08-31T14:50:36.420798 err: Try the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/graph/simple_graph.rb:242:in `report_cycles_in_graph'
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:106:in `traverse'
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-6 2015-08-31T14:50:36.420798 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
node-6 2015-08-31T14:50:36.420948 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
node-6 2015-08-31T14:50:36.421951 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
node-6 2015-08-31T...

[Artem Panchenko (apanchenko-8)]

Sergey reproduced it on iso #257, but looks like it doesn't contain the fix https://github.com/stackforge/fuel-library/commit/47819aa7b071ee022e4f8e7747febdbffa58d769
So closing this bug back as duplicate.

[Sergey Galkin (sgalkin)]

Reproduced on build 260 (with https://review.openstack.org/#/c/218557)

[root@fuel ~]# grep by_network_scheme /etc/puppet/2015.1.0-7.0/modules/l23network/manifests/l3/route.pp
    $by_network_scheme = false,
    if ! $by_network_scheme {

[root@fuel ~]# fuel --fuel-version
api: '1.0'
astute_sha: 53c86cba593ddbac776ce5a3360240274c20738c
auth_required: true
build_id: '260'
build_number: '260'
feature_groups:
- mirantis
fuel-agent_sha: 082a47bf014002e515001be05f99040437281a2d
fuel-library_sha: e055af9dee6fbaf84cb8e86a9ebff9b25b07792e
fuel-nailgun-agent_sha: d7027952870a35db8dc52f185bb1158cdd3d1ebd
fuel-ostf_sha: 582a81ccaa1e439a3aec4b8b8f6994735de840f4
fuelmain_sha: 994bb9a8a2a3c4ab1f16e57b479d052abe45f921
nailgun_sha: 3de0f32fe9e09fbeee8a695d738f31fe2347e55a
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 9643fa07f1290071511066804f962f62fe27b512
release: '7.0'

In logs -
2015-09-01 13:02:19 ERR Could not apply complete catalog: Found 1 dependency cycle:

[Sergey Vasilenko (xenolog)]

root cause of this issue is bridges, that not placed before ports.

[Sergey Vasilenko (xenolog)]

transformation from this deployment here:
http://pastie.org/private/0vdlynhb2ekfbfsogqneg

In the lines 32-44 we can see, that subinterface became earlier, than interface.
IT is wrong way.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/219710

[Andrey Maximov (maximov)]

Late for 7.0, moving to 8.0. Also we can consider delivering bugfix of this problem via updates.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Stanislav Makar (<email address hidden>) on branch: master
Review: https://review.openstack.org/219710
Reason: not needed anymore

[Stanislav Makar (smakar)]

Sergey Vasilenko Sep 3 8:41 PM

Order-checking should be processed at template loading procedure.

[Ryan Moe (rmoe)]

Instead of checking transformation order when the template is loaded we could just sort the transformations correctly at serialization time. The available actions in the transformations have clear dependencies so they could just be topologically sorted and passed to l23network that way. Here's the general idea: http://paste.openstack.org/show/475670/

[Aleksey Kasatkin (alekseyk-ru)]

Priorities for templates were introduced within https://bugs.launchpad.net/fuel/+bug/1540374 . So, user now has a mechanism to set order of transformations. We need to decide how to combine auto-sorting with priorities set by user. Also, bug priority probably can be lowered.

[slava valyavskiy (slava-val-al)]

@Sergei, your template has wrong order of network schemes for roles:
http://pastebin.com/nv5wCJYG
Physical interface should be connected to the bridge before the its subinterfaces, so, first network scheme should be 'common'.

[Aleksey Kasatkin (alekseyk-ru)]

AFAIC, we have at least two options:

1. Keep priorities introduced here https://bugs.launchpad.net/fuel/+bug/1540374 and add validation of resulting sequences of transformations.

2. Implement auto-sorting of transformations https://bugs.launchpad.net/fuel/+bug/1490619/comments/13 and disregard prioritization.

Anyway, we have prioritization now and it helps with this issue (validation is required though). So, bug's priority can be lowered.