Neutron_router_interface[router04:net04__subnet]/ensure) change from absent to present failed

[Dennis Dmitriev]

Scenario:
            1. Create cluster
            2. Add 1 controller node
            3. Deploy the cluster
            4. Add 2 controller nodes
            5. Deploy changes

Deploy failed:
 - on step 5
- in the task "openstack-network/routers.pp"
- on the node with "role"=>"primary-controller"
- with the following message in puppet.log:

node-2 2015-10-31T04:24:03.319167 notice: (Scope(Class[main])) MODULAR: openstack-network/routers.pp
node-2 2015-10-31T04:24:03.485990 notice: Compiled catalog for node-2.test.domain.local in environment production in 0.20 seconds
node-2 2015-10-31T04:24:38.304984 err: did not get expected message on interface addition, got
node-2 2015-10-31T04:24:38.305450 err: /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
node-2 2015-10-31T04:24:38.306124 err: /etc/puppet/modules/neutron/lib/puppet/provider/neutron_router_interface/neutron.rb:76:in `create'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:197:in `call_valuemethod'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:498:in `set'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-2 2015-10-31T04:24:38.306124 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
node-2 2015-10-31T04:24:38.306901 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
node-2 2015-10-31T04:24:38.307747 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
node-2 2015-10-31T04:24:38.307747 err: /usr/bin/puppet:4:in `<main>'
node-2 2015-10-31T04:24:38.308250 err: (/Stage[main]/Main/Neutron_router_interface[router04:net04__subnet]/ensure) change from absent to present failed: did not get expected message on interface addition, got
node-2 2015-10-31T04:24:38.906555 notice: Finished catalog run in 31.37 seconds

ISO version:
 shotgun_sha 25dd78a3118267e3616df0727ce746e7dead2d67
 api 1.0
 fuelmain_sha b9b39ee9d3bbdb1883d924840ab691b95e1ef4d1
 astute_sha 687458e949dedfa6877e3a2e683f8ee0c1cbd43c
 fuel-ostf_sha 25c87380395f2e42ca01ea6357b72d4407ff4201
 build_id 97
 openstack_version 2015.1.0-8.0
 auth_required True
 fuel-library_sha 01272e0a719c940ef3df5b4167990ff811da1b10
 fuel-agent_sha 9da73b497be5f91cb79f91e74d73eb0525be1c71
 production docker
 network-checker_sha 518f7ef2706e564950469435f7cf27df175e1b27
 release_versions:
    2015.1.0-8.0:
       VERSION:
          build_id 97
          fuelmenu_sha 153c91e884a4601d30f8ebe3349170f96e3456e2
          openstack_version 2015.1.0-8.0
          network-checker_sha 518f7ef2706e564950469435f7cf27df175e1b27
          fuel-upgrade_sha 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
          fuel-library_sha 01272e0a719c940ef3df5b4167990ff811da1b10
          fuel-createmirror_sha 5b61266e19eb7cdc745fea7a7d97c90f2515c921
          feature_groups:
              - mirantis
          fuel-nailgun-agent_sha 00b4b11553c250f22c0079fb74c8b782dcb7b740
          fuel-nailgun_sha af02bd8f9aac1a1007f3ce17e5d7842853bd7758
          shotgun_sha 25dd78a3118267e3616df0727ce746e7dead2d67
          fuel-agent_sha 9da73b497be5f91cb79f91e74d73eb0525be1c71
          api 1.0
          fuelmain_sha b9b39ee9d3bbdb1883d924840ab691b95e1ef4d1
          astute_sha 687458e949dedfa6877e3a2e683f8ee0c1cbd43c
          fuel-ostf_sha 25c87380395f2e42ca01ea6357b72d4407ff4201
          release 8.0
          build_number 97
          python-fuelclient_sha 555eeac8810c36b0616f5d12277e9099a53bb338
          production docker
 fuel-upgrade_sha 1e894e26d4e1423a9b0d66abd6a79505f4175ff6
 fuel-createmirror_sha 5b61266e19eb7cdc745fea7a7d97c90f2515c921
 build_number 97
 fuelmenu_sha 153c91e884a4601d30f8ebe3349170f96e3456e2
 fuel-nailgun-agent_sha 00b4b11553c250f22c0079fb74c8b782dcb7b740
 fuel-nailgun_sha af02bd8f9aac1a1007f3ce17e5d7842853bd7758
 python-fuelclient_sha 555eeac8810c36b0616f5d12277e9099a53bb338
 feature_groups:
     - mirantis
 release 8.0

[Sergey Vasilenko (xenolog)]

root of case is prefetch of router ports resources.
http://paste.openstack.org/show/bGKN6tTRGlrSY4sFqIKf/

Fix to upstream puppet-neutron on review: https://review.openstack.org/#/c/239473/

[Sergey Vasilenko (xenolog)]

In the Fuel-library should be fixed after merge https://review.openstack.org/#/c/240594/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240594
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9fb81504f73066f9e6835467461b4b4c2b748f90
Submitter: Jenkins
Branch: master

commit 9fb81504f73066f9e6835467461b4b4c2b748f90
Author: Sergey Vasilenko <email address hidden>
Date: Fri Oct 30 21:04:04 2015 +0300

    Backport last version of Iba1105bc119272715fa211ac129a366fe50cbf47

    from upstream.

    Change-Id: Icdc214ce794e8d608ce51eab77f7fa5c4a1a4526
    Closes-bug: #1511755
    Closes-bug: #1512021
    Related-bug: #1508511

[Dmitry Belyaninov (dbelyaninov)]

Verified on ISO #231
Environment created by dos.py script.
The first controller was deployed. After that two additional controllers were deployed successfully.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "231"
  build_id: "231"
  fuel-nailgun_sha: "7186bebeddf21c7a8b667c1e000f922bfd6961ce"
  python-fuelclient_sha: "b77559b309a62650658424a43051cfeaba518748"
  fuel-agent_sha: "f0eddf7873a7a99d464d6dd6a1cfd19e31ea0525"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "b60624ee2c5f1d6d805619b6c27965a973508da1"
  fuel-library_sha: "b9f8c28f0e8f7d725ed4211df418fc54a1abd728"
  fuel-ostf_sha: "a98973482f839554d90cc1c071d625a01e018cfe"
  fuel-createmirror_sha: "f3c84d7338d5bf81b8564a5ffa67a3adebc226b4"
  fuelmenu_sha: "f7815c99abcaa7e8fa93f3966f38072d155cd39d"
  shotgun_sha: "25a0cc461a9fa4f7684f04cef0ff4ad9aa99a64d"
  network-checker_sha: "0b1b94a9685c6471d6911dff7ecac10b7bd2625f"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "2eca6adc33f02e02cd812e1d4be7c70e05fd07db"