Deployment with networking template fails if 'keystone/api' network is moved to separate interface: "Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 600 seconds!"

[Artem Panchenko]

Fuel version info (7.0 build #273): http://paste.openstack.org/show/448322/

Environment deployment fails if 'keystone/api' network is moved to separate interface in networking templates:

node-1 2015-09-05T08:24:01.551154 err: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 600 seconds!
node-1 2015-09-05T08:24:01.551154 err: /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:97:in `fail'
node-1 2015-09-05T08:24:01.553736 err: /etc/puppet/modules/haproxy/lib/puppet/provider/haproxy_backend_status/haproxy.rb:70:in `ensure='
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:178:in `call_provider'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:503:in `set'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/property.rb:581:in `sync'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:191:in `sync'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:128:in `sync_if_needed'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:81:in `perform_changes'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:174:in `apply'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:187:in `eval_resource'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `call'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block (2 levels) in evaluate'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:327:in `block in thinmark'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:326:in `thinmark'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:117:in `block in evaluate'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `block in apply'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `block in apply_catalog'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block (2 levels) in run'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `block in run'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
node-1 2015-09-05T08:24:01.553736 err: /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'
node-1 2015-09-05T08:24:01.553736 err: /usr/bin/puppet:4:in `<main>'

HAProxy is not able to connect to Keystone API because it is running in net namespace and using IP network '240.0.0.2/30' instead of management to access the service:

https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/firewall.pp#L106

In common deployments 'keystone/api' net role is assigned to management network, so Keystone deployment works fine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220896

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220896
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a4b935453e2021d0fa38f1b10c80f7f1454dc736
Submitter: Jenkins
Branch: master

commit a4b935453e2021d0fa38f1b10c80f7f1454dc736
Author: Artem Panchenko <email address hidden>
Date: Mon Sep 7 10:35:52 2015 +0300

    Allow to connect to Keystone from HAProxy netns

    Change-Id: I23668831fa32d5088ac41be871fa83fc6010c39b
    Closes-bug: #1492933

[Tatyanka (tatyana-leontovich)]

fix was merged in master but not back-ported in stable/7-0 as result issue is reproduced on the latest iso

[Tatyanka (tatyana-leontovich)]

move to fix committed in 8.0 according to issue closed by this one review https://review.openstack.org/220896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/221167

[Tatyanka (tatyana-leontovich)]

for 7.0 reproduced here https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.network_templates/14/artifact/logs/fail_error_deploy_ceph_net_tmpl-fuel-snapshot-2015-09-08_03-14-02.tar.xz.filtered.log

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/221167
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=121016a09b0e889994118aa3ea42fa67eabb8f25
Submitter: Jenkins
Branch: stable/7.0

commit 121016a09b0e889994118aa3ea42fa67eabb8f25
Author: Artem Panchenko <email address hidden>
Date: Mon Sep 7 10:35:52 2015 +0300

    [stable/7.0]Allow to connect to Keystone from HAProxy netns

    Author: Artem Panchenko
    Change-Id: I23668831fa32d5088ac41be871fa83fc6010c39b
    Closes-bug: #1492933
    (cherry picked from commit a4b935453e2021d0fa38f1b10c80f7f1454dc736)

[Artem Panchenko (apanchenko-8)]

verified on iso #288

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "288"
  build_id: "288"
  nailgun_sha: "93477f9b42c5a5e0506248659f40bebc9ac23943"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "121016a09b0e889994118aa3ea42fa67eabb8f25"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"