IBP: Ubuntu provision has been failed due to parallel generating two images for different envs

[Anastasia Palkina]

"build_id": "2015-03-24_22-54-44",
"ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867",
"build_number": "226",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-24_22-54-44", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "226", "api": "1.0", "nailgun_sha": "3b9f2c83c63653113701971492a6d8039db2cce2", "production": "docker", "python-fuelclient_sha": "3624051242c83fdbdd1df9a0e466797c06b75043", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "acd7659a727ab890076de41272c2054744be4d20"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "3b9f2c83c63653113701971492a6d8039db2cce2",
"production": "docker",
"python-fuelclient_sha": "3624051242c83fdbdd1df9a0e466797c06b75043",
"astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10",
"fuellib_sha": "acd7659a727ab890076de41272c2054744be4d20"

I tested on this ISO 2 environment today and all was successful.

But after it I rebooted my computer and installed master ISO #216 again.

Created 2 environments. Start deployments. Both deployments failed before provisioning with next errors:

Error in Web Backend log:

2015-03-25 15:53:10 ERROR

[7f1c6fb0e740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'

2015-03-25 15:53:09 ERROR

[7f1c6fb0e740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'
[pid: 1078|app: 0|req: 282/573] 10.20.0.1 () {44 vars in 901 bytes} [Wed Mar 25 15:53:09 2015] GET /api/nodes?cluster_id=1&_=1427297817026 => generated 13410 bytes in 477 msecs (HTTP/1.1 200) 4 headers in 185 bytes (2 switches on core 0)

Error in Astute log:
http://paste.openstack.org/show/196656/

[Kyrylo Romanenko (kromanenko)]

I have failure of deployment with this error too. But after these errors i got one more:

[610] Error running RPC method granular_deploy: Failed to execute hook .
---
priority: 1800
type: shell
uids:
- '1'
- '3'
- '2'
parameters:
  retries: 3
  cmd: apt-get update
  cwd: "/"
  timeout: 180
  interval: 1
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:54:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine/granular_deployment.rb:201:in `pre_deployment_actions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:32:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:133:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

[Kyrylo Romanenko (kromanenko)]

{"build_id": "2015-03-25_09-13-15", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "227", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-25_09-13-15", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "227", "api": "1.0", "nailgun_sha": "aaeee045ea755c9a81f48de2019fce3e368afa60", "production": "docker", "python-fuelclient_sha": "3624051242c83fdbdd1df9a0e466797c06b75043", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "96669c3b59b2286c4fe5a06940ef1c004f6aaeae"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "aaeee045ea755c9a81f48de2019fce3e368afa60", "production": "docker", "python-fuelclient_sha": "3624051242c83fdbdd1df9a0e466797c06b75043", "astute_sha": "631f96d5a09cc48bfbddcbf056b946c8a80438f0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "96669c3b59b2286c4fe5a06940ef1c004f6aaeae"}

Cluster of 3 nodes:
1 Controller, 1 Compute, 1 Cinder
Ubuntu, Multi-node with HA, qemu, Neutron with VLAN.

[Ihor Kalnytskyi (ikalnytskyi)]

@Nastia's case:

IBP script for Ubuntu has one limitation now - it couldn't generate provisioning images in parallel for different envs. Basically it means, you can't lunch provisioning of two or more Ubuntu envs in parallel. This is a known issue, and scheduled to be in SCF. Assign this to Alex Gordeev who is responsible for the script.

[Alexander Gordeev (a-gordeev)]

Thanks, I'm already working on that.

[Ihor Kalnytskyi (ikalnytskyi)]

@Kyrylo, this is a different issue. You have a wrong setup. In /etc/fuel/astute.yaml I see that you have MASTER IP in dhcp_gateway.

    dhcp_gateway: 10.109.24.2

Looks like you have to setup it properly and change it to your host's IP address (which is probable 10.109.24.1).

[Kyrylo Romanenko (kromanenko)]

One more occurrence of this bug

2015-03-27 13:40:22 ERROR
[7fcd7f677740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'
2015-03-27 13:40:22 INFO
[7fcd7f677740] (provisioning_serializers) Node Untitled (63:c3) (id=3, mac=64:bf:6e:e0:63:c3) seems booted with bootstrap image
2015-03-27 13:40:22 INFO
[7fcd7f677740] (provisioning_serializers) Node Untitled (22:df) (id=2, mac=64:09:d4:75:22:df) seems booted with bootstrap image
2015-03-27 13:40:22 INFO
[7fcd7f677740] (provisioning_serializers) Node Untitled (63:1c) (id=1, mac=64:2a:fd:8e:63:1c) seems booted with bootstrap image
2015-03-27 13:40:22 ERROR
[7fcd7f677740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'
2015-03-27 13:40:21 ERROR
[7fcd7f677740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'
2015-03-27 13:40:21 INFO
[7fcd7f677740] (checker) Untagged networks found, checking intersection between them...
2015-03-27 13:40:21 ERROR
[7fcd7f677740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'
2015-03-27 13:40:21 INFO
[7fcd7f677740] (manager) ApplyChangesTask: execute async starting for task 8

MOS 6.1 build:
{"build_id": "2015-03-26_22-54-44", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "234", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-26_22-54-44", "ostf_sha": "a4cf5f218c6aea98105b10c97a4aed8115c15867", "build_number": "234", "api": "1.0", "nailgun_sha": "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "3f1ece0318e5e93eaf48802fefabf512ca1dce40", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf", "production": "docker", "python-fuelclient_sha": "e5e8389d8d481561a4d7107a99daae07c6ec5177", "astute_sha": "3f1ece0318e5e93eaf48802fefabf512ca1dce40", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "320b5f46fc1b2798f9e86ed7df51d3bda1686c10", "fuellib_sha": "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"}

[Kyrylo Romanenko (kromanenko)]

 (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'

This error occurs also with provision of CentOS.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "248"
  build_id: "2015-03-30_03-08-59"
  nailgun_sha: "a3c259a4875787274fa01f0eba6514cc01b34308"
  python-fuelclient_sha: "05ec53f94206decdce19bb9373523022e5616b83"
  astute_sha: "f595715750a2c4820722a96e0236f5c89ca6521c"
  fuellib_sha: "3c85c9f16541c6ef461eb93816db51f798aba90c"
  ostf_sha: "e59c905566ed701117d7c643b435b13e6b5f8c3b"
  fuelmain_sha: "320b5f46fc1b2798f9e86ed7df51d3bda1686c10"

[Dmitry Pyzhov (dpyzhov)]

Fix is here: https://review.openstack.org/#/c/168090/

[Anastasia Palkina (apalkina)]

Verified on ISO #314

"build_id": "2015-04-15_23-18-33", "ostf_sha": "90f232a25d10f98db9d5531a3b133319529f57aa", "build_number": "314", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-15_23-18-33", "ostf_sha": "90f232a25d10f98db9d5531a3b133319529f57aa", "build_number": "314", "api": "1.0", "nailgun_sha": "24b56283fb4caaba6b6414056dad30e503a7c30f", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "ff186de358cbda489a320723de47a9e61ae23364", "astute_sha": "36a3d54baf86480326866279778b867086aa4b05", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "d3aa72d814c3f18799b566363004a22591161cef", "fuellib_sha": "ab6dbba825a5e54f3183ed8582921a0328b211c7"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "24b56283fb4caaba6b6414056dad30e503a7c30f", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "ff186de358cbda489a320723de47a9e61ae23364", "astute_sha": "36a3d54baf86480326866279778b867086aa4b05", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "d3aa72d814c3f18799b566363004a22591161cef", "fuellib_sha": "ab6dbba825a5e54f3183ed8582921a0328b211c7"