parallel juju deployments race on the same maas
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Jeroen T. Vermeulen | ||
1.5 |
Fix Released
|
Critical
|
Jeroen T. Vermeulen | ||
juju-core |
Invalid
|
High
|
Unassigned | ||
maas (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Trusty |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Test Case]
1. Install MAAS
2. Add multiple nodes
3. deploy parallel nodes.
4. Juju in one of the parallel jobs will fail to obtain a node, even though it seems it has been allocated.
With the fix:
4. With the fix, Juju won't ever get the same node in the parallel environments.
Two jenkins-slaves running juju-deployer bundles attempt to create openstack deploys against a MAAS environment using the same pool of hardware. One environment (slave4) has a pending machine with no DNS name, log file says no instance information available. Examining maas, I can see it was allocated to a different user and env.
jenkins@
1.18.1-
jenkins@
Ubuntu 12.04.4 LTS \n \l
Here's the debug-log tail:
$ juju debug-log -e oil-slave-4
Warning: Permanently added 'suangi.oil' (ECDSA) to the list of known hosts.
machine-6: 2014-04-29 21:03:46 INFO juju runner.go:262 worker: start "machineenviron
machine-6: 2014-04-29 21:03:46 INFO juju runner.go:262 worker: start "rsyslog"
machine-6: 2014-04-29 21:03:46 DEBUG juju.worker.logger logger.go:60 logger setup
machine-6: 2014-04-29 21:03:46 DEBUG juju.worker.
machine-6: 2014-04-29 21:03:46 DEBUG juju.worker.rsyslog worker.go:76 starting rsyslog worker mode 1 for "machine-6" ""
machine-6: 2014-04-29 21:03:46 INFO juju runner.go:262 worker: start "authentication
machine-6: 2014-04-29 21:03:46 INFO juju.worker.
machine-6: 2014-04-29 21:03:46 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>
machine-0: 2014-04-29 21:18:02 WARNING juju.worker.
machine-0: 2014-04-29 21:18:14 WARNING juju.worker.
I'll attach the juju status from the two environments.
Related branches
- Raphaël Badin (community): Approve
-
Diff: 47 lines (+10/-2)2 files modifiedsrc/maas/development.py (+5/-1)
src/maas/settings.py (+5/-1)
- Jeroen T. Vermeulen (community): Approve
-
Diff: 47 lines (+10/-2)2 files modifiedsrc/maas/development.py (+5/-1)
src/maas/settings.py (+5/-1)
- MAAS Maintainers: Pending requested
-
Diff: 1315 lines (+716/-221) (has conflicts)20 files modifieddocs/conf.py (+53/-16)
docs/hardware-enablement-kernels.rst (+96/-0)
docs/installing-ubuntu.rst (+88/-0)
etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect.py (+8/-6)
src/maascli/api.py (+1/-0)
src/maasserver/api.py (+3/-3)
src/maasserver/migrations/0072_remove_ipmi_autodetect.py (+9/-0)
src/maasserver/models/node.py (+2/-42)
src/maasserver/models/tests/test_node.py (+2/-117)
src/maasserver/support/pertenant/tests/test_migration.py (+5/-9)
src/maasserver/testing/factory.py (+2/-2)
src/maasserver/tests/test_api_node.py (+4/-2)
src/maasserver/views/nodes.py (+5/-0)
src/maasserver/views/tests/test_clusters.py (+5/-5)
src/maasserver/views/tests/test_general.py (+2/-1)
src/maasserver/views/tests/test_nodes.py (+13/-4)
src/provisioningserver/custom_hardware/seamicro.py (+9/-0)
src/provisioningserver/import_images/boot_resources.py (+138/-0)
src/provisioningserver/import_images/tests/test_boot_resources.py (+256/-14)
src/provisioningserver/tests/test_tasks.py (+15/-0)
Changed in juju-core: | |
status: | New → Triaged |
importance: | Undecided → High |
tags: | added: maas-provider |
Changed in maas: | |
assignee: | nobody → Jeroen T. Vermeulen (jtv) |
status: | Triaged → In Progress |
Changed in maas: | |
status: | In Progress → Fix Committed |
description: | updated |
Changed in maas: | |
status: | Fix Committed → Fix Released |
Changed in maas (Ubuntu): | |
status: | New → Fix Released |
tags: |
added: verification-done removed: verification-needed |
Changed in juju-core: | |
status: | Triaged → Invalid |
To summarise the conversation I just had on IRC:
* juju is using separate maas users for each maas environment c4cd-11e3- 824b-00163efc50 68" is also appearing in oil4's status output as pending
* the oil2 machine "node-9df8a42a-
The question is, how can the same node seemingly be appearing on both juju environments? MAAS thinks it's allocated to oil2 (I think?)