[SRU] ERROR Nonce already used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
High
|
Julian Edwards | ||
1.5 |
Fix Released
|
High
|
Julian Edwards | ||
1.6 |
Fix Released
|
High
|
Julian Edwards | ||
juju-core |
Fix Released
|
High
|
Ian Booth | ||
juju (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned | ||
Saucy |
Won't Fix
|
Undecided
|
Unassigned | ||
Trusty |
Confirmed
|
Undecided
|
Unassigned | ||
maas (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Saucy |
Won't Fix
|
Medium
|
Unassigned | ||
Trusty |
Fix Released
|
High
|
Unassigned |
Bug Description
SRU for secondary fix (branch lp:~julian-edwards/maas/nonce-fix-bug-1190986)
[Impact]
* Minimize possibility of hitting a nonce collision by using a longer nonce (uuid4) in both python-maas-client and commissioning scripts.
[Testcase]
* Do some api calls and make sure that the piston_nonce postgres table is using the larger uuid nonce for the calls
[Regression Potential]
* Minimal; replace random string of length 8 with uuidgen.
The error below happens sometimes. (~5% of all requests fail)
System: Ubuntu 12.04.2
MAAS-version: 1.2+bzr1373+
Juju-version: 0.7+bzr628+
$ juju status
2013-06-14 13:59:36,213 INFO Connecting to environment...
2013-06-14 13:59:36,295 ERROR Cannot connect to environment: Nonce already used: 55941407
Traceback (most recent call last):
File "/usr/lib/
client = yield self._internal_
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
candidates = yield self._provider.
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
machine = yield provider.
File "/usr/lib/
result = result.
File "/usr/lib/
return g.throw(self.type, self.value, self.tb)
File "/usr/lib/
instances = yield self.maas_
File "/usr/lib/
current.result = callback(
File "/usr/lib/
raise ProviderError(
ProviderError: Nonce already used: 55941407
2013-06-14 13:59:36,297 ERROR Nonce already used: 55941407
I think the problem is caused by the MAAS-database table "piston_nonce"
In my system this table contains a hughe amount of tokens that are never deleted:
maasdb=# SELECT count(id) FROM piston_nonce;
count
---------
3619099
(1 row)
Related branches
- Gavin Panella (community): Approve
-
Diff: 142 lines (+88/-1)5 files modifiedetc/celeryconfig.py (+17/-1)
etc/celeryconfig_common.py (+1/-0)
services/region-worker/run (+1/-0)
src/maasserver/tasks.py (+30/-0)
src/maasserver/tests/test_tasks.py (+39/-0)
- Gavin Panella (community): Approve
-
Diff: 222 lines (+213/-0)2 files modifiedsrc/maasserver/nonces_cleanup.py (+94/-0)
src/maasserver/tests/test_nonces_cleanup.py (+119/-0)
- Julian Edwards (community): Approve
-
Diff: 42 lines (+5/-2)2 files modifiedetc/maas/templates/commissioning-user-data/snippets/maas_api_helper.py (+2/-1)
src/apiclient/maas_client.py (+3/-1)
- Julian Edwards (community): Approve
-
Diff: 42 lines (+5/-2)2 files modifiedetc/maas/templates/commissioning-user-data/snippets/maas_api_helper.py (+2/-1)
src/apiclient/maas_client.py (+3/-1)
- Julian Edwards (community): Approve
-
Diff: 42 lines (+5/-2)2 files modifiedetc/maas/templates/commissioning-user-data/snippets/maas_api_helper.py (+2/-1)
src/apiclient/maas_client.py (+3/-1)
- Julian Edwards (community): Approve
-
Diff: 42 lines (+5/-2)2 files modifiedetc/maas/templates/commissioning-user-data/snippets/maas_api_helper.py (+2/-1)
src/apiclient/maas_client.py (+3/-1)
Changed in maas: | |
status: | Incomplete → Triaged |
importance: | Undecided → Low |
tags: | added: theme-oil |
Changed in maas: | |
importance: | Low → High |
Changed in maas: | |
assignee: | nobody → Raphaël Badin (rvb) |
status: | Triaged → In Progress |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas (Ubuntu Saucy): | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in maas: | |
milestone: | none → 14.04 |
Changed in maas: | |
status: | Fix Committed → Fix Released |
Changed in juju-core: | |
milestone: | none → 1.21-alpha1 |
milestone: | 1.21-alpha1 → 1.20.6 |
importance: | Undecided → High |
status: | New → Triaged |
Changed in juju-core: | |
assignee: | nobody → Ian Booth (wallyworld) |
status: | Triaged → In Progress |
Changed in juju-core: | |
status: | In Progress → Fix Committed |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
description: | updated |
summary: |
- ERROR Nonce already used + [SRU] ERROR Nonce already used |
Changed in maas (Ubuntu Trusty): | |
importance: | Undecided → High |
milestone: | none → trusty-updates |
status: | New → Fix Committed |
Changed in maas (Ubuntu): | |
status: | Triaged → Fix Released |
Changed in maas (Ubuntu Saucy): | |
status: | Triaged → Won't Fix |
Can you attach all the maas logs under /var/log/maas please.
Thanks.