Authorization Error: 'Nonce already used' error when deploying machines

Bug #1851708 reported by Marcelo Subtil Marcal on 2019-11-07
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Undecided
Unassigned

Bug Description

When deploying 24 machines using 2.6.1, nine (9) of them failed when trying to get /MAAS/metadata/curtin/2012-03-01/user-data.

The network analysis showed an "Authorization Error: 'Nonce already used."

https://pastebin.canonical.com/p/FkXvvvK7Ch/

The error happens inconsistently, ie, it doesn't happen on the same nodes between the deployment.

Also I got some errors on regiond.log:
2019-11-06 21:25:28 regiond: [info] 10.243.165.2 GET /MAAS/metadata/curtin/2012-03-01/user-data HTTP/1.0 --> 401 UNAUTHORIZED (referrer: -; agent: python-requests/2.18.4)
2019-11-06 21:26:27 regiond: [info] 10.243.165.2 GET /MAAS/metadata/curtin/2012-03-01/user-data HTTP/1.0 --> 401 UNAUTHORIZED (referrer: -; agent: python-requests/2.18.4)
2019-11-07 12:37:31 regiond: [info] 10.243.165.2 GET /MAAS/metadata/curtin/2012-03-01/user-data HTTP/1.0 --> 401 UNAUTHORIZED (referrer: -; agent: python-requests/2.18.4)

Andre Ruiz (andre-ruiz) wrote :

Just to add more info, the PXE process goes well, kernel and rootfs are downloaded ok, boot starts and at some point in the commissioning process cloud-init says that could not find meta-data and gives up, the commission fails.

Around that moment in time, we can see in a packet capture dump analyzed in wireshark that the machine gets this error from maas. All other machines that succeed do not get it.

subscribed ~field-critical

Alberto Donato (ack) wrote :

@Marcelo, from IRC log you linked a part of region log showing that a rackcontroller is failing to connect to the region. Is that happening concurrently with this issue?

@Alberto, I thing that was related to a service restart I did. I'm testing again just to check if a connection error will be shown on logs.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers