| 2020-03-23 13:41:10 |
Edward Hope-Morley |
bug |
|
|
added bug |
| 2020-03-23 13:57:28 |
Ian Johnson |
bug |
|
|
added subscriber Ian Johnson |
| 2020-03-23 14:13:36 |
Edward Hope-Morley |
vaultlocker: assignee |
|
Edward Hope-Morley (hopem) |
|
| 2020-03-23 14:13:38 |
Edward Hope-Morley |
vaultlocker: status |
New |
In Progress |
|
| 2020-03-23 14:48:32 |
Edward Hope-Morley |
bug task added |
|
vaultlocker (Ubuntu) |
|
| 2020-03-23 14:48:37 |
Edward Hope-Morley |
vaultlocker (Ubuntu): status |
New |
In Progress |
|
| 2020-03-23 14:48:40 |
Edward Hope-Morley |
vaultlocker (Ubuntu): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
bug task added |
|
vaultlocker (Ubuntu Bionic) |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
nominated for series |
|
Ubuntu Focal |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
bug task added |
|
vaultlocker (Ubuntu Focal) |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
nominated for series |
|
Ubuntu Eoan |
|
| 2020-03-23 14:48:53 |
Edward Hope-Morley |
bug task added |
|
vaultlocker (Ubuntu Eoan) |
|
| 2020-03-23 15:51:56 |
James Page |
vaultlocker: status |
In Progress |
Fix Released |
|
| 2020-03-30 13:43:20 |
James Page |
vaultlocker (Ubuntu Focal): status |
In Progress |
Fix Released |
|
| 2020-03-30 13:47:42 |
James Page |
vaultlocker (Ubuntu Focal): status |
Fix Released |
Fix Committed |
|
| 2020-03-30 15:26:49 |
Launchpad Janitor |
vaultlocker (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2020-04-09 08:24:45 |
James Page |
bug task deleted |
vaultlocker (Ubuntu Bionic) |
|
|
| 2020-04-09 08:24:57 |
James Page |
bug task added |
|
bionic-backports |
|
| 2020-04-09 08:25:03 |
James Page |
vaultlocker (Ubuntu Eoan): status |
New |
Triaged |
|
| 2020-04-09 08:25:09 |
James Page |
vaultlocker (Ubuntu Eoan): importance |
Undecided |
High |
|
| 2020-04-09 08:25:14 |
James Page |
vaultlocker (Ubuntu Focal): importance |
Undecided |
High |
|
| 2020-04-09 08:25:27 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-04-09 08:34:13 |
James Page |
description |
On a node that has multiple networks configured and vaultlocker is used for decrypting ceph osds, if vaultlocker starts (specifically vaultlocker-decrypt systemd units) prior to dns being configured, it appears that it will spin forever when the vault url contains hostnames (i.e. not IP addresses). What we see is that there are no crypt- devices and there are per-osd vaultocker processes running that if we strace we see are spinning in select(NULL, NULL, ...) which is socket.gethostname() at [1]. The only way to fix this currently is to manually restart the vaultlocker process so that current dns settings are picked up. It appears that this behavior was introduced by the fix for bug 1838607 [2] which means that vaultlocker no longer waits for all networking to be UP and ready and therefor does not wait for dns to be setup.
We tried adding After=nss-lookup.target to the vaultlocker-decrypt unit configs and rebooted the node and that resolved the problem.
[1] https://github.com/openstack-charmers/vaultlocker/blob/master/vaultlocker/shell.py#L54
[2] https://github.com/openstack-charmers/vaultlocker/pull/7/files |
[Impact]
[Test Case]
[Regression Potential]
[Original Bug Report]
On a node that has multiple networks configured and vaultlocker is used for decrypting ceph osds, if vaultlocker starts (specifically vaultlocker-decrypt systemd units) prior to dns being configured, it appears that it will spin forever when the vault url contains hostnames (i.e. not IP addresses). What we see is that there are no crypt- devices and there are per-osd vaultocker processes running that if we strace we see are spinning in select(NULL, NULL, ...) which is socket.gethostname() at [1]. The only way to fix this currently is to manually restart the vaultlocker process so that current dns settings are picked up. It appears that this behavior was introduced by the fix for bug 1838607 [2] which means that vaultlocker no longer waits for all networking to be UP and ready and therefor does not wait for dns to be setup.
We tried adding After=nss-lookup.target to the vaultlocker-decrypt unit configs and rebooted the node and that resolved the problem.
[1] https://github.com/openstack-charmers/vaultlocker/blob/master/vaultlocker/shell.py#L54
[2] https://github.com/openstack-charmers/vaultlocker/pull/7/files |
|
| 2020-04-09 08:36:19 |
James Page |
description |
[Impact]
[Test Case]
[Regression Potential]
[Original Bug Report]
On a node that has multiple networks configured and vaultlocker is used for decrypting ceph osds, if vaultlocker starts (specifically vaultlocker-decrypt systemd units) prior to dns being configured, it appears that it will spin forever when the vault url contains hostnames (i.e. not IP addresses). What we see is that there are no crypt- devices and there are per-osd vaultocker processes running that if we strace we see are spinning in select(NULL, NULL, ...) which is socket.gethostname() at [1]. The only way to fix this currently is to manually restart the vaultlocker process so that current dns settings are picked up. It appears that this behavior was introduced by the fix for bug 1838607 [2] which means that vaultlocker no longer waits for all networking to be UP and ready and therefor does not wait for dns to be setup.
We tried adding After=nss-lookup.target to the vaultlocker-decrypt unit configs and rebooted the node and that resolved the problem.
[1] https://github.com/openstack-charmers/vaultlocker/blob/master/vaultlocker/shell.py#L54
[2] https://github.com/openstack-charmers/vaultlocker/pull/7/files |
[Impact]
vaultlocker decrypt systemd units start to early in boot process and as a result can't determine the local hostname of the machine they are running on, resulting in failure to retrieve keys from vault.
[Test Case]
This is somewhat tricky to reproduce as its a bit of a race condition - the original bug reporter will help with testing as it was fairly reliable reproduced in the impacted deployment.
[Regression Potential]
Low - the fix (release as the only change in 1.0.6) simple ensures that nss-lookup.target has completed before running the vaultlocker-decrypt units and has been tested using overrides in the impacted deployment.
[Original Bug Report]
On a node that has multiple networks configured and vaultlocker is used for decrypting ceph osds, if vaultlocker starts (specifically vaultlocker-decrypt systemd units) prior to dns being configured, it appears that it will spin forever when the vault url contains hostnames (i.e. not IP addresses). What we see is that there are no crypt- devices and there are per-osd vaultocker processes running that if we strace we see are spinning in select(NULL, NULL, ...) which is socket.gethostname() at [1]. The only way to fix this currently is to manually restart the vaultlocker process so that current dns settings are picked up. It appears that this behavior was introduced by the fix for bug 1838607 [2] which means that vaultlocker no longer waits for all networking to be UP and ready and therefor does not wait for dns to be setup.
We tried adding After=nss-lookup.target to the vaultlocker-decrypt unit configs and rebooted the node and that resolved the problem.
[1] https://github.com/openstack-charmers/vaultlocker/blob/master/vaultlocker/shell.py#L54
[2] https://github.com/openstack-charmers/vaultlocker/pull/7/files |
|
| 2020-04-14 21:45:20 |
Brian Murray |
vaultlocker (Ubuntu Eoan): status |
Triaged |
Fix Committed |
|
| 2020-04-14 21:45:24 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2020-04-14 21:45:30 |
Brian Murray |
tags |
|
verification-needed verification-needed-eoan |
|
| 2020-04-16 12:57:46 |
Edward Hope-Morley |
tags |
verification-needed verification-needed-eoan |
verification-done verification-done-eoan |
|
| 2020-04-16 12:57:55 |
Edward Hope-Morley |
tags |
verification-done verification-done-eoan |
sts-sru-needed verification-done verification-done-eoan |
|
| 2020-04-27 12:45:31 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-04-27 12:55:32 |
Launchpad Janitor |
vaultlocker (Ubuntu Eoan): status |
Fix Committed |
Fix Released |
|
| 2020-04-27 12:59:29 |
James Page |
bionic-backports: status |
New |
Fix Released |
|
