MAAS

Bug #1958433
Comment #13

Comment 13 for bug 1958433

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2022-05-20:

#13

Submitted MRs with (clean) backports for the stable branches (3.1, 3.0, 2.9).

All tested successfully with MAAS SNAP 3.1/3.0/2.9.

Test Steps:
---

Scenario:
- 1 maas VM
- 1 test VM (boot from network/hd)
- 1 bridge with both VMs (no DHCP)

1) Deploy MAAS 3.1/3.0/2.9 SNAP from stable channel:

$ sudo snap install --channel=3.1/stable maas
$ sudo snap install --channel=3.1/stable maas-test-db
$ sudo maas init region+rack --database-uri maas-test-db:///
$ sudo maas createadmin --username admin --password admin --email <email address hidden> --ssh-import lp:mfo

2) Configure MAAS server in the Web UI
- Images: Ubuntu 18.04/20.04/22.04 and CentOS 7/8.
- Subnet: Enable DHCP in subnet/vlan w/ bridge to test VM.

3) Setup test VM in MAAS
- Boot test VM (enlist in MAAS)
- Edit test VM (set power: manual)
- Provision it (start it manually)
- Configure it (add XFS partition/mount)

4) Setup deployment tests (stop/start VM as needed)
- Configure the commissioning image to Ubuntu 20.04 (or other versions later)
- Deploy CentOS 7 (or 8 later)

5) Tests and results:
[commisioning image / deployed OS / original or patched curtin_userdata_centos]

- 20.04 / CentOS 7 / original: FAIL (problem reproduces; see [1])
- 20.04 / CentOS 7 / modified: PASS (problem fixed; see [2]
- 20.04 / CentOS 8 / modified: PASS (no regression; wrapper not setup; see [3])

- 18.04 / CentOS 7 / modified: PASS (no regression; wrapper setup, reflink still off; see [4])
- 18.04 / CentOS 8 / modified: PASS (no regression; wrapper not setup)

- 22.04 / CentOS 7 / modified: PASS (problem fixed)
- 22.04 / CentOS 8 / modified: PASS (no regression; wrapper not setup)

In order to test the changes, just replace the existing
'curtin_userdata_centos' file in snap's read-only side,
with the updated one, using a bind mount:

Before)

$ diff -U0 /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos ~/curtin_userdata_centos | tail -n+4
+early_commands:
+ centos70_xfs_lp1958433: [ '/bin/sh', '-c', 'if [ "{{release}}" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi' ]
+

Switch)

$ sudo mount --bind ~/curtin_userdata_centos /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos

After)

$ diff -U0 /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos ~/curtin_userdata_centos | tail -n+4
$

To revert it back:

$ sudo umount /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos

To uninstall/move to another version:

$ sudo snap remove --purge maas
$ sudo snap remove --purge maas-test-db

Examples:
---

[1] 20.04 / CentOS 7 / original: FAIL

...
Ubuntu 20.04.4 LTS vmaas ttyS0
...
[ 78.349650] cloud-init[1430]: start: cmd-install/stage-early: preparing for installation
[ 78.366818] cloud-init[1430]: stage_early took 0.000 seconds
[ 78.369113] cloud-init[1430]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
...
...
[ OK ] Reached target Local File Systems (Pre).
...
Mounting /xfs...
...
[ 4.363480] SGI XFS with ACLs, security attributes, no debug enabled
[ 4.393128] XFS (vda3): Superblock has unknown read-only compatible features (0x4) enabled.
[ 4.394334] XFS (vda3): Attempted to mount read-only compatible filesystem read-write.
[ 4.395509] XFS (vda3): Filesystem can only be safely mounted read only.
[ 4.396522] XFS (vda3): SB validate failed with error -22.
[FAILED] Failed to mount /xfs.
See 'systemctl status xfs.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Mark the need to relabel after reboot.
[DEPEND] Dependency failed for Relabel all filesystems, if necessary.
[DEPEND] Dependency failed for Migrate local... structure to the new structure.
...
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
try again to boot into default mode.

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.

[2] 20.04 / CentOS 7 / modified: PASS

...
Ubuntu 20.04.4 LTS vmaas ttyS0
...
[ 69.728496] cloud-init[1435]: start: cmd-install/stage-early: preparing for installation
[ 69.770552] cloud-init[1435]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 69.836359] cloud-init[1435]: Wrapper: /usr/local/sbin/mkfs.xfs
[ 69.846608] cloud-init[1435]: #!/bin/sh
[ 69.848658] cloud-init[1435]: exec /usr/sbin/mkfs.xfs -m reflink=0 "$@"
[ 69.851900] cloud-init[1435]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 69.862270] cloud-init[1435]: centos70_xfs_lp1958433 took 0.093 seconds
[ 69.864442] cloud-init[1435]: stage_early took 0.093 seconds
[ 69.866318] cloud-init[1435]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
...
Mounting /xfs...
...
[ 3.757024] XFS (vda3): Mounting V5 Filesystem
[ 3.766105] XFS (vda3): Ending clean mount
[ OK ] Mounted /xfs.
...
CentOS Linux 7 (Core)
Kernel 3.10.0-1160.45.1.el7.x86_64 on an x86_64

vmaas login:

[3] 20.04 / CentOS 8 / modified: PASS

Note `if [ "8" = "centos70" ]`; so wrapper isn't setup on CentOS 8.

[ 53.511438] cloud-init[1034]: start: cmd-install/stage-early: preparing for installation
[ 53.525050] cloud-init[1034]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "8" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 53.536931] cloud-init[1034]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "8" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 53.546719] cloud-init[1034]: centos70_xfs_lp1958433 took 0.023 seconds
[ 53.548471] cloud-init[1034]: stage_early took 0.023 seconds
[ 53.550020] cloud-init[1034]: finish: cmd-install/stage-early: SUCCESS: preparing for installation

[4] 18.04 / CentOS 7 / modified: PASS

...
Welcome to Ubuntu 18.04.6 LTS!
...
[ 51.165424] cloud-init[1004]: start: cmd-install/stage-early: preparing for installation
[ 51.178881] cloud-init[1004]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 51.209899] cloud-init[1004]: Wrapper: /usr/local/sbin/mkfs.xfs
[ 51.213089] cloud-init[1004]: #!/bin/sh
[ 51.214400] cloud-init[1004]: exec /sbin/mkfs.xfs -m reflink=0 "$@"
[ 51.218629] cloud-init[1004]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
[ 51.228455] cloud-init[1004]: centos70_xfs_lp1958433 took 0.049 seconds
[ 51.232247] cloud-init[1004]: stage_early took 0.050 seconds
[ 51.233700] cloud-init[1004]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
...
Mounting /xfs...
...
[ 3.781090] SGI XFS with ACLs, security attributes, no debug enabled
[ 3.811470] XFS (vda3): Mounting V5 Filesystem
[ 3.818785] XFS (vda3): Ending clean mount
[ OK ] Mounted /xfs.
...
CentOS Linux 7 (Core)
Kernel 3.10.0-1160.45.1.el7.x86_64 on an x86_64

vmaas login:

Submitted MRs with (clean) backports for the stable branches (3.1, 3.0, 2.9).

All tested successfully with MAAS SNAP 3.1/3.0/2.9.

Test Steps:
---

Scenario:
- 1 maas VM
- 1 test VM (boot from network/hd)
- 1 bridge with both VMs (no DHCP)

1) Deploy MAAS 3.1/3.0/2.9 SNAP from stable channel:

$ sudo snap install --channel=3.1/stable maas
	$ sudo snap install --channel=3.1/stable maas-test-db
	$ sudo maas init region+rack --database-uri maas-test-db:///
	$ sudo maas createadmin --username admin --password admin --email admin@admin.adm --ssh-import lp:mfo

2) Configure MAAS server in the Web UI
- Images: Ubuntu 18.04/20.04/22.04 and CentOS 7/8.
- Subnet: Enable DHCP in subnet/vlan w/ bridge to test VM.

3) Setup test VM in MAAS
- Boot test VM (enlist in MAAS)
- Edit test VM (set power: manual)
- Provision it (start it manually)
- Configure it (add XFS partition/mount)

4) Setup deployment tests (stop/start VM as needed)
- Configure the commissioning image to Ubuntu 20.04 (or other versions later)
- Deploy CentOS 7 (or 8 later)

5) Tests and results:
[commisioning image / deployed OS / original or patched curtin_userdata_centos]

- 18.04 / CentOS 7 / modified: PASS (no regression; wrapper setup, reflink still off; see [4])
- 18.04 / CentOS 8 / modified: PASS (no regression; wrapper not setup)

- 22.04 / CentOS 7 / modified: PASS (problem fixed)
- 22.04 / CentOS 8 / modified: PASS (no regression; wrapper not setup)

In order to test the changes, just replace the existing
'curtin_userdata_centos' file in snap's read-only side,
with the updated one, using a bind mount:

Before)

$ diff -U0 /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos ~/curtin_userdata_centos | tail -n+4
	+early_commands:
	+  centos70_xfs_lp1958433: [ '/bin/sh', '-c', 'if [ "{{release}}" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi' ]
	+

Switch)

$ sudo mount --bind ~/curtin_userdata_centos /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos

After)

$ diff -U0 /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos ~/curtin_userdata_centos | tail -n+4
	$

To revert it back:

$ sudo umount /snap/maas/current/etc/maas/preseeds/curtin_userdata_centos

To uninstall/move to another version:

$ sudo snap remove --purge maas
	$ sudo snap remove --purge maas-test-db

Examples:
---

[1] 20.04 / CentOS 7 / original: FAIL

...
	Ubuntu 20.04.4 LTS vmaas ttyS0
	...
	[   78.349650] cloud-init[1430]: start: cmd-install/stage-early: preparing for installation
	[   78.366818] cloud-init[1430]: stage_early took 0.000 seconds
	[   78.369113] cloud-init[1430]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
	...
	...
	[  OK  ] Reached target Local File Systems (Pre).
	...
		 Mounting /xfs...
	...
	[    4.363480] SGI XFS with ACLs, security attributes, no debug enabled
	[    4.393128] XFS (vda3): Superblock has unknown read-only compatible features (0x4) enabled.
	[    4.394334] XFS (vda3): Attempted to mount read-only compatible filesystem read-write.
	[    4.395509] XFS (vda3): Filesystem can only be safely mounted read only.
	[    4.396522] XFS (vda3): SB validate failed with error -22.
	[FAILED] Failed to mount /xfs.
	See 'systemctl status xfs.mount' for details.
	[DEPEND] Dependency failed for Local File Systems.
	[DEPEND] Dependency failed for Mark the need to relabel after reboot.
	[DEPEND] Dependency failed for Relabel all filesystems, if necessary.
	[DEPEND] Dependency failed for Migrate local... structure to the new structure.
	...
	Welcome to emergency mode! After logging in, type "journalctl -xb" to view
	system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
	try again to boot into default mode.

Cannot open access to console, the root account is locked.
	See sulogin(8) man page for more details.

Press Enter to continue.

[2] 20.04 / CentOS 7 / modified: PASS

...
	Ubuntu 20.04.4 LTS vmaas ttyS0
	...
	[   69.728496] cloud-init[1435]: start: cmd-install/stage-early: preparing for installation
	[   69.770552] cloud-init[1435]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   69.836359] cloud-init[1435]: Wrapper: /usr/local/sbin/mkfs.xfs
	[   69.846608] cloud-init[1435]: #!/bin/sh
	[   69.848658] cloud-init[1435]: exec /usr/sbin/mkfs.xfs -m reflink=0 "$@"
	[   69.851900] cloud-init[1435]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   69.862270] cloud-init[1435]: centos70_xfs_lp1958433 took 0.093 seconds
	[   69.864442] cloud-init[1435]: stage_early took 0.093 seconds
	[   69.866318] cloud-init[1435]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
	...
		 Mounting /xfs...
	...
	[    3.757024] XFS (vda3): Mounting V5 Filesystem
	[    3.766105] XFS (vda3): Ending clean mount
	[  OK  ] Mounted /xfs.
	...
	CentOS Linux 7 (Core)
	Kernel 3.10.0-1160.45.1.el7.x86_64 on an x86_64

vmaas login:

[3] 20.04 / CentOS 8 / modified: PASS

Note  `if [ "8" = "centos70" ]`; so wrapper isn't setup on CentOS 8.

[   53.511438] cloud-init[1034]: start: cmd-install/stage-early: preparing for installation
	[   53.525050] cloud-init[1034]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "8" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   53.536931] cloud-init[1034]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "8" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   53.546719] cloud-init[1034]: centos70_xfs_lp1958433 took 0.023 seconds
	[   53.548471] cloud-init[1034]: stage_early took 0.023 seconds
	[   53.550020] cloud-init[1034]: finish: cmd-install/stage-early: SUCCESS: preparing for installation

[4] 18.04 / CentOS 7 / modified: PASS

...
	Welcome to Ubuntu 18.04.6 LTS!
	...
	[   51.165424] cloud-init[1004]: start: cmd-install/stage-early: preparing for installation
	[   51.178881] cloud-init[1004]: start: cmd-install/stage-early/centos70_xfs_lp1958433: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   51.209899] cloud-init[1004]: Wrapper: /usr/local/sbin/mkfs.xfs
	[   51.213089] cloud-init[1004]: #!/bin/sh
	[   51.214400] cloud-init[1004]: exec /sbin/mkfs.xfs -m reflink=0 "$@"
	[   51.218629] cloud-init[1004]: finish: cmd-install/stage-early/centos70_xfs_lp1958433: SUCCESS: running '/bin/sh -c if [ "centos70" = "centos70" ] && mkfs.xfs 2>&1 | grep -q "reflink=0|1"; then WRAPPER=/usr/local/sbin/mkfs.xfs; echo "#!/bin/sh" >$WRAPPER && echo "exec $(which mkfs.xfs) -m reflink=0 \"\$@\"" >>$WRAPPER && chmod +x $WRAPPER && echo "Wrapper: $WRAPPER" && cat $WRAPPER; fi'
	[   51.228455] cloud-init[1004]: centos70_xfs_lp1958433 took 0.049 seconds
	[   51.232247] cloud-init[1004]: stage_early took 0.050 seconds
	[   51.233700] cloud-init[1004]: finish: cmd-install/stage-early: SUCCESS: preparing for installation
	...
		 Mounting /xfs...
	...
	[    3.781090] SGI XFS with ACLs, security attributes, no debug enabled
	[    3.811470] XFS (vda3): Mounting V5 Filesystem
	[    3.818785] XFS (vda3): Ending clean mount
	[  OK  ] Mounted /xfs.
	...
	CentOS Linux 7 (Core)
	Kernel 3.10.0-1160.45.1.el7.x86_64 on an x86_64

vmaas login: