Watchdog didn't set timeout after ubuntu core recovering

[LIAO, YU-SIANG]

[Title]
Watchdog didn't set timeout after ubuntu core recovering

[Summary]
After recovering the ubuntu core image and wait to finish the snap auto update, testing watchdog function always fail, because there is the error that message text is "Running in chroot, ignoring command 'daemon-reexec'" in the "watchdog/general/set-timeout" test block.

[Reproduce Steps]
1. Use command "$ sudo snap reboot install" to recover in the ubuntu core system.
2. Use command "sudo snap install --devmode --channel=latest/stable checkbox-iiotg" to install snap checkbox and add watchdog enable variable in checkbox manifest.
3. Run the "watchdog/general/set-timeout" test case.

[Results]

Expected: Can set timeout and can see this "RuntimeWatchdogSec" varable that has the value in the "/etc/systemd/system.conf" file.

Actual: It can run but always not change this "RuntimeWatchdogSec" varable value, test case can run successfully, but there are error message "Running in chroot, ignoring command 'daemon-reexec'" in the console.It seems that do the "sudo systemctl daemon-reexec" command has some problem.

[Additional Information]
<I try to iotg device and always have same problem>
CID1: 202109-29492 Lookout-Canyon Aaeon EHL
CID2: 202110-29509 Lookout-Canyon Aaeon TGL
Base Image: stock image in iotg [ubuntu-core-22-amd64+intel-iot]
BIOS Version: UNEHAM11
Lernel version: 5.15.0-1031-intel-iotg

[Vic Liu (zongminl)]

This problem is not observed in QA testing cycles, please provide more information

[Vic Liu (zongminl)]

Attached C3 submission for reference: https://certification.canonical.com/hardware/202305-31589/submission/316066/test-results/

[LIAO, YU-SIANG (flyjerry0415)]

Because we always do recovery action before running jenkins checkbox test plane in cert team, they always fail in the first time, do power cycle and it works normal after this.

[Vic Liu (zongminl)]

@jerry,

I'm seeing something suspicious can you please remove the sudo from the following lines and make them to be only `systemctl daemon-reexec; RET=$?` sideload the provider and see if this fix the issue?

https://git.launchpad.net/~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg/tree/units/watchdog/jobs.pxu#n125

https://git.launchpad.net/~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg/tree/units/watchdog/jobs.pxu#n164

[LIAO, YU-SIANG (flyjerry0415)]

I use the sideload provider and try to modify these command that you recommend, and after recovering ubuntu core image, it still error on it, I will attach the picture for your reference.

[Vic Liu (zongminl)]

I have reproduced the issue with uc22 image, setting bug status to confirmed

[Vic Liu (zongminl)]

With checkbox remote, this problem is not observed, I'm lowering the priority to medium for it's not going to block SRU testing.

[LIAO, YU-SIANG (flyjerry0415)]

But we will have the same problem whether it is local or remote, This problem let us run jenkins test job fail every time. We use "checkbox-iiotg devmode --channel=latest/stable" and this test plan "com.canonical.qa.intliotg::intliotg-ubuntucore-22-automated". Any parameters you can refer to this submission. https://certification.canonical.com/hardware/202110-29509/submission/322523/