My raspberry pi 3B+ no longer boots core24 with memory limit set to 395MB - (assuming 128MB is used by the videocore side, this leaves just 267MB of memory for all of Linux. I have nothing on the serial port but there's a backtrace on HDMI where plymouthd crashes somewhere in freetype. With some more memory I was able to move to checkbox for core-24 from edge: zyga@pi3-2:~$ snap list Name Version Rev Tracking Publisher Notes checkbox 4.0.0-dev297 7268 uc24/edge ce-certification-qa devmode checkbox18 3.3.0-dev19 3005 latest/stable ce-certification-qa disabled checkbox24 4.0.0-dev297 25 latest/edge ce-certification-qa - console-conf 24.04.1+git45g5f9fae19+gd81a15d 41 24/stable canonical✓ - core18 20240416 2826 latest/stable canonical✓ base core24 20240528 424 latest/stable canonical✓ base pi 24-1 142 24/stable canonical✓ gadget pi-kernel 6.8.0-1005.5 852 24/stable canonical✓ kernel snapd 2.63 21761 latest/stable canonical✓ snapd Note that "some more memory" is still significantly less than 512MB: zyga@pi3-2:~$ free -m total used free shared buff/cache available Mem: 299 150 57 4 104 148 Swap: 0 0 0 Again I limited memory with config.txt: # XXX: artificially limit available memory to stress-test the system. total_mem=395 gpu_mem=64 With this limit I can complete a checkbox run. While not the best overall estimate, the total memory needed in the system.slice was about 180MB (1024) zyga@pi3-2:/sys/fs/cgroup$ cat system.slice/memory.peak 190345216 The result I've obtained: Finalizing session that hasn't been submitted anywhere: checkbox-run-2024-06-10T08.58.22 ==================================[ Results ]=================================== ☑ : Collect information about system memory (/proc/meminfo) ☑ : Enumerate available system executables ☑ : Attach information about block devices and their mount points. ☑ : Collect information about the CPU ☐ : Attaches JSON dumps of system info tools ☑ : Provide links to requirements documents ☑ : Attach a copy of /proc/cmdline ☑ : Collect information about installed software packages ☑ : Resource to detect if dmi data is present ☐ : Attaches JSON dumps of raw DMI devices ☑ : Collect information about the running kernel ☑ : Collect information about the EFI configuration ☑ : Collect information about installed system (lsb-release) ☑ : Attaches json dumps of installed dkms package information. ☑ : Create resource info for environment variables ☐ : Attach detailed sysfs property output from udev ☑ : Collect information about hardware devices (udev) ☒ : Collect information about installed snap packages ☑ : Collect information about kernel modules ☑ : Attach PCI configuration space hex dump ☐ : Attach a copy of /sys/class/dmi/id/* ☐ : Collect information about hardware devices (DMI) ☑ : Collect information about dpkg version ☑ : Attaches JSON dumps of udev resource information. ☑ : Attach dump of udev database ☑ : Attach the contents of /etc/modprobe.* ☑ : Collect information about installation media (casper) ☐ : Check that data for a complete result are present ☑ : Attach the contents of /proc/acpi/sleep for further analysis. ☒ : Attach a report of installed codecs for Intel HDA. ☑ : Attach a copy of /proc/cpuinfo ☑ : Attach a copy of dmesg or the current dmesg buffer to the test results. ☐ : Attach output of dmidecode ☑ : Attaches firmware version info ☑ : Attach a list of PCI devices ☑ : Attach copy of /proc/meminfo ☑ : Attach the contents of /etc/modprobe.* ☑ : Attach the contents of /etc/modules ☑ : Identify what service is managing each physical network interface ☑ : Collect logging from the net_if_management job ☑ : Attach sysctl configuration files. ☑ : Attach a list of currently running kernel modules ☑ : Hardware Manifest ☐ : Check that at least one audio capture device exists ☐ : Check that at least one audio playback device exists ☐ : Captured sound matches played one (automated) ☐ : bluetooth/detect-output ☑ : Test the CPU scaling capabilities ☑ : Attach CPU scaling capabilities log ☑ : cpu_offlining ☐ : Test offlining of each CPU core ☐ : Check CPU topology for accuracy between proc and sysfs ☐ : Automated test of SD Card reading & writing (udisks2) ☑ : Check the amount of memory reported by meminfo against DMI ☐ : Detect if at least one ethernet device is detected ☑ : Gather info on the current state of network devices ☒ : Ensure downloading files through HTTP works correctly. ☑ : Can ping the gateway with any cable Ethernet interface ☑ : Creates resource info for RTC ☐ : Test that RTC functions properly (if present) ☑ : Executes ACPI Wakealarm test to validate functionality. ☑ : Attach and display fwts wakealarm test log. ☑ : Display USB devices attached to SUT ☐ : Test USB 2.0 or 1.1 ports ☒ : Collect information about connections ☐ : Test system can discover Wi-Fi networks on wlan0 ☐ : Connect to WPA-encrypted 802.11b/g Wi-Fi network on wlan0 ☐ : Connect to an unencrypted 802.11b/g Wi-Fi network on wlan0 ☐ : Connect to a WPA-encrypted 802.11n Wi-Fi network on wlan0 ☐ : Connect to an unencrypted 802.11n Wi-Fi network on wlan0 ☑ : Resource job to identify Wi-Fi STA supported protocols ☐ : Connect to WPA-encrypted 802.11ac Wi-Fi network on wlan0 ☐ : Connect to unencrypted 802.11ac Wi-Fi network on wlan0 ☒ : Test system can get beacon EddyStone URL advertisements on the hci0 adapter ☑ : Verify that all the CPUs are online before suspending ☑ : Store memory info before suspending ☑ : Create resource info for supported sleep states ☐ : Automated test of suspend function ☐ : Automated check of the suspend log for errors reported by fwts ☐ : Attaches the log from the single suspend/resume test ☐ : Network reconnect resume test (wired) ☐ : Network reconnect resume test (wifi) ☐ : Test USB 2.0 or 1.1 ports after suspend (S3) ☐ : Captured sound matches played one (automated) after suspend (S3) ☐ : Test system can get beacon EddyStone URL advertisements on the hci0 adapter after suspend (S3) ☐ : Verify that all the CPUs are online after suspending ☐ : Compare memory info to the state prior to suspend ☐ : Connect to an unencrypted 802.11b/g Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to an unencrypted 802.11n Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to WPA-encrypted 802.11b/g Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to a WPA-encrypted 802.11n Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to WPA-encrypted 802.11ac Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to unencrypted 802.11ac Wi-Fi network on wlan0 after suspend (S3) All of this was done with snapd 2.63. Reverting to snapd 2.62 I have the following results: ==================================[ Results ]=================================== ☑ : Collect information about dpkg version ☑ : Collect information about the running kernel ☑ : Collect information about the CPU ☐ : Attaches JSON dumps of system info tools ☑ : Attaches json dumps of installed dkms package information. ☑ : Collect information about installation media (casper) ☑ : Create resource info for environment variables ☑ : Resource to detect if dmi data is present ☐ : Collect information about hardware devices (DMI) ☑ : Attach a copy of /proc/cmdline ☑ : Attach the contents of /etc/modprobe.* ☑ : Attach dump of udev database ☑ : Collect information about installed software packages ☑ : Collect information about system memory (/proc/meminfo) ☑ : Collect information about installed system (lsb-release) ☑ : Attach PCI configuration space hex dump ☐ : Attach detailed sysfs property output from udev ☒ : Collect information about installed snap packages ☑ : Enumerate available system executables ☑ : Attach information about block devices and their mount points. ☐ : Attaches JSON dumps of raw DMI devices ☑ : Provide links to requirements documents ☐ : Attach a copy of /sys/class/dmi/id/* ☑ : Collect information about kernel modules ☑ : Collect information about hardware devices (udev) ☑ : Attaches JSON dumps of udev resource information. ☑ : Collect information about the EFI configuration ☐ : Check that data for a complete result are present ☑ : Attach the contents of /proc/acpi/sleep for further analysis. ☒ : Attach a report of installed codecs for Intel HDA. ☑ : Attach a copy of /proc/cpuinfo ☑ : Attach a copy of dmesg or the current dmesg buffer to the test results. ☐ : Attach output of dmidecode ☑ : Attaches firmware version info ☑ : Attach a list of PCI devices ☑ : Attach copy of /proc/meminfo ☑ : Attach the contents of /etc/modprobe.* ☑ : Attach the contents of /etc/modules ☑ : Identify what service is managing each physical network interface ☑ : Collect logging from the net_if_management job ☑ : Attach sysctl configuration files. ☑ : Attach a list of currently running kernel modules ☑ : Hardware Manifest ☐ : Check that at least one audio capture device exists ☐ : Check that at least one audio playback device exists ☐ : Captured sound matches played one (automated) ☐ : bluetooth/detect-output ☑ : Test the CPU scaling capabilities ☑ : Attach CPU scaling capabilities log ☑ : cpu_offlining ☐ : Test offlining of each CPU core ☐ : Check CPU topology for accuracy between proc and sysfs ☐ : Automated test of SD Card reading & writing (udisks2) ☑ : Check the amount of memory reported by meminfo against DMI ☐ : Detect if at least one ethernet device is detected ☑ : Gather info on the current state of network devices ☒ : Ensure downloading files through HTTP works correctly. ☑ : Can ping the gateway with any cable Ethernet interface ☑ : Creates resource info for RTC ☐ : Test that RTC functions properly (if present) ☑ : Executes ACPI Wakealarm test to validate functionality. ☑ : Attach and display fwts wakealarm test log. ☑ : Display USB devices attached to SUT ☐ : Test USB 2.0 or 1.1 ports ☒ : Collect information about connections ☐ : Test system can discover Wi-Fi networks on wlan0 ☐ : Connect to WPA-encrypted 802.11b/g Wi-Fi network on wlan0 ☐ : Connect to an unencrypted 802.11b/g Wi-Fi network on wlan0 ☐ : Connect to a WPA-encrypted 802.11n Wi-Fi network on wlan0 ☐ : Connect to an unencrypted 802.11n Wi-Fi network on wlan0 ☑ : Resource job to identify Wi-Fi STA supported protocols ☐ : Connect to WPA-encrypted 802.11ac Wi-Fi network on wlan0 ☐ : Connect to unencrypted 802.11ac Wi-Fi network on wlan0 ☒ : Test system can get beacon EddyStone URL advertisements on the hci0 adapter ☑ : Verify that all the CPUs are online before suspending ☑ : Store memory info before suspending ☑ : Create resource info for supported sleep states ☐ : Automated test of suspend function ☐ : Automated check of the suspend log for errors reported by fwts ☐ : Attaches the log from the single suspend/resume test ☐ : Network reconnect resume test (wired) ☐ : Network reconnect resume test (wifi) ☐ : Test USB 2.0 or 1.1 ports after suspend (S3) ☐ : Captured sound matches played one (automated) after suspend (S3) ☐ : Test system can get beacon EddyStone URL advertisements on the hci0 adapter after suspend (S3) ☐ : Verify that all the CPUs are online after suspending ☐ : Compare memory info to the state prior to suspend ☐ : Connect to an unencrypted 802.11b/g Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to an unencrypted 802.11n Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to WPA-encrypted 802.11b/g Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to a WPA-encrypted 802.11n Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to WPA-encrypted 802.11ac Wi-Fi network on wlan0 after suspend (S3) ☐ : Connect to unencrypted 802.11ac Wi-Fi network on wlan0 after suspend (S3) The memory stats for that run as as follows. Freshly after booting: zyga@pi3-2:~$ cat /sys/fs/cgroup/system.slice/memory.peak 160915456 zyga@pi3-2:~$ cat /sys/fs/cgroup/system.slice/memory.peak 199782400 To clarify, this is more memory than on 2.63. As such I think there's no clearly measurable difference where snapd 2.63 uses dramatically more memory than snapd 2.62 did. One important caveat is that I was neither in the certification environment - some tests got skipped (like WIFI connection tests) nor was I using the same hardware (I was using Raspberry Pi 3B+ as 3B does not boot with core 18 due to u-boot, presumably). Given that I severely limited memory on my device, to much less than what the certification team was using, and given that I was able to complete the test on both version of snapd with roughly the same amount of memory I will now seek help from the cert team to debug what they are observing in their environment.