Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables.
In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems.
```
$ systemctl status systemd-pstore
○ systemd-pstore.service - Platform Persistent Storage Archival
Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago
└─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met
Docs: man:systemd-pstore(8)
Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
$ ls -la /sys/fs/pstore
total 0
drwxr-x--- 2 root root 0 Jun 9 09:11 .
drwxr-xr-x 8 root root 0 Jun 9 09:11 ..
-r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001
-r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001
-r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001
-r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001
-r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001
-r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001
-r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001
-r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001
-r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001
-r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001
-r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001
-r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001
-r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001
-r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001
-r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001
-r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001
-r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001
-r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001
-r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001
-r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001
-r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001
-r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001
-r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001
-r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001
```
This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled.
Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables.
In Ubuntu, the kernel is configured with CONFIG_ EFI_VARS_ PSTORE= m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems.
``` pstore. service - Platform Persistent Storage Archival system/ systemd- pstore. service; enabled; vendor preset: enabled) oryNotEmpty= /sys/fs/ pstore was not met pstore( 8)
$ systemctl status systemd-pstore
○ systemd-
Loaded: loaded (/lib/systemd/
Active: inactive (dead)
Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago
└─ ConditionDirect
Docs: man:systemd-
Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
$ ls -la /sys/fs/pstore 165476562001001 165476562002001 165476562003001 165476562004001 165476562005001 165476562006001 165476562007001 165476562008001 165476562009001 165476562010001 165476562011001 165476562012001 165476562013001 165476562014001 165476562015001 165476589801001 165476589802001 165476589803001 165476589804001 165476589805001 165476589806001 165476589807001 165476589808001 165476589809001 165476589810001 165476589811001 165476589812001 165476589813001 165476589814001 165476589815001
total 0
drwxr-x--- 2 root root 0 Jun 9 09:11 .
drwxr-xr-x 8 root root 0 Jun 9 09:11 ..
-r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-
-r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-
-r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-
```
This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_ EFI_VARS_ PSTORE= y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled.