[Focal] zsys still offers auto snapshotting when reach near full disk space
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
zsys (Ubuntu) |
Fix Released
|
High
|
Jean-Baptiste Lallement | ||
Focal |
Fix Released
|
High
|
Jean-Baptiste Lallement |
Bug Description
[Impact]
* ZSys is still auto snapshooting and allow for state save when reaching full disk space, erroring out.
* We now limit it to 80% of full disk space (either bpool or rpool - which can be tweaked in ZSys configuration file), so that ZSys itself is not responsible for ZFS performance degradation which is considered when disk reach 85% capacity.
* This is covered by an extensive testsuite, for either bpool or rpool or both, both for automated and manual tests.
[Test Case]
1. Ensure the disk has reached more than 80% capacity
2. Run zsysctl save -> it should fail due to max capacity
3. Run sudo zsysctl save --system -> it should fail due to max capacity
4. Run sudo zsysctl save <manual_name> -> it should fail due to max capacity
extra:
1. Try the above on a disk which hasn’t reached this capacity -> the save should work.
[Regression Potential]
* This code impacts automated and manual save and existing tests hasn’t been impacted.
* New tests with mock functionality for simulating disk capacity has been added.
------
I had been running Focal since the March 4th and installed it via the daily ISO at that time. I opted for ZFS root during the installation process. It was mostly the reason why I went with Focal in the first place.
I ran my frequent, almost daily, apt upgrades to get the latest packages and was happy to see a lot of fixes going in during that period until the release date.
This frequent apt upgrade cadence created a snapshot in rpool/bpool every time. This feature proved useful for me since I was able to restore to a usable state using this once in the past. I was aware of the rpool snapshots, so I was cleaning them out as they built up, but I did not consider the bpool snapshots.
Eventually the small 2GB bpool will fill up with /boot changes and this did happen to me on 30-APR-2020 when I was attempting to upgrade to the latest kernel release. It gave me a cryptic error when trying to run update-initramfs, so I did a G search on the error and it pointed to a disk full situation. When I checked, I saw that only 14MB was free on bpool. I then removed all snapshots that didn't have a coordinating snapshot in rpool and that freed up about 1.2GB in bpool.
I will upload the /var/log/
I will leave my laptop in the current state in case you need more information. I will probably send snapshot lists from both bpool and rpool on that laptop. (Once again, I did clean up rpool snapshots manually on the laptop).
Let me know if you need more information, log files, command outputs.
Great work on this package so far!
Chris
Changed in zsys (Ubuntu): | |
assignee: | nobody → Jean-Baptiste Lallement (jibel) |
Changed in zsys (Ubuntu Focal): | |
assignee: | nobody → Jean-Baptiste Lallement (jibel) |
summary: |
- [Focal] zsys does not clear out old snapshots, potentially filling up - bpool + [Focal] zsys still offer auto snapshotting when reach near full disk + space |
description: | updated |
summary: |
- [Focal] zsys still offer auto snapshotting when reach near full disk + [Focal] zsys still offers auto snapshotting when reach near full disk space |
Thanks for your report. It's something we discussed yesterday with Didier and we have to revisit the way we handle bpool ( would a bigger pool be enough for example or would it just push the wall further).
Could you please run
apport-collect 1876334
It'll collect useful information to help us sort out this issue.
Thanks.