[SRU] fs quota calculation is wrong

[Thomas Sommer]

[Impact]

Dovecot users using the fs quota plugin see wrong quota calculation (incorrect overquota situation).

[Test Case]
On host system:
---------------
$ multipass launch 18.04 --name ubuntu-bionic
$ multipass exec ubuntu-bionic -- bash

In vm:
------
$ sudo passwd ubuntu # set to 'ubuntu'

# Enable quota support on root partition
$ sudo apt-get install linux-image-extra-virtual
$ sudo nano /etc/fstab
  LABEL=cloudimg-rootfs / ext4 usrquota,grpquota 0 0
$ sudo mount -o remount /
$ sudo cat /proc/mounts | grep ' / '
  /dev/sda1 / ext4 rw,relatime,quota,usrquota,grpquota,data=ordered 0 0

# Set a quota
$ sudo apt-get install quota
$ sudo quotacheck -ugm /
$ sudo quotaon -v /
  /dev/sda1 [/]: group quotas turned on
  /dev/sda1 [/]: user quotas turned on
$ sudo edquota -u ubuntu
  Disk quotas for user ubuntu (uid 1000):
    Filesystem blocks soft hard inodes soft hard
    /dev/sda1 36 0 0 12 0 0

# Set these to some arbitrary values:
  Filesystem blocks soft hard inodes soft hard
  /dev/sda1 36 10 20 12 0 0
$ sudo quota -vs ubuntu
  Disk quotas for user ubuntu (uid 1000):
    Filesystem space quota limit grace files quota limit grace
    /dev/sda1 48K* 10K 20K 7days 14 0 0
$ sudo quota -vs ubuntu
  Disk quotas for user ubuntu (uid 1000):
   Filesystem space quota limit grace files quota limit grace
    /dev/sda1 48K* 10K 20K 7days 14 0 0
$ sudo setquota -u ubuntu 200M 220M 0 0 /
$ sudo quota -v ubuntu
  Disk quotas for user ubuntu (uid 1000):
   Filesystem blocks quota limit grace files quota limit grace
    /dev/sda1 52 204800 225280 15 0 0

# Install dovecot and configure with the quota plugin
$ sudo apt-get install dovecot-core dovecot-imapd
$ sudo sed -i 's/^#mail_plugins = /mail_plugins = quota/' /etc/dovecot/conf.d/10-mail.conf
$ sudo sed -i 's/ #mail_plugins = / mail_plugins = quota imap_quota #/' /etc/dovecot/conf.d/20-imap.conf
$ sudo sed -i 's/ #quota = fs:User quota/ quota = fs:User quota/' /etc/dovecot/conf.d/90-quota.conf
$ sudo service dovecot restart
$ doveadm quota get -u ubuntu
  Quota name Type Value Limit %
  User quota STORAGE 53248 204800 26

# Connect via imap
$ openssl s_client -connect localhost:993
  * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot (Ubuntu) ready.
? LOGIN ubuntu ubuntu
? OK [... QUOTA] Logged in
? GETQUOTAROOT INBOX
* QUOTAROOT INBOX "User quota"
* QUOTA "User quota" (STORAGE 53248 204800)
? OK Getquotaroot completed (0.001 + 0.000 secs).

[Regression Potential]

 * only users using the fs quota plugin are affected

[Original Report]
Packages:
dovecot 2.2.33.2-1-ubuntu4.1
ubuntu 18.04.1 LTS

The quota is erroneously calculated with the fs quota plugin.
It was working fine on ubuntu 16.04 with the same config.

 * Prerequisite: user quota configured on filesystem where dovecot stores your email (e.g. /home partition)

 * configure dovecot fs quota plugin.

Quota configuration:
plugin {
   quota = fs:User quota
}

 * use doveadm command to verify the result:
   $doveadm quota get -u user
   The result is shown in kb and as a percentage value

Here's the IMAP Session:

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN AUTH=LOGIN] Dovecot (Ubuntu) ready.
? LOGIN user password
? OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE QUOTA] Logged in
? GETQUOTAROOT INBOX
* QUOTAROOT INBOX "User quota"
* QUOTA "User quota" (STORAGE 2775445504 4096001)
? OK Getquotaroot completed (0.001 + 0.000 secs).

Storage should actually be around 2775445 KB.
The quota is correct from my ext3 filesystem quota of 4096001 KB.

Output of repquota:
*** Report for user quotas on device /dev/sdc
Block grace time: 7days; Inode grace time: 7days
                        Block limits File limits
User used soft hard grace used soft hard grace
----------------------------------------------------------------------
user -- 2710396 4096001 4194304 65743 0 0

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Unfortunately I don't see anything I can do about this in Ubuntu right now. Can you reproduce this problem with a build direct from upstream sources (of this version - 2.2.33.2), without packaging? If so, then filing a bug upstream would be appropriate. If it does not, please let us know here with full steps to reproduce on a fresh Ubuntu system and we might be able to figure out if packaging is breaking it somehow.

[Thomas Sommer (flightsupport)]

Somebody already reported it to the dovecot mailing list:
https://www.dovecot.org/list/dovecot/2017-June/108423.html

It seems it's related to a glibc change.
Unfortunately there's no resolution.

[Thomas Sommer (flightsupport)]

I think I found the problem when looking at the source which is related to glibc.
The fs quota plugin is not compatible with the shipped glibc.

Current ubuntu packaged dovecot version v2.2.33:
File: core/src/plugins/quota/quota-fs.c
https://github.com/dovecot/core/blob/release-2.2.33/src/plugins/quota/quota-fs.c

/* Older sys/quota.h doesn't define _LINUX_QUOTA_VERSION at all, which means
   it supports only v1 quota */
#ifndef _LINUX_QUOTA_VERSION
# define _LINUX_QUOTA_VERSION 1
#endif

Current dovecot release v2.3.3
File: core/src/plugins/quota/quota-fs.c
https://github.com/dovecot/core/blob/release-2.3.3/src/plugins/quota/quota-fs.c

/* Very old sys/quota.h doesn't define _LINUX_QUOTA_VERSION at all, which means
   it supports only v1 quota. However, new sys/quota.h (glibc 2.25) removes
   support for v1 entirely and again it doesn't define it. I guess we can just
   assume v2 now, and if someone still wants v1 support they can add
   -D_LINUX_QUOTA_VERSION=1 to CFLAGS. */
#ifndef _LINUX_QUOTA_VERSION
# define _LINUX_QUOTA_VERSION 2
#endif

Ubuntu 1804 glibc is at version 2.27.

I compiled dovecot core release 2.3.3 from source.
Result: The fs quota plugin works as expected.

Any chance the package is upgraded to version 2.3.3?

[Robie Basak (racb)]

On Sun, Oct 07, 2018 at 08:50:21PM -0000, Thomas Sommer wrote:
> Any chance the package is upgraded to version 2.3.3?

Not for 18.04. Please see https://wiki.ubuntu.com/StableReleaseUpdates
for rationale and policy. The easiest way to fix the quota plugin on
18.04 is to find the minimal patch that fixes it - not a wholesale
version update that could break/upset other users who expect stability.

[Thomas Sommer (flightsupport)]

Are you able to test the attached patch?
I wasn't able to build the package on my current system.
Or I just don't know how to do it. With apt source dovecot-core I got the sources for dovecot 2.2.22, but the installed version is 2.2.33.2

[Robie Basak (racb)]

I uploaded a build to my experimental PPA at: https://launchpad.net/~racb/+archive/ubuntu/experimental/

You should be able to try it from there, assuming the build succeeds.

You can use "pull-lp-source" (from ubuntu-dev-tools) to grab the Ubuntu sources for any given package and version.

Sorry it's currently rather arcane to rebuild something from Ubuntu with just a patch applied. It should be easier, and we're working on it. It's still experimental, but you can see a git view of the sources at: https://code.launchpad.net/ubuntu/+source/dovecot

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "quota-fs.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Thomas Sommer (flightsupport)]

The patch seems to work. Dovecot now displays the correct quota.
Also the command doveadm quota get -u user shows correct values.
What I'm unable to verify is the quota-status-service for postfix.

I have it configured in dovecot.conf as follows:
plugin {
    quota_grace = 10%%
    # 10% is the default
    quota_status_success = DUNNO
    quota_status_nouser = DUNNO
    quota_status_overquota = "552 5.2.2 Mailbox is full"
}

service quota-status {
    executable = quota-status -p postfix
    inet_listener {
        port = 8881
    }
    client_limit = 1
}

When I test it, while being clearly over quota, I get the following:

$doveadm quota get -u user
Quota name Type Value Limit %
User quota STORAGE 3901636 3300000 118

$printf "recipient=user\nsize=1230444\n\n" | nc -q1 localhost 8881
action=DUNNO

In my opinion it should return "552 5.2.2 Mailbox is full" instead of DUNNO.

Maybe somebody is able to test this?

[Thomas Sommer (flightsupport)]

I was away for a while, but in the meantime the patched dovecot was running all the time without any issues.
I tried to replicate the service quota-status on a freshly installed ubuntu 16.04.5. I get the same result of DUNNO instead of Mailbox full. So no regression here.

May I suggest to release a new dovecot-core package with the patch?

[Thomas Sommer (flightsupport)]

~racb what do you think? Could you release a new version with the patch included?

[Robie Basak (racb)]

Sorry I missed your previous comment. Please could you see https://wiki.ubuntu.com/StableReleaseUpdates#Procedure and handle as much of that as you can? I'll be happy to help with with the rest. Please could you also verify that this bug is fixed in Ubuntu 18.10 (Cosmic)?

[Thomas Sommer (flightsupport)]

I've added the SRU Bug Template. Could you take it from here?

[Andreas Hasenack (ahasenack)]

https://github.com/dovecot/core/commit/39eefc53d06cf9790c6e3180a67b68e89eb71313#diff-e72982e5f369f46babc0f4b79df68878 is the upstream commit corresponding to the attached patch.

I see that this fix is present in the Ubuntu Disco and Cosmic packages, so I'm marking the main task of this bug as fix released.

[Thomas Sommer (flightsupport)]

But should still be backported to bionic IMHO.

[Robie Basak (racb)]

Yes, it's on our list to do soon.

[Nick Maynard (nick-maynard)]

Thank you - we have just upgraded our mail system from xenial to bionic, and unexpectedly hit this.

It seems it *might* have other side effects, like breaking indexing (checks quota before writing files) and potentially writing to Sent and Drafts (also, checking quota).

High priority for us - a backport to bionic would be highly appreciated.

[Bryce Harrington (bryce)]

This issue is not reproducing on bionic for me. I've fleshed out the test case in the bug description with a more precise series of steps to set things up in a multipass VM, but the quota values I'm seeing appear to be correct. I'm guessing there's either some additional configuration step, or the issue has been fixed elsewhere (maybe in the kernel?)

Please see if you are still able to reproduce this bug in your environment, and if so please review the test case in the bug report description and identify what additional step or configuration may be missing.

[Bryce Harrington (bryce)]

I'll add that the identified patch still applies to bionic's dovecot, so it doesn't appear that dovecot itself has already been fixed; if the issue is resolved the fix could be in the kernel or elsewhere.

I've prepared a PPA with a dovecot package containing the patch, if anyone would like to test it:

https://launchpad.net/~bryce/+archive/ubuntu/dovecot-sru-lp1795919

Before submitting the SRU, I'd like to have verification that the issue still exists, and that this PPA resolves it. I'd like to be able to do this myself, but if someone else is able to verify it that would be a big help.

[Bryce Harrington (bryce)]

I'm still unable to reproduce the originally reported problem on bionic, with a fresh container, following the steps I outlined in [Test Case].

Given earlier comments in this bug, perhaps there was a glibc fix to address the root cause. In any case, given no one has chimed in I'll assume the issue resolved now and close out the bug report.

If you're still reproducing this issue on a currently updated bionic, feel free to reopen, and please indicate exact steps to reproduce.