lxcfs: update the 'btime' field in /proc/stat to reflect guest boot time not host

[Jason Baron]

The 'btime' field in /proc/stat reflects the host boot time and not the time that the guest container has started.

[Serge Hallyn (serge-hallyn)]

Thanks I've created https://github.com/lxc/lxcfs/issues/164 to track this.

[Jason Baron (jibaron)]

I've created a pull request to address this issue:
https://github.com/lxc/lxcfs/pull/169

Thanks.

[Evgeni Golov (evgeni)]

This is fixed in LXCFS 2.0.7

[gozdal (gozdal)]

This introduces a different problem: process creation time is counted as offset from btime:

(22) starttime %llu
          The time the process started after system boot. In
          kernels before Linux 2.6, this value was expressed
          in jiffies. Since Linux 2.6, the value is expressed
          in clock ticks (divide by sysconf(_SC_CLK_TCK)).

(http://man7.org/linux/man-pages/man5/proc.5.html)

This means that because starttime is not virtualized, all the processes within the LXC container appear as time travellers from the future.

[Christian Brauner (cbrauner)]

On Mon, Jun 19, 2017 at 01:04:38PM -0000, gozdal wrote:
> This introduces a different problem: process creation time is counted as
> offset from btime:
>
> (22) starttime %llu
> The time the process started after system boot. In
> kernels before Linux 2.6, this value was expressed
> in jiffies. Since Linux 2.6, the value is expressed
> in clock ticks (divide by sysconf(_SC_CLK_TCK)).

Yes, we reverted that patch since we consider STTIME to be more important.

>
> (http://man7.org/linux/man-pages/man5/proc.5.html)
>
> This means that because starttime is not virtualized, all the processes
> within the LXC container appear as time travellers from the future.
>
> --
> You received this bug notification because you are a member of Ubuntu
> containers team, which is subscribed to lxcfs in Ubuntu.
> Matching subscriptions: lxcfs
> https://bugs.launchpad.net/bugs/1654310
>
> Title:
> lxcfs: update the 'btime' field in /proc/stat to reflect guest boot
> time not host
>
> Status in lxcfs package in Ubuntu:
> Fix Released
>
> Bug description:
> The 'btime' field in /proc/stat reflects the host boot time and not
> the time that the guest container has started.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1654310/+subscriptions

[Jason Baron (jibaron)]

Ok, I guess we could intercept the /proc/<pid>/stat read() call and overwrite the starttime there with the correct value? That is, add the starttime from /proc/<pid>/stat to the host boot time and then subtract that from the guest boot time.

[Christian Brauner (cbrauner)]

On Mon, Jun 19, 2017 at 04:28:07PM -0000, Jason Baron wrote:
> Ok, I guess we could intercept the /proc/<pid>/stat read() call and
> overwrite the starttime there with the correct value? That is, add the
> starttime from /proc/<pid>/stat to the host boot time and then subtract
> that from the guest boot time.

We can't really do that since this is equivalent to over-mounting /proc itself.
Instead, we have reverted the "btime" virtualization patch for now and we'll
likely include this in the next round of SRUs.

>
> --
> You received this bug notification because you are a member of Ubuntu
> containers team, which is subscribed to lxcfs in Ubuntu.
> Matching subscriptions: lxcfs
> https://bugs.launchpad.net/bugs/1654310
>
> Title:
> lxcfs: update the 'btime' field in /proc/stat to reflect guest boot
> time not host
>
> Status in lxcfs package in Ubuntu:
> Fix Released
>
> Bug description:
> The 'btime' field in /proc/stat reflects the host boot time and not
> the time that the guest container has started.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1654310/+subscriptions

[gozdal (gozdal)]

I believe that packages for several distributions (esp. Xenial) still contain this now-revered patch. Would it be possible to release packages with btime virtualization reverted?

[Serge Hallyn (serge-hallyn)]

@gozdal,

would you be willing to add a SRU justification as per section 3.1 of https://wiki.ubuntu.com/StableReleaseUpdates to the description of this bug?

[Serge Hallyn (serge-hallyn)]

I've pushed a package reverting that field to xenial-proposed - it will await approval there.

I've set the priority to high as this seems to be affecting other software adversely.

[gozdal (gozdal)]

[Impact]

Processes within LXC container are seen as created in the future.

[Test Case]

1. create an LXC container on a host which has been up for untrivial amount of time (say 1 hour)
2. look at "ps aux" output in START column - all the process are seen as started in the future

[Regression Potential]

Software comparing process start time to any other time may fail in unexpected ways. As an example some software compare process start time to PID file timestamp on disk and disregard old PID files. In LXC this code will disregard all PID files.

[Stéphane Graber (stgraber)]

Hello Jason, or anyone else affected,

Accepted lxcfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stéphane Graber (stgraber)]

Hello Jason, or anyone else affected,

Accepted lxcfs into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu1~16.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-yakkety to verification-done-yakkety.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-yakkety. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stéphane Graber (stgraber)]

Hello Jason, or anyone else affected,

Accepted lxcfs into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxcfs/2.0.7-0ubuntu1~17.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty.If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Candler (b-candler)]

After upgrading to lxcfs/xenial-proposed (2.0.7-0ubuntu1~16.04.2), the problem is fixed for me

Inside a container:

root@apt-cacher:~# ps auxwww | grep apt
apt-cac+ 309 0.0 0.7 653476 7128 ? Ssl Jul08 1:02 /usr/sbin/apt-cacher-ng SocketPath=/run/apt-cacher-ng/socket -c /etc/apt-cacher-ng ForeGround=1
root 1922 0.0 0.1 14620 1016 ? S+ 08:04 0:00 grep --color=auto apt
root@apt-cacher:~# date
Tue Jul 11 08:04:09 UTC 2017

- start time of apt-cacher-ng is correct (in the past)
- start time of 'grep' process is correct (current time)

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 2.0.7-0ubuntu1~16.04.2

---------------
lxcfs (2.0.7-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * Cherrypick patch reverting the virtualization of bstat field (LP: #1654310)

 -- Serge Hallyn <email address hidden> Tue, 04 Jul 2017 10:43:18 -0500

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for lxcfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 2.0.7-0ubuntu1~16.10.2

---------------
lxcfs (2.0.7-0ubuntu1~16.10.2) yakkety-proposed; urgency=medium

  * Cherrypick patch reverting the virtualization of bstat field (LP: #1654310)

 -- Serge Hallyn <email address hidden> Tue, 04 Jul 2017 13:29:14 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 2.0.7-0ubuntu1~17.04.2

---------------
lxcfs (2.0.7-0ubuntu1~17.04.2) zesty-proposed; urgency=medium

  * Cherrypick patch reverting the virtualization of bstat field (LP: #1654310)

 -- Serge Hallyn <email address hidden> Tue, 04 Jul 2017 13:29:40 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 2.0.8-1ubuntu1

---------------
lxcfs (2.0.8-1ubuntu1) bionic; urgency=medium

  * Re-sync with Debian, remaining delta (for backports):
    - Include upstart jobs on older Ubuntu releases
    - Build-depend on dh-systemd with older debootstrap

 -- Stéphane Graber <email address hidden> Tue, 31 Oct 2017 18:40:42 -0400