Vivid + bootchart quietly fails

Bug #1458170 reported by David Favor on 2015-05-23
46
This bug affects 10 people
Affects Status Importance Assigned to Milestone
bootchart (Ubuntu)
Medium
Unassigned

Bug Description

Nothing shown via journalctl -k -b | grep bootchart after boot.

/var/log/bootchart does exist + is empty after boot.

net8-rmt# uname -a
Linux ns515383.ip-167-114-159.net 3.19.0-18-generic #18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

net8-rmt# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid

net8-rmt# pkg-list | grep bootchart
bootchart 0.90.2-8ubuntu2
pybootchartgui 0+r141-0ubuntu5

David Favor (davidfavor) wrote :

I'd have thought /etc/default/grub should have some directive about bootchart + there is none.

Maybe this is the problem.

David Favor (davidfavor) wrote :

Removed + Reinstalled...

Looks like all's well...

net8-rmt# apt-get install bootchart
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  bootchart
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/11.2 kB of archives.
After this operation, 98.3 kB of additional disk space will be used.
Selecting previously unselected package bootchart.
(Reading database ... 84900 files and directories currently installed.)
Preparing to unpack .../bootchart_0.90.2-8ubuntu2_amd64.deb ...
Unpacking bootchart (0.90.2-8ubuntu2) ...
Processing triggers for systemd (219-7ubuntu5) ...
Setting up bootchart (0.90.2-8ubuntu2) ...
update-initramfs: Generating /boot/initrd.img-3.19.0-18-generic
update-initramfs: Generating /boot/initrd.img-3.19.0-16-generic
Processing triggers for systemd (219-7ubuntu5) ...

net8-rmt# sys-ll | grep bootchart
  dev-.bootchart-proc.mount loaded active mounted /dev/.bootchart/proc

Reboot + /var/log/bootchart still empty.

David Favor (davidfavor) wrote :

This looks suspicious...

net8-rmt# service bootstart start
Failed to start bootstart.service: Unit bootstart.service failed to load: No such file or directory.

net8-rmt# systemctl restart bootstart.service
Failed to restart bootstart.service: Unit bootstart.service failed to load: No such file or directory.

Ah...

net8-rmt# sys-ll | grep bootchart
  dev-.bootchart-proc.mount loaded active mounted /dev/.bootchart/proc

Shows there is no bootstart.service available, so likely this is the problem.

David Favor (davidfavor) wrote :

This should be assigned to bsdtools as the package.

affects: lxc (Ubuntu) → bootchart (Ubuntu)
David Favor (davidfavor) wrote :

Er, package bootchart, not bsdtools.

David Favor (davidfavor) wrote :

Be appreciated if someone can update this bug with a workaround.

Thanks.

David Favor (davidfavor) wrote :

Maybe problem is bootchart is filling up /dev.

udev 16G 16G 0 100.0 [#####] /dev

net4-dev# find /dev/.bootchart/log -ls
 14348 0 drwxr-xr-x 2 root root 100 Jun 11 18:42 /dev/.bootchart/log
 12299 15094408 -rw-r--r-- 1 root root 15456673792 Jun 11 22:37 /dev/.bootchart/log/proc_ps.log
 12298 566784 -rw-r--r-- 1 root root 580386797 Jun 11 22:37 /dev/.bootchart/log/proc_diskstats.log
 12297 781824 -rw-r--r-- 1 root root 800587747 Jun 11 22:37 /dev/.bootchart/log/proc_stat.log

David Favor (davidfavor) wrote :

apt-get --purge remove bootchart leaves all the /dev cruft/residue also.

bootchart seems faulty to me in 15.04.

At first I only noticed that there was a long delay at shutdown, IIRC having appeared after the upgrade to "vivid". After much painful investigation, I could trace it to swapoff.

After boot, the swap usage always kept growing, maxing out at approx. 2 GiB -- despite there was no memory pressure (1.6 of 4 GiB RAM used, default swappiness of 60). Then I read somewhere that RAM filesystems like tmpfs filesystems tend to be swapped out quickly. df showed the culprit:

~$ df -h /dev
Filesystem Size Used Avail Use% Mounted on
udev 2,0G 2,0G 0 100% /dev
~$ du -sh /dev/.bootchart/log
2,0G /dev/.bootchart/log
~$

Wondering what bootchart was doing, I found the last successful logs in /var/log/bootchart with PNGs of "trusty". From "utopic" there were only TGZs; nothing of "vivid" there.

So it seems to me too that bootchart is failing silently in vivid. It seems the collector process is not terminated properly and the logs aren't saved in the right place afterwards.

Wouldn't it make sense to deprecate the package and at least put a note in the README or package description? Especially since systemd, which is used in "vivid" by default, can do bootcharting on its own.

Also note that a few days after upgrading to "vivid", I manually switched back from systemd to upstart; bootchart fails nonetheless.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in bootchart (Ubuntu):
status: New → Confirmed
David Favor (davidfavor) wrote :

Be great if you added the steps you used to revert back to upstart.

I've concluded that all Vivid instability - hung reboots, MySQL start/stop silently failing, same with Apache + bootchart - all relate to the braindeadedness of systemd.

I've started using the latest mainline ppa kernel with Utopic + my systems are rock solid again.

I also notice any bugs I open related to systemd are just ignored... likely because systemd is just so broken no one feels excited to debug any bugs related to it's nefariousness.

David Favor (davidfavor) wrote :

Braindeadedness of systemd is just my opinion.

Ilya Bizyaev (bizyaev-game) wrote :

Same problem - no bootcharts generated.

tags: added: vivid xenial
Changed in bootchart (Ubuntu):
importance: Undecided → Medium
David Favor (davidfavor) wrote :

I believe the problem relates to packaging + documentation.

All bootchart packages appear to be deprecated, in lieu of systemd handling the bootchart function now... so... likely good to...

1) remove all bootchart packages, as they create confusion + are NOPs (produce now effect)

2) update bootchart related documentation, explaining this deprecation

3) update documentation about how to correct generate bootchart output

Regards #3, suggestions like "install bootchart" or "journalctl --full | grep -i bootchart" are insufficient, as no bootchart data is ever generated.

Please update this ticket with correct documentation about how to...

1) setup systemd to generate bootchart data

2) how to view data (where data lives)

Thanks.

Kim Dahlgren (ping-the-one) wrote :

I have problem rebooting, takes about an hour before systemd outputs the error message "unable to unmount /dev/.bootchart/proc" which just flashes by.

From Ubuntu 16.04

wc -l proc_ps.log
16347205 proc_ps.log

free -m
              total used free shared buff/cache available
Mem: 7868 4763 382 2257 2722 542
Swap: 8071 4419 3652

du -h proc_ps.log
3,7G proc_ps.log

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers