Some sysctls are ignored on boot

Bug #50093 reported by Colm MacCarthaigh
334
This bug affects 71 people
Affects Status Importance Assigned to Milestone
procps (Ubuntu)
Won't Fix
Medium
Unassigned
systemd (Ubuntu)
Confirmed
Wishlist
Unassigned

Bug Description

Binary package hint: procps

/etc/init.d/procps.sh comes too early in the boot process to apply a lot of sysctl's. As it runs before networking modules are loaded and filesystems are mounted, there are quite a lot of commonly-used sysctl's which are simply ignored on boot and produce errors to the console.

Simply renaming the symlink from S17 to > S40 probably isn't a great solution, as there are probably folk who want and expect some sysctl's to be applied before filesystems are mounted and so on. However, simply ugnoring something as important as sysctl settings isn't really on. Administrators expect the settings in /etc/sysctl.conf to take effect.

One sto-gap solution would be to run sysctl -p twice; once at S17 and once at S41. There may still be some warnings and errors, but everything would be applied. A different, more complex approach might be to re-architect the sysctl configuration into something like;

    /etc/sysctl.d/$modulename

and have the userland module-loading binaries take care of applying them after modules are loaded. Though this may take care of explicitly loaded modules only, I'm not sure.

Incidentally, /etc/sysctl.conf still refers to /etc/networking/options, but hasn't that been deprecated?

Simon Law (sfllaw)
Changed in procps:
importance: Untriaged → Low
status: Unconfirmed → Confirmed
Revision history for this message
Yves Junqueira (yves.junqueira) wrote :

Hi.

After giving some thought on this, I don't think the complex approach you described would work. Entries in /proc/sys may not exist even if the module was loaded.

Take this case:

yv:~# lsmod|grep 1394
eth1394 18212 0
ohci1394 30800 0
ieee1394 86904 3 sbp2,eth1394,ohci1394

yv:~# dmesg|grep 1394
ieee1394: Initialized config rom entry `ip1394'
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[209] MMIO=[e2004000-e20047ff] Max Packet=[2048] IR/IT contexts=[4/8]
ieee1394: Host added: ID:BUS[0-00:1023] GUID[000fea000027c5c2]
eth1394: eth0: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
ieee1394: sbp2: Driver forced to serialize I/O (serialize_io=1)
ieee1394: sbp2: Try serialize_io=0 for better performance

yv:~# ls /proc/sys/net/ipv4/conf/
all default eth1 lo

Notice there is no "eth0" entry. So processing sysctl parameters after loading a module is not enough, at least for network related issues.

Can you describe cases when you believe processing some later sysctl directives are needed?

I believe that using post-up parameters for /etc/network/interfaces is the current supported method. Maybe you could agree that this works for you? In that case, we should reject this bug, I guess.

Thank you

Changed in procps:
assignee: nobody → yves.junqueira
Revision history for this message
Daniel (daniel123+launchpad) wrote :

I'm having this as an issue trying to create network bridge for KVM virtualization. Following instructions here, http://wiki.libvirt.org/page/Networking#Debian.2FUbuntu_Bridging

Libvirt is standard in ubuntu now, and these instructions are referenced elsewhere. Makes the whole thing a bit problematic.

Revision history for this message
jollyroger (jeffrey-crawford) wrote :

This seems pretty old and I'm not sure against which version of Ubuntu it was filed against (Sorry kinda new to this bug system) however to appears that the problem is two fold. I'm using karmic right now which uses upstart. I've had trouble getting the following settings to work:

=={/etc/sysctl.conf}==
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0

kernel.shmmax = 68719476736
kernel.shmall = 4294967296
===

The upstart system had the procps starting after the event "virtual-filesystems" which is triggered by /etc/init/mountall.conf. It appears that due to the fact that many kernel modules are not yet loaded the, for example "bridge.ko", the entries above do not exist yet in sysctl/procfs, so I changed "/etc/init/procps.conf" "start" line to read "start on starting network-interface" This allows the bridge modules to be loaded and and available when procps is allowed to run. Additionally I modified the procps.conf script line to include the "-e" option so that if there are entries that do not exist yet at least the entries that do exist are executed and the incorrect entries are ignored. Seems like that would be good practice anyway, I've attached a copy of the procps.conf file that I've modifed. I didn't think to keep the originals for diff's and don't want to make incorrect diffs so I just uploaded the file as it is now.

Revision history for this message
Drew Hess (dhess) wrote :

I'm having the exact same problem as jollyroger and Daniel, and it was maddening to debug.

This bug's been filed since 2006 and it's still not fixed? Anyone who uses KVM in a bridged configuration is probably affected by it. At least make a note of it in the procps or upstart documentation so that users don't pull their hair out trying to figure out why sysctl.d conf files are being (silently) ignored.

Revision history for this message
Yves Junqueira (yves.junqueira) wrote :

I didn't realize this was assigned to me. I don't have a recent Ubuntu system available so I can't help with this. Sorry.

Changed in procps (Ubuntu):
assignee: Yves Junqueira (yves.junqueira) → nobody
Revision history for this message
Simon Eisenmann (longsleep) wrote :

Guys somebody really should take care about this as this is not fixed and still a problem with natty.

At least please assign to the maintainer of procps service.

Revision history for this message
W3ird_N3rd (w3ird-n3rd) wrote :

Ah, so this is why my vm.laptop_mode=5, vm.dirty_writeback_centisecs=6000 and vm.dirty_expire_centisecs=7000 lines are totally ignored.

Workaround, anyone?

Revision history for this message
david wood (david-wood) wrote :

All documentation on the net referring to changing certain settings in /etc/sysctl.conf such as net.ipv4.netfilter.ip_conntrack_max is wrong for Ubuntu. In addition, workarounds suggesting that ordering of module load vs. sysctl.conf execution can be helped by i.e. putting ip_conntrack into /etc/modules also do not work.

I'm sure on some level this is Low priority - aka bury for 5 years and never look at it again - and I can appreciate that this is not a simple problem to properly solve. But left as is, this is just another mine laid in the field for sysadmins foolish enough to use Ubuntu Server. I respectfully suggest that it might be useful to make multiple attempts to run sysctl -p at various milestones during the boot process.

Revision history for this message
Andreas Ntaflos (daff) wrote :

A possible workaround is putting "sysctl -p /etc/sysctl.conf" in /etc/rc.local, which is ridiculous and doesn't even solve the problem that networking and bridges are started *before* the sysctl settings are applied.

It is even more ridiculous that this bug hasn't had any attention from anyone except affected users in over five years. *Every* piece of documentation regarding any kind of sysctl setting refers to /etc/sysctl.conf as the place to put everything. And for over five years there has been a really good chance that most of /etc/sysctl.conf gets ignored because Ubuntu's boot process apparently doesn't know how when and often to read and apply the settings in there. This isn't just an upstart problem since upstart wasn't even around in 2006.

Bugs like this make it really difficult to recommend Ubuntu as a server operating system and I wonder if Canonical really use Ubuntu as their server OS. If they did it seems this bug would have been fixed years ago.

Revision history for this message
John DeStefano (deesto) wrote :

Just ran into this issue myself and was _quite_ surprised to see how long this bug has been reported yet unresolved.

Revision history for this message
philip550c (philip550c) wrote :

This will probably be a bug in Ubuntu 20.04 aaron aardvark

Revision history for this message
Gary Richards (ashak) wrote :

We've had this issue for ages. Thought we would investigate it more now since it recently caused us a bunch of pain due to some sysctl settings not being loaded.

It's madness that no one has looked at this.

Revision history for this message
Doka (doka-wepoca) wrote :

It seems to me it is a Debian Squeeze bug, see for example here, also for a workaround:

http://wiki.debian.org/BridgeNetworkConnections

Strange enough...

Revision history for this message
Roman Ovchinnikov (coolcold) wrote :

bumping +3 years , still no good solution

Revision history for this message
Kimia (operaciones-0) wrote :

This is still happening.. Debian7.8

Revision history for this message
Brian Martin (7-launchpad-9) wrote :

This is still a problem in 16.04 LTS/xenial. I lost a whole workday chasing this down after an upgrade. I don't think it's a duplicate of 771372, as the above discussion indicates there is no "right" place to run procps, and 771372 works on that presumption. Instead, the start-up process needs to be reworked, or at least the network-related settings need to be reassigned to the network start-up process instead of living in procps. With the increased use of bridges (KVM, LXC, etc.), we should have a smooth start-up process for bridges and bridge-related settings.

I confess I don't understand all the complexities others apparently see. We need someone that really understands the relevant start-up processes to architect a good solution. What can we do to get a little attention on this problem?

Revision history for this message
Dave Vree (hdave) wrote :

bumping another year -- now 11 years old and still a real problem

Paul Gear (paulgear)
summary: - Some sysctl's are ignored on boot
+ Some sysctls are ignored on boot
Revision history for this message
Tomasz Konefal (twkonefal-j) wrote :

A workaround for this that we're using now is to put the sysctl processing into crontab after reboot.

e.g.
@reboot root sleep 5 && sysctl --system

Maybe one day someone will fix this bug :)

Revision history for this message
Stanley Sisneros (stanley-sisneros) wrote :

In Ubuntu 16.04 I made this work by adding the following to crontab -e

@reboot sleep 15; /sbin/sysctl -p

information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
Zsolt Ero (hyperknot) wrote :

I feel it's important to add that the very last comment (Stanley Sisneros (stanley-sisneros) wrote on 2018-05-30) is wrong, and the right workaround is by Tomasz Konefal (twkonefal-j) wrote on 2018-05-07.

That is, add this to crontab:
@reboot root sleep 5 && sysctl --system

Revision history for this message
Zsolt Ero (hyperknot) wrote :

Actually, the line which works without specifying a PATH in a cron.d file is the following:

@reboot root /bin/sleep 5 && /sbin/sysctl --system

Revision history for this message
Радогор (igor0386) wrote :

I have the same problem in Debian 9 Stretch, net.netfilter.nf_conntrack_max and net.nf_conntrack_max does not apply after reboot. The pure solution is load module nf_conntrack before sysctl. You can do this with the command:
echo "nf_conntrack" >> /etc/modules

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

I have added a file in /etc/sysctl.d/98-* setting up NFS sysctl parameters and they were not loaded by default until I have added:

nfsd

to /etc/modules, re-generated the initramfs and rebooted, checking it again, when it worked.

Changed in procps (Ubuntu):
assignee: nobody → Rafael David Tinoco (rafaeldtinoco)
importance: Low → Medium
Revision history for this message
Martin Streďanský (stredini) wrote :

Still no fix? I have just ran into the same problem...

Changed in procps (Ubuntu):
status: Confirmed → Triaged
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
status: Triaged → Confirmed
Revision history for this message
Drew Freiberger (afreiberger) wrote :

Still seeing this in bionic 18.04.3 with the following kernel and procps package.

ii procps 2:3.3.12-3ubuntu1.2 amd64 /proc file system utilities

kernel 5.4.0-62-generic

Revision history for this message
Dan Streetman (ddstreet) wrote :

As procps has been replaced by systemd in recent releases (at least the sysctl service part), I'm switching the target package to systemd.

Also, I'm changing this to wishlist as this isn't actually a bug; from the sysctl.d manpage:

"Many sysctl parameters only become available when certain kernel
modules are loaded. Modules are usually loaded on demand, e.g.
when certain hardware is plugged in or network brought up. This
means that systemd-sysctl.service(8) which runs during early boot
will not configure such parameters if they become available after
it has run. To set such parameters, it is recommended to add an
udev(7) rule to set those parameters when they become available.
Alternatively, a slightly simpler and less efficient option is
to add the module to modules-load.d(5), causing it to be loaded
statically before sysctl settings are applied (see example below)."

Changed in systemd (Ubuntu):
importance: Undecided → Wishlist
Changed in procps (Ubuntu):
status: Confirmed → Won't Fix
Revision history for this message
Dan Streetman (ddstreet) wrote :

Just to reiterate, I'm only marking procps as 'wontfix' because the sysctl service used in all current releases comes from systemd, *not* the procps package, so there is nothing to 'fix' in the procps package.

Also, before people get upset, I've only marked this as Wishlist because the sysctl.d docs clearly explain this is a known issue with clearly documented ways to work around it. That doesn't mean i don't think this should be fixed; I absolutely do agree it should be fixed to properly apply all sysctl settings at boot, including for loadable modules loaded after boot. That will require upstream systemd work to implement, however, before backporting into Ubuntu.

Revision history for this message
fuckyouopensource (ugh123) wrote :

"sysctl.d docs clearly explain this is a known issue with clearly documented ways to work around it"

Try googling "how to disable ipv6 in ubuntu". The results suggest that the word "clearly" is misused.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.