/etc/sysctl.conf values do not apply to haproxy namespace after a failover

[Denis Ipatov]

We are deploying a new environment with MOS 6.1.
The Fuel master has been upgraded with "yum upgrade" which pulled in some dependencies.
After the upgrade we ran:

"fuel release --sync-deployment-tasks --dir /etc/puppet/"

According to a bug filed in https://bugs.launchpad.net/fuel/+bug/1475530

Due to a bug in the Broadcom NIC we use in our environment we also needed to upgrade the kernel from 3.13 (trusty default) to 3.19 (vivid-lts-backport).

When performing HA tests on the controllers and during the install the /etc/sysctl.conf values don't get applied correctly and as a result haproxy fails to start.

I suspect this may be a change in the kernel behavior so my problem is to understand when in the boot process the haproxy namespace is created and if sysctl is ran inside the namespace or it is expected to be inherited from the global namespace.

[Denis Ipatov (dipatov)]

I've managed to track down the problem on my own and patch the appropriate scripts.
Please find attached the used scripts.
The required modification is to run sysctl -p /etc/sysctl.conf in the IP namespace to set the necessary values (especially the net.ipv4.ip_nonlocal_bind = 1 which allows haproxy to function correctly).

These are part of the fuel library.

I would like to request a build of the fuel-library ubuntu packages (fuel-ha-utils, fuel-misc and fuel-rabbit-fence) which take into account the corrections required for the new kernel version.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/229367

[Stanislaw Bogatkin (sbogatkin)]

Set "Won't fix" for 7.0 cause we don't backport medium bugs.

[Vladimir Kuklin (vkuklin)]

"The required modification is to run sysctl -p /etc/sysctl.conf in the IP namespace to set the necessary values (especially the net.ipv4.ip_nonlocal_bind = 1 which allows haproxy to function correctly)."

nonlocal_bind cannot be set within the networking namespace in 3.13. It could have happened that 3.19 allows you to set it per-namespace, but we need proof for that.

Last time I saw this issue it was due to sysctl binary removed from the node completely.

[Stanislaw Bogatkin (sbogatkin)]

Okay, I'll try your assumption tomorrow. Denis, could you tell something about Vova's comment?

[Denis Ipatov (dipatov)]

Vladimir, "sysctl" was not deleted. And the customer is using kernel 3.19. What do you need proof?
We can ask configuration settings or log files.

[Vladimir Kuklin (vkuklin)]

Denis

For 3.13 kernel we nonlocal_bind cannot be altered in network namespace.
This may not be the case for 3.19 kernel though. If 3.19 can have nonlocal_bind per namespace, please provide proof for that. Otherwise the bugfix seems to be incorrect.

[Vitaly Sedelnik (vsedelnik)]

Targeted to 7.0-mu-1 - this is customer-found bug and we do want to backport the fix to 7.0

[Stanislaw Bogatkin (sbogatkin)]

Vladimir, let's look on this situation. Developer stuck into bug. Tried some solutions and found this one which helped. Fix was created. I see several sources where we could look the documentation about this fix gut:

1. Sysctl kernel documentation. First of all, it's a little bit stalled. Second, it said nothing about inability of nonlocal_bind to be non-working in namespaces.
2. We can try to find some patches which realizes such functionality. But we don't know which exactly patches was in ubuntu 3.19 kernel (or it is too boring task).
3. We can suppose that patches from 3.19 vanilla kernel will be in 3.19-some-distro-related kernel. In such case you can easily find proofs by yourself: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=49a601589caaf0e93194c0cc9b4ecddbe75dd2d5
This patch was placed in Sep, 2014. 3.19 vanilla kernel was issued in Feb, 2015. Is it enough proof for you?

[Vladimir Kuklin (vkuklin)]

Okay, I see, but there is still something to work on to get this commit merged. I will follow up in the review

[Vitaly Sedelnik (vsedelnik)]

Confirmed for 7.0 MU 1 per conversation with Stanislaw

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229367
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4aa596fd94888362e1d77b723b01609e90b5c4ae
Submitter: Jenkins
Branch: master

commit 4aa596fd94888362e1d77b723b01609e90b5c4ae
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed Sep 30 14:06:43 2015 +0300

    Add sysctl call to set values in namespace

    Set ip_nonlocal_bind in namespace explicitly to
    prevent HAProxy failures at start.

    Change-Id: Ibf6b2a737e97002edba2dc074c5ad239d2ef6c10
    Closes-Bug: 1500871
    Co-Authored-By: <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/237128

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237128
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5bd97fc6fae42da686be9552760b68db0dc2505b
Submitter: Jenkins
Branch: stable/7.0

commit 5bd97fc6fae42da686be9552760b68db0dc2505b
Author: Stanislaw Bogatkin <email address hidden>
Date: Wed Sep 30 14:06:43 2015 +0300

    Add sysctl call to set values in namespace

    Set ip_nonlocal_bind in namespace explicitly to
    prevent HAProxy failures at start.

    Change-Id: Ibf6b2a737e97002edba2dc074c5ad239d2ef6c10
    Closes-Bug: 1500871
    Co-Authored-By: <email address hidden>
    (cherry picked from commit 4aa596fd94888362e1d77b723b01609e90b5c4ae)

[Artem Panchenko (apanchenko-8)]

verified on 7.0 GA + MU1:

root@node-16:~# uname -r
3.19.0-32-generic
root@node-16:~# ip netns exec haproxy cat /proc/sys/net/ipv4/ip_nonlocal_bind
1

[Andrey Sledzinskiy (asledzinskiy)]

move to fix released according to previous comment

[Maksym Strukov (unbelll)]

root@node-2:~# uname -a
Linux node-2.test.domain.local 3.19.0-49-generic #55~14.04.1-Ubuntu SMP Fri Jan 22 11:24:31 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@node-2:~# ip netns exec haproxy cat /proc/sys/net/ipv4/ip_nonlocal_bind
1

Verified as fixed in 8.0-570

However merged fix assume /proc/sys/net/ipv4/ip_nonlocal_bind exists. So on kernels until 3.19 it tries write to non-existent file.