keepalived does not autoload the ip_vs kernel module when it is required

[Thorsten]

1)
Description: Ubuntu 16.04.5 LTS
Release: 16.04
2) keepalived:
  Installed: 1:1.2.24-1ubuntu0.16.04.1
  Candidate: 1:1.2.24-1ubuntu0.16.04.1
  Version table:
 *** 1:1.2.24-1ubuntu0.16.04.1 500
        500 http://ftp.hosteurope.de/mirror/archive.ubuntu.com xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status

3) not loading the kernel module
systemctl start keepalived.service
Keepalived_healthcheckers[1680]: IPVS: Protocol not available
Keepalived_healthcheckers[1680]: message repeated 8 times: [ IPVS: Protocol not available]
...

4) loading the module manually
systemctl stop keepalived.service
modprobe ip_vs
kernel: [ 445.363609] IPVS: ipvs loaded.
systemctl start keepalived.service
Keepalived_healthcheckers[5533]: Initializing ipvs
kernel: [ 600.828683] IPVS: [wlc] scheduler registered.

[Thorsten (itopsatheg)]

Bionic is working

[Patrick Quentin Armitage (pqa)]

This looks as though you are running keepalived from within a container. Due to the isolation of containers, applications running within the container cannot load kernel modules. Once you manually load the module, keepalived can use the module from within the container.

If keepalived is run natively within the host (i.e. not in a container) then keepalived can cause the module to be loaded.

The version of keepalived you are using is very old; there is a much newer version available as a snap.

[Karl Stenerud (kstenerud)]

Hi Thorsten,

I tried installing keepalived on a fresh xenial VM, and all seems to be working:

karl@tp-work:~$ uvt-kvm create x release=xenial arch=amd64 label=daily
karl@tp-work:~$ uvt-kvm wait x
karl@tp-work:~$ uvt-kvm ssh x
ubuntu@x:~$ sudo apt update && sudo apt dist-upgrade -y && sudo apt install -y keepalived
ubuntu@x:~$ echo "global_defs {

   notification_email {
       <email address hidden>
   }
   notification_email_from <email address hidden>
   smtp_server 127.0.0.1
   smtp_connect_timeout 60
}" | sudo tee /etc/keepalived/keepalived.conf >/dev/null
ubuntu@x:~$ sudo service keepalived start
ubuntu@x:~$ sudo service keepalived status
● keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-11-14 11:35:56 UTC; 18s ago
  Process: 8850 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 8854 (keepalived)
    Tasks: 3
   Memory: 1.7M
      CPU: 11ms
   CGroup: /system.slice/keepalived.service
           ├─8854 /usr/sbin/keepalived
           ├─8855 /usr/sbin/keepalived
           └─8856 /usr/sbin/keepalived

Nov 14 11:35:56 x Keepalived_vrrp[8856]: Registering gratuitous ARP shared channel
Nov 14 11:35:56 x Keepalived_vrrp[8856]: Unable to load ipset library
Nov 14 11:35:56 x Keepalived_vrrp[8856]: Unable to initialise ipsets
Nov 14 11:35:56 x Keepalived_vrrp[8856]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 14 11:35:56 x Keepalived_vrrp[8856]: Using LinkWatch kernel netlink reflector...
Nov 14 11:35:56 x Keepalived_healthcheckers[8855]: Initializing ipvs
Nov 14 11:35:56 x Keepalived_healthcheckers[8855]: Registering Kernel netlink reflector
Nov 14 11:35:56 x Keepalived_healthcheckers[8855]: Registering Kernel netlink command channel
Nov 14 11:35:56 x Keepalived_healthcheckers[8855]: Opening file '/etc/keepalived/keepalived.conf'.
Nov 14 11:35:56 x Keepalived_healthcheckers[8855]: Using LinkWatch kernel netlink reflector...

Are you running it in a container? If so, Patrick's comments will apply.

[Thorsten (itopsatheg)]

@pqa It's not running in a container we want stable software

----------------------------------------------------------------------

@kstenerud

# dmsg
...
[Fri Nov 16 10:45:58 2018] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[Fri Nov 16 10:45:58 2018] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[Fri Nov 16 10:45:58 2018] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[Fri Nov 16 10:45:58 2018] IPVS: ipvs loaded.

# systemctl status keepalived.service
● keepalived.service - Keepalive Daemon (LVS and VRRP)
   Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-11-16 10:45:54 CET; 1min 13s ago
  Process: 1751 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 1832 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─1832 /usr/sbin/keepalived -d
           ├─1833 /usr/sbin/keepalived -d
           └─1834 /usr/sbin/keepalived -d

# lsmod | grep ip_
ip_vs 151552 0
nf_conntrack 131072 1 ip_vs
libcrc32c 16384 3 nf_conntrack,xfs,ip_vs

# systemctl stop keepalived.service
# lsmod | grep ip_
  ip_vs 151552 0
  nf_conntrack 131072 1 ip_vs
  libcrc32c 16384 3 nf_conntrack,xfs,ip_vs
# modprobe -r ip_vs
# lsmod | grep ip
  ipmi_ssif 32768 0
  ipmi_si 57344 0
  ipmi_devintf 20480 0
  ipmi_msghandler 53248 3 ipmi_devintf,ipmi_si,ipmi_ssif

# modprobe ip_vs
# lsmod | grep ip
  ip_vs 151552 0
  nf_conntrack 131072 1 ip_vs
  libcrc32c 16384 3 nf_conntrack,xfs,ip_vs
# systemctl start keepalived.service
# lsmod | grep ip_
  ip_vs_wlc 16384 3
  ip_vs 151552 5 ip_vs_wlc
  nf_conntrack 131072 1 ip_vs
  libcrc32c 16384 3 nf_conntrack,xfs,ip_vs

So the problem is "ip_vs_wlc" if this module isn't loading
# ipvsadm -Ln
  IP Virtual Server version 1.2.1 (size=4096)
  Prot LocalAddress:Port Scheduler Flags
     -> RemoteAddress:Port Forward Weight ActiveConn InActConn

with the module
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.10.10:1234 wlc persistent 1440
  -> 10.1.1.2:1234 Route 10 0 0
  -> 10.1.1.3:1234 Route 10 0 0

[Andreas Hasenack (ahasenack)]

Can you share your config /etc/keepalived/keepalived.conf ?

[Thorsten (itopsatheg)]

global_defs {
    notification_email {
        <email address hidden>
    }
    notification_email_from <email address hidden>
    smtp_server 127.0.0.1
    smtp_connect_timeout 15
    router_id lb2.domain.tld
}

vrrp_instance VRRP_SERVICE {
        virtual_router_id 24
        interface uplink0
        state BACKUP
        priority 200
        nopreempt
        garp_master_delay 5
        advert_int 3
        authentication {
                auth_type AH
                auth_pass secret
        }
        notify "/usr/local/bin/script"
        notify_stop "/usr/local/bin/script backup"

        virtual_ipaddress {
               10.1.2.3
        }
}

virtual_server 10.1.2.3 4119 {
 delay_loop 10
 lb_algo wlc
 lb_kind DR
 protocol TCP
 persistence_timeout 1440

 real_server 10.80.22.54 4119 {
  weight 10
  TCP_CHECK {
   connect_port 4119
   connect_timeout 3
  }
 }
 real_server 10.80.22.53 4119 {
  weight 10
  TCP_CHECK {
   connect_port 4119
   connect_timeout 3
  }
 }
    }
virtual_server 10.1.2.3 4120 {
 delay_loop 10
 lb_algo wlc
 lb_kind DR
 protocol TCP
 persistence_timeout 1440

 real_server 10.80.22.54 4120 {
  weight 10
  TCP_CHECK {
   connect_port 4120
   connect_timeout 3
  }
 }
 real_server 10.80.22.53 4120 {
  weight 10
  TCP_CHECK {
   connect_port 4120
   connect_timeout 3
  }
 }
    }
virtual_server 10.1.2.3 4122 {
 delay_loop 10
 lb_algo wlc
 lb_kind DR
 protocol TCP
 persistence_timeout 1440

 real_server 10.80.22.54 4122 {
  weight 10
  TCP_CHECK {
   connect_port 4122
   connect_timeout 3
  }
 }
 real_server 10.80.22.53 4122 {
  weight 10
  TCP_CHECK {
   connect_port 4122
   connect_timeout 3
  }
 }
    }

[Robie Basak (racb)]

Reproduced with your keepalived.conf, thanks - specifically that what goes to syslog seems to be a problem on startup without ip_vs manually loaded, and after a manual modprobe there is less of an error (I get other errors presumably because I don't have your network set up).

I'm surprised to hear of the report in the Debian bug that adding ip_vs to /etc/modules does not work. @Thorsten please could you try this? It's a workaround and not a proper permanent fix but if the workaround doesn't work that will help us inform us for a proper fix.

I also don't see any changes in packaging that would have caused this to have been fixed in Bionic. I wonder if there is a change keepalived that landed after Xenial's version that fixes this.

[Thorsten (itopsatheg)]

My work-a-round is that I'm knewing this problem and so I load it manually.
Bionic is working, I wrote it 2018-10-26.
We still use Debian 6-9 and Ubuntu 12-18

[Robie Basak (racb)]

Understood, but if you'd like this properly fixed in Xenial, it would be very helpful to us if you could tell us if the /etc/modules workaround works for you. This will help inform our work.

[Thorsten (itopsatheg)]

It's also a workaround

# date
  Thu Nov 22 19:06:07 CET 2018
# stat /etc/modules
  File: '/etc/modules'
  Size: 200 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 272949 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-11-22 19:02:52.440822259 +0100
Modify: 2018-11-22 19:02:52.440822259 +0100
Change: 2018-11-22 19:02:52.444822297 +0100
 Birth: -
# cat /etc/modules
  # /etc/modules: kernel modules to load at boot time.
  #
  # This file contains the names of kernel modules that should be loaded
  # at boot time, one per line. Lines beginning with "#" are ignored.
  ip_vs
# last reboot
  reboot system boot 4.15.0-42-generi Thu Nov 22 19:05 still running

[Thorsten (itopsatheg)]

# lsmod | grep ip_
  ip_vs_wlc 16384 3
  ip_vs 151552 5 ip_vs_wlc
  nf_conntrack 131072 1 ip_vs
  libcrc32c 16384 3 nf_conntrack,xfs,ip_vs

[Robie Basak (racb)]

Great, thanks!

[Robie Basak (racb)]

I agree that the workaround isn't sufficient and this should really work out of the box, so I've added this to the server team's backlog. But since it is reported fixed in Bionic and a workaround is available for Xenial, I'm prioritising other work over this bug. If someone can figure out and provide a patch, we'll help you get it landed. Otherwise, I expect this bug to remain inactive for a while, but hopefully we can get to it the next time someone from the server team tries to clear out some of the bugs open against keepalived.

[Paride Legovini (paride)]

Xenial is now in Extended Security Maintenance and this bug doesn't qualify for it, therefore I'm marking the Xenial task as Won't Fix.

[Paride Legovini (paride)]

As this is reported fixed in Bionic with no indication of it *not* being fixed in >= Cosmic, I'm marking the main bug task as Fix Released. Should this not be the case do not hesitate to change the bug status again and/or comment back. Thanks!