keepalived reload buggy due to improper ipv4 address comparison

Bug #1211876 reported by Peter Matulis on 2013-08-13
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
keepalived (Ubuntu)
High
Unassigned
Precise
High
Adam Stokes
Quantal
High
Adam Stokes
Raring
High
Adam Stokes

Bug Description

[Impact]
There are at least 2 identified problems with keepalived on Precise when trying to reload the configuration (/etc/keepalived/keepalived.conf):

(a) The removal of a server from the cluster does not work when the port of that server is in use by another member server

(b) The change of weight of a cluster member does not work

Reload is done by either:

1. sudo service keepalived reload
2. sudo kill -HUP `pidof keepalived`

The upstream devel list suggests that issue (a) [1] has been resolved in version 1.2.3 which is currently only in Debian unstable via version 1.2.7. I can't find the actual bug supposedly filed by 'Ronie Gilberto Henrich'. I have not tested 1.2.7 as it is not easily available to me.

The pertinent entry for (a) in the upstream changelog [2] I believe is here:

{{{ * check : Fix IPv4 address comparison routine. }}}

This bug is a request to have fixes for both issues ([3] for (a)) introduced into Ubuntu 12.04.

[1]: http://comments.gmane.org/gmane.linux.keepalived.devel/3708
[2]: http://www.keepalived.org/changelog.html
[3]: https://github.com/acassen/keepalived/commit/e7931c851ecd359379c6bbc5b39287e36cbfbf25

[Test Case]

4A. Steps to reproduce issue #1:

(a) Stop keepalived

$ sudo service keepalived stop

(b) Begin with a simple configuration file (attached: keepalived.conf)

$ cat keepalived.conf | sudo tee /etc/keepalived/keepalived.conf
$ sudo service keepalived start
$ sudo ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0

(c) Add a stanza to the configuration file (a server with a port that is in use by another server)

--------------
    real_server 192.168.1.103 8080 {
        weight 1
        inhibit_on_failure
    }
--------------

(d) Reload the configuration

$ sudo service keepalived reload
$ sudo ipvsadm -Ln

--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0
  -> 192.168.1.103:8080 Masq 1 0 0
--------------

(e) Remove the .103 stanza
(f) Perform step (d) above

--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0
  -> 192.168.1.103:8080 Masq 1 0 0
--------------

Note the .103 server remains.

4B. Steps to reproduce issue #2:

(a) Perform steps (a) and (b) from 4A
(b) Edit a stanza in the configuration file (change weight from '1' to '0' in the .102 server)

--------------
    real_server 192.168.1.102 8080 {
        weight 0
        inhibit_on_failure
    }
--------------

(c) Perform step (d) from 4A

--------------
P Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0
--------------

Note the original weight of '1' for .102 server remains.

4C. Steps to counterprove issue #1:

(a) Perform steps (a) and (b) from 4A
(b) Add a stanza to the configuration file (a server with a port that is not in use by another server)

--------------
    real_server 192.168.1.103 8081 {
        weight 1
        inhibit_on_failure
    }
--------------

(c) Perform step (d) from 4A

--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0
  -> 192.168.1.103:8081 Masq 1 0 0
--------------

(d) Remove the .103 stanza from the configuration file
(e) Perform step (d) from 4A

--------------
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 wrr
  -> 192.168.1.101:8080 Masq 1 0 0
  -> 192.168.1.102:8080 Masq 1 0 0
--------------

Note the .103 server does not remain.

5. Known Workaround:

- Restart the daemon:

$ sudo service keepalived restart

OR

- Manage the daemon with ipvsadm:

(a) Add a server

$ sudo ipvsadm -a -t 192.168.0.100:80 -r 192.168.1.103:8080 -m

(b) Edit a server (change weight)

$ sudo ipvsadm -e -t 192.168.0.100:80 -r 192.168.1.103:8080 -m -w 0

(c) Delete a server

$ sudo ipvsadm -d -t 192.168.0.100:80 -r 192.168.1.103:8080

[Regression Potential]
This introduces the proper behavior for comparing ipv4 addresses. We may see issues where keepalived was working in an environment and after upgrade fails. This probably indicates that the environment may have issues outside of keepalived as the previous behavior was incorrect.

summary: - [Precise] keepalived reload buggy
+ keepalived reload buggy
Changed in keepalived (Ubuntu):
importance: Undecided → High

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in keepalived (Ubuntu Precise):
status: New → Confirmed
Changed in keepalived (Ubuntu Quantal):
status: New → Confirmed
Changed in keepalived (Ubuntu Raring):
status: New → Confirmed
Changed in keepalived (Ubuntu):
status: New → Confirmed
Adam Stokes (adam-stokes) wrote :

Ill have the SRU ready by Friday (Sept 27th)

Changed in keepalived (Ubuntu Precise):
assignee: nobody → Adam Stokes (adam-stokes)
Changed in keepalived (Ubuntu Quantal):
assignee: nobody → Adam Stokes (adam-stokes)
Changed in keepalived (Ubuntu Raring):
assignee: nobody → Adam Stokes (adam-stokes)
Adam Stokes (adam-stokes) wrote :
description: updated
Adam Stokes (adam-stokes) wrote :
summary: - keepalived reload buggy
+ keepalived reload buggy due to improper ipv4 address comparison
description: updated
Changed in keepalived (Ubuntu):
status: Confirmed → Fix Released
Changed in keepalived (Ubuntu Quantal):
importance: Undecided → High
Changed in keepalived (Ubuntu Raring):
importance: Undecided → High
Changed in keepalived (Ubuntu Precise):
importance: Undecided → High

Hello Peter, or anyone else affected,

Accepted keepalived into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keepalived/1:1.2.2-3ubuntu2.13.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in keepalived (Ubuntu Raring):
status: Confirmed → Fix Committed
tags: added: verification-needed
Changed in keepalived (Ubuntu Quantal):
status: Confirmed → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Peter, or anyone else affected,

Accepted keepalived into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keepalived/1:1.2.2-3ubuntu2.12.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in keepalived (Ubuntu Precise):
status: Confirmed → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Peter, or anyone else affected,

Accepted keepalived into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/keepalived/1:1.2.2-3ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Peter Matulis (petermatulis) wrote :

I redid both tests (description) and re-confirmed both failures. I then installed the -proposed version below and re-did the tests. They both passed.

$ apt-cache policy keepalived
keepalived:
  Installed: 1:1.2.2-3ubuntu1.1
  Candidate: 1:1.2.2-3ubuntu1.1
  Version table:
 *** 1:1.2.2-3ubuntu1.1 0
        500 http://archive.ubuntu.com/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.2-3ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

tags: added: verification-done-precise
Peter Matulis (petermatulis) wrote :

Obviously, the above verification was done for Precise. I will soon do the same for Quantal and Raring.

Peter Matulis (petermatulis) wrote :

Ditto for Quantal - all good.

$ sudo apt-cache policy keepalived
keepalived:
  Installed: 1:1.2.2-3ubuntu2.12.10.1
  Candidate: 1:1.2.2-3ubuntu2.12.10.1
  Version table:
 *** 1:1.2.2-3ubuntu2.12.10.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ quantal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.2-3ubuntu2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ quantal/main amd64 Packages

tags: added: verification-done-quantal
Peter Matulis (petermatulis) wrote :

There is something seriously wrong with the Raring package. Starting keepalived caused my system to lock up. The same happened after trying it on a few different platforms. My last attempt was with a traditional KVM guest. After the freeze I reset the guest and I was able to capture a screenshot of a kernel panic.

Peter Matulis (petermatulis) wrote :

Hm, I tried again and it installed ok. I need more time to verify this. Don't know what's going on.

$ apt-cache policy keepalived
keepalived:
  Installed: 1:1.2.2-3ubuntu2.13.04.1
  Candidate: 1:1.2.2-3ubuntu2.13.04.1
  Version table:
 *** 1:1.2.2-3ubuntu2.13.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.2-3ubuntu2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main amd64 Packages

Adam Stokes (adam-stokes) wrote :

500 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring-proposed/main amd64 -- maybe use the default archive to do your test instead of nova.clouds?

Peter Matulis (petermatulis) wrote :

On a KVM guest the freeze happens upon starting the normal (non-proposed) keepalived.

Evan Callicoat (diopter) wrote :

For Raring, the kernel panic is happening due to this bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1238494

The patched kernel linked there fixes the issue.

Peter Matulis (petermatulis) wrote :

Confirmed. Once the henrix kernel [1] is installed the new keepalived on Raring works.

$ apt-cache policy keepalived
keepalived:
  Installed: 1:1.2.2-3ubuntu2.13.04.1
  Candidate: 1:1.2.2-3ubuntu2.13.04.1
  Version table:
 *** 1:1.2.2-3ubuntu2.13.04.1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:1.2.2-3ubuntu2 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ raring/main amd64 Packages

[1]: http://people.canonical.com/~henrix/lp1238494/

tags: added: verification-done-raring
removed: verification-needed
Peter Matulis (petermatulis) wrote :

Let me clarify. Once the henrix kernel is installed, the normal and proposed keepalived can be installed and tested.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package keepalived - 1:1.2.2-3ubuntu2.13.04.1

---------------
keepalived (1:1.2.2-3ubuntu2.13.04.1) raring-proposed; urgency=low

  * debian/patches/05_fix_address_comparison.patch:
    - Two different IPv4 addresses were returned to be
      equal while this was not the case. (LP: #1211876)
 -- Adam Stokes <email address hidden> Fri, 27 Sep 2013 11:47:47 -0400

Changed in keepalived (Ubuntu Raring):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package keepalived - 1:1.2.2-3ubuntu2.12.10.1

---------------
keepalived (1:1.2.2-3ubuntu2.12.10.1) quantal-proposed; urgency=low

  * debian/patches/05_fix_address_comparison.patch:
    - Two different IPv4 addresses were returned to be
      equal while this was not the case. (LP: #1211876)
 -- Adam Stokes <email address hidden> Fri, 27 Sep 2013 11:43:13 -0400

Changed in keepalived (Ubuntu Quantal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package keepalived - 1:1.2.2-3ubuntu1.1

---------------
keepalived (1:1.2.2-3ubuntu1.1) precise-proposed; urgency=low

  * debian/patches/05_fix_address_comparison.patch:
    - Two different IPv4 addresses were returned to be
      equal while this was not the case. (LP: #1211876)
 -- Adam Stokes <email address hidden> Fri, 27 Sep 2013 11:09:39 -0400

Changed in keepalived (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers