ethmonitor does not list interfaces without assigned IP address

Bug #1825250 reported by Heitor Alves de Siqueira on 2019-04-17
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
resource-agents (Debian)
Fix Released
Unknown
resource-agents (Ubuntu)
Status tracked in Eoan
Xenial
Medium
Heitor Alves de Siqueira
Bionic
Medium
Heitor Alves de Siqueira
Cosmic
Medium
Heitor Alves de Siqueira
Disco
Medium
Heitor Alves de Siqueira
Eoan
Medium
Heitor Alves de Siqueira

Bug Description

[Impact]
Some network interfaces will not be monitored by ethmonitor

[Description]
The is_interface() function in ethmonitor tries to match an interface to a list obtained from the 'ip' tool. It lists interfaces using the 'inet' family, which omits interfaces that don't have an IP address assigned.

If the interface that we're looking for is e.g. a VLAN bridge that does not have an IP address, it won't show up in the listing and is_interface() will return false. ethmonitor will miss that interface, and it won't be available for monitoring.

Upstream commits:
- https://github.com/ClusterLabs/resource-agents/commit/40d05029ce0b
- https://github.com/ClusterLabs/resource-agents/commit/c0ac191c73f1

[Test Case]
1) Ensure there's a network interface without an assigned IP address. For example, virbr0-nic will be created automatically by uvt-kvm:
# ip addr show dev virbr0-nic
11: virbr0-nic: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:e9:5e:af brd ff:ff:ff:ff:ff:ff

2) Install pcs+arping and create a new ethmonitor resource with the target interface:
# sudo apt update && sudo apt install pcs arping -y
# pcs resource create p_nic ocf:heartbeat:ethmonitor interface=virbr0-nic op monitor timeout="10s"

3) Debug-start ethmonitor resource and check for "Interface does not exist messages"
# pcs resource debug-start p_nic
Operation start for p_nic (ocf:heartbeat:ethmonitor) returned: 'ok' (0)
 > stderr: WARNING: Interface virbr0-nic does not exist
 > stderr: NOTICE: link_status: DOWN

[Regression Potential]
The regression potential is low, since we are relaxing the monitoring conditions for interfaces without an assigned IP address. The patches have been tested against Travis-CI before being merged upstream, and will be tested against autopkgtest for each target distro.

Changed in resource-agents (Ubuntu Xenial):
status: New → Confirmed
Changed in resource-agents (Ubuntu Bionic):
status: New → Confirmed
Changed in resource-agents (Ubuntu Cosmic):
status: New → Confirmed
Changed in resource-agents (Ubuntu Xenial):
importance: Undecided → Medium
Changed in resource-agents (Ubuntu Bionic):
importance: Undecided → Medium
Changed in resource-agents (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in resource-agents (Ubuntu Xenial):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in resource-agents (Ubuntu Cosmic):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in resource-agents (Ubuntu Bionic):
assignee: nobody → Heitor Alves de Siqueira (halves)
Changed in resource-agents (Debian):
status: Unknown → New
Changed in resource-agents (Ubuntu Disco):
status: New → Confirmed
importance: Undecided → Critical
importance: Critical → Medium
assignee: nobody → Heitor Alves de Siqueira (halves)
tags: added: sts-sponsor
Eric Desrochers (slashd) on 2019-04-23
Changed in resource-agents (Ubuntu Eoan):
status: Confirmed → In Progress
Eric Desrochers (slashd) wrote :

Sponsored for 'eoan' considering that the:
1) Both upstream fixes are merged upstream
2) Bug is reported in debian against resource-agents (debbugs #927311)
3) Eoan is now open (#ubuntu-release topic: ... Archive: Open | Eoan Release Coordination...)

Nitpick:
d/changelog was mentionning 'unstable' instead of 'eoan'.

- Eric

Changed in resource-agents (Ubuntu Eoan):
status: In Progress → Fix Committed

The attachment "lp1825250-eoan.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package resource-agents - 1:4.2.0-1ubuntu2

---------------
resource-agents (1:4.2.0-1ubuntu2) eoan; urgency=medium

  * Fix heartbeat/ethmonitor not listing network interfaces without IP
    addresses (LP: #1825250). Fixed by upstream patches:
    - d/p/lp1825250-ethmonitor-check-if-interface-exists-by-link.patch:
      ethmonitor: check if interface exists by link
    - d/p/lp1825250-ethmonitor-fix-is_interface-regression.patch:
      ethmonitor: fix is_interface() regression

 -- Heitor Alves de Siqueira <email address hidden> Tue, 23 Apr 2019 14:17:29 -0300

Changed in resource-agents (Ubuntu Eoan):
status: Fix Committed → Fix Released
Simon Quigley (tsimonq2) wrote :

Unsubscribing sponsors because there is nothing else to sponsor.

Eric Desrochers (slashd) on 2019-04-29
Changed in resource-agents (Ubuntu Disco):
status: Confirmed → In Progress
Changed in resource-agents (Ubuntu Cosmic):
status: Confirmed → In Progress
Changed in resource-agents (Ubuntu Bionic):
status: Confirmed → In Progress
Changed in resource-agents (Ubuntu Xenial):
status: Confirmed → In Progress
Dan Streetman (ddstreet) on 2019-04-29
tags: added: sts-sponsor-ddstreet

Hello Heitor, or anyone else affected,

Accepted resource-agents into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resource-agents/1:4.2.0-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in resource-agents (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Brian Murray (brian-murray) wrote :

Hello Heitor, or anyone else affected,

Accepted resource-agents into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resource-agents/1:4.1.1-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in resource-agents (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed-cosmic
Changed in resource-agents (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Brian Murray (brian-murray) wrote :

Hello Heitor, or anyone else affected,

Accepted resource-agents into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resource-agents/1:4.1.0~rc1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in resource-agents (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Brian Murray (brian-murray) wrote :

Hello Heitor, or anyone else affected,

Accepted resource-agents into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/resource-agents/1:3.9.7-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verified resource-agents from xenial-proposed according to test case from description:

root@xenial:~# dpkg -l | grep resource-agents
ii resource-agents 1:3.9.7-1ubuntu1.1 amd64 Cluster Resource Agents

root@xenial:~# ip addr show dev virbr0-nic
3: virbr0-nic: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 4a:6b:29:99:b5:a7 brd ff:ff:ff:ff:ff:ff

root@xenial:~# pcs resource create p_nic ocf:heartbeat:ethmonitor interface=virbr0-nic op monitor timeout="10s"

root@xenial:~# pcs resource debug-start p_nic
Operation start for p_nic (ocf:heartbeat:ethmonitor) returned 0
 > stderr: DEBUG: link_status: 1 (1=up, 0=down)
 > stderr: DEBUG: watch for packet counter changes
 > stderr: DEBUG: ip -o -s link show dev virbr0-nic
 > stderr: DEBUG: ip -o -s link show dev virbr0-nic
 > stderr: DEBUG: RX_PACKETS_OLD: 0 RX_PACKETS_NEW: 0
 > stderr: DEBUG: ip -o -s link show dev virbr0-nic
 > stderr: DEBUG: RX_PACKETS_OLD: 0 RX_PACKETS_NEW: 0
 ...

tags: added: verification-done-xenial
removed: verification-needed-xenial

Verified resource-agents from bionic-proposed according to test case from description. The WARNING messages about monitoring failure are fine, as long as we don't get "WARNING: Interface virbr0-nic does not exist".

root@bionic:~# dpkg -l | grep resource-agents
ii resource-agents 1:4.1.0~rc1-1ubuntu1.1 amd64 Cluster Resource Agents

root@bionic:~# ip addr show dev virbr0-nic
3: virbr0-nic: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:4d:9e:13:ea:b9 brd ff:ff:ff:ff:ff:ff

root@bionic:~# pcs resource create p_nic ocf:heartbeat:ethmonitor interface=virbr0-nic op monitor timeout="10s"

root@bionic:~# pcs resource debug-start p_nic
Operation start for p_nic (ocf:heartbeat:ethmonitor) returned: 'ok' (0)
 > stderr: INFO: No ARP cache entries found to arping
 > stderr: WARNING: Monitoring of p_nic failed, 4 retries left.
 > stderr: INFO: No ARP cache entries found to arping
 > stderr: WARNING: Monitoring of p_nic failed, 3 retries left.
...

tags: added: verification-done-bionic
removed: verification-needed-bionic

Verified resource-agents from cosmic-proposed according to test case from description. The WARNING messages about monitoring failure are fine, as long as we don't get "WARNING: Interface virbr0-nic does not exist".

root@cosmic:~# dpkg -l | grep resource-agents
ii resource-agents 1:4.1.1-2ubuntu1 amd64 Cluster Resource Agents

root@cosmic:~# ip addr show dev virbr0-nic
3: virbr0-nic: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 72:45:b7:dc:d5:36 brd ff:ff:ff:ff:ff:ff

root@cosmic:~# pcs resource create p_nic ocf:heartbeat:ethmonitor interface=virbr0-nic op monitor timeout="10s"

root@cosmic:~# pcs resource debug-start p_nic
Operation start for p_nic (ocf:heartbeat:ethmonitor) returned: 'ok' (0)
 > stderr: INFO: No ARP cache entries found to arping
 > stderr: WARNING: Monitoring of p_nic failed, 4 retries left.
 > stderr: INFO: No ARP cache entries found to arping
 > stderr: WARNING: Monitoring of p_nic failed, 3 retries left.
...

tags: added: verification-done-cosmic
removed: verification-needed-cosmic

# NOTE
Due to bug 1826045, I've done the test below with a patched version of pcs to work around the pacemaker/pcs version problems. The pcs package does not affect the resource-agents SRU, and is only used for an easy, straightforward reproducer.
#

Verified resource-agents from disco-proposed according to test case from description. The WARNING messages about monitoring failure are fine, as long as we don't get "WARNING: Interface virbr0-nic does not exist".

root@disco:~# dpkg -l | grep resource-agents
ii resource-agents 1:4.2.0-1ubuntu1.1 amd64 Cluster Resource Agents

root@disco:~# ip addr show dev virbr0-nic
3: virbr0-nic: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:c1:6a:4e:09:ea brd ff:ff:ff:ff:ff:ff

root@disco:~# pcs resource create p_nic ocf:heartbeat:ethmonitor interface=virbr0-nic op monitor timeout="10s"

root@disco:~# pcs resource debug-start p_nic
Operation start for p_nic (ocf:heartbeat:ethmonitor) returned: 'ok' (0)
 > stderr: May 14 19:01:41 INFO: No ARP cache entries found to arping
 > stderr: May 14 19:01:41 WARNING: Monitoring of p_nic failed, 4 retries left.
 > stderr: May 14 19:01:51 INFO: No ARP cache entries found to arping
 > stderr: May 14 19:01:51 WARNING: Monitoring of p_nic failed, 3 retries left.

tags: added: verification-done-disco
removed: verification-needed-disco

Quick note on the autopkgtest regression for resource-agents:

that autopkgtest fails due to bug 1828258, which causes a package configuration from dpkg to fail (and consequently, the test). It's not related to this SRU, and the rest of the autopkgtest log looks fine.

tags: added: verification-done
removed: verification-needed
Łukasz Zemczak (sil2100) wrote :

Excellent. Thank you for the detailed verification and autopkgtest regression investigation. It's so satisfying to see competent SRU handling - makes everyone's work much easier. Hinted the failure. Thanks!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package resource-agents - 1:4.2.0-1ubuntu1.1

---------------
resource-agents (1:4.2.0-1ubuntu1.1) disco; urgency=medium

  * Fix heartbeat/ethmonitor not listing network interfaces without IP
    addresses (LP: #1825250). Fixed by upstream patches:
    - d/p/lp1825250-ethmonitor-check-if-interface-exists-by-link.patch:
      ethmonitor: check if interface exists by link
    - d/p/lp1825250-ethmonitor-fix-is_interface-regression.patch:
      ethmonitor: fix is_interface() regression

 -- Heitor Alves de Siqueira <email address hidden> Tue, 23 Apr 2019 13:56:29 -0300

Changed in resource-agents (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for resource-agents has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package resource-agents - 1:4.1.1-2ubuntu1

---------------
resource-agents (1:4.1.1-2ubuntu1) cosmic; urgency=medium

  * Fix heartbeat/ethmonitor not listing network interfaces without IP
    addresses (LP: #1825250). Fixed by upstream patches:
    - d/p/lp1825250-ethmonitor-check-if-interface-exists-by-link.patch:
      ethmonitor: check if interface exists by link
    - d/p/lp1825250-ethmonitor-fix-is_interface-regression.patch:
      ethmonitor: fix is_interface() regression

 -- Heitor Alves de Siqueira <email address hidden> Tue, 23 Apr 2019 14:07:57 -0300

Changed in resource-agents (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package resource-agents - 1:4.1.0~rc1-1ubuntu1.1

---------------
resource-agents (1:4.1.0~rc1-1ubuntu1.1) bionic; urgency=medium

  * Fix heartbeat/ethmonitor not listing network interfaces without IP
    addresses (LP: #1825250). Fixed by upstream patches:
    - d/p/lp1825250-ethmonitor-check-if-interface-exists-by-link.patch:
      ethmonitor: check if interface exists by link
    - d/p/lp1825250-ethmonitor-fix-is_interface-regression.patch:
      ethmonitor: fix is_interface() regression

 -- Heitor Alves de Siqueira <email address hidden> Tue, 23 Apr 2019 14:12:58 -0300

Changed in resource-agents (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package resource-agents - 1:3.9.7-1ubuntu1.1

---------------
resource-agents (1:3.9.7-1ubuntu1.1) xenial; urgency=medium

  * Fix heartbeat/ethmonitor not listing network interfaces without IP
    addresses (LP: #1825250). Fixed by upstream patches:
    - d/p/lp1825250-ethmonitor-check-if-interface-exists-by-link.patch:
      ethmonitor: check if interface exists by link
    - d/p/lp1825250-ethmonitor-fix-is_interface-regression.patch:
      ethmonitor: fix is_interface() regression

 -- Heitor Alves de Siqueira <email address hidden> Tue, 23 Apr 2019 14:15:40 -0300

Changed in resource-agents (Ubuntu Xenial):
status: Fix Committed → Fix Released
Dan Streetman (ddstreet) on 2019-05-31
tags: removed: sts-sponsor sts-sponsor-ddstreet
Changed in resource-agents (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.