[SRU] /usr/sbin/fanctl: arithmetic expression: expecting primary | unconfigured interfaces cause ifup failures

Bug #1737640 reported by Jason Hobbs on 2017-12-11
92
This bug affects 21 people
Affects Status Importance Assigned to Milestone
OPNFV
Critical
Narinder Gupta
juju
High
Unassigned
ubuntu-fan (Ubuntu)
Undecided
Andreas Hasenack
Xenial
Undecided
Andreas Hasenack
Zesty
Undecided
Andreas Hasenack
Artful
Undecided
Andreas Hasenack
Bionic
Undecided
Andreas Hasenack

Bug Description

[Impact]

 * This bug effects anyone deploying LXD with juju on a machine with interfaces configured as IP/mask. Since this is a very common use case, SRU is needed.

[Test Case]

 * Detailed process can be seen at https://paste.ubuntu.com/26169937/

[Regression Potential]

 * Potential for regression is low because of the nature of the error. It's just missing a case stanza for interfaces that are defined with IP/mask

[Other Info]

 * Original report:

I'm seeing this error as the status of multiple containers in my deploy:

http://paste.ubuntu.com/26166720/

I can't connect to the parent machines anymore either - it seems networking is totally hosed on the machines.

This is with juju 2.3.1.

[SRU Justification]

== Impact ==

Due to unexpectedly passing up a non-critical error to the ifup hooks, this causes network interfaces to not come up.

== Fix ==

This was fixed in Xenial (and already forward ported to Bionic) effectively allowing to bring up network interfaces via ifupdown after installing ubuntu-fan.

== Testcase ==

Basic installation with ifupdown method for configuring network interfaces (no longer default for Artful and later). Then installing ubuntu-fan and either reboot or use ifdown -a && ifup -a in case of being logged in locally. Without the fix interfaces will not come up properly.

== Regression Risk ==

Minimal, well tested in Xenial and not impacting Artful and later if ifupdown is not used.

John A Meinel (jameinel) wrote :

This might be related to ubuntu-fan and machines that have an unconfigured interface.
See also bug #1737721 which is a private bug with more information.

Changed in juju:
importance: Undecided → High
status: New → Triaged
Dmitrii Shcherbakov (dmitriis) wrote :

ubuntu-fan clearly triggers the problem.

Clean machine, no bridges after Juju installs ubuntu-fan and networking still works as bridges were not created yet => no ifup/down executions:

root@maas-xenial3:/home/ubuntu# ifdown -a ; ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# apt purge ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  ubuntu-fan*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 148 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 65843 files and directories currently installed.)
ubuntu-fan: removing default /etc/network/fan configuration
Purging configuration files for ubuntu-fan (0.12.8~16.04.1) ...

root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

root@maas-xenial3:/home/ubuntu# ifdown ens3
ifdown: interface ens3 not configured

root@maas-xenial3:/home/ubuntu# ip a d 10.10.101.103/24 dev ens3
root@maas-xenial3:/home/ubuntu# ifup -a

root@maas-xenial3:/home/ubuntu# apt install ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  ubuntu-fan
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/35.6 kB of archives.
Selecting previously unselected package ubuntu-fan.ce will be used.
Processing triggers for systemd (229-4ubuntu21) ...urrently installed.)
Processing triggers for ureadahead (0.100.0-19) ...ll.deb ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21) ...
root@maas-xenial3:/home/ubuntu# ifdown -a
root@maas-xenial3:/home/ubuntu# ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.
root@maas-xenial3:/home/ubuntu# ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.

summary: - bridge activaction error: bridge activation failed
+ bridge activaction error: bridge activation failed | unconfigured
+ interfaces cause ifup failures
tags: added: cpe-onsite
Dmitrii Shcherbakov (dmitriis) wrote :

Before and after ifdown -> ifup with fan:

http://paste.ubuntu.com/26169937/

After purging:

http://paste.ubuntu.com/26169950/

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-fan (Ubuntu):
status: New → Confirmed
Dmitrii Shcherbakov (dmitriis) wrote :

This is reproducible on a machine with a single NIC:

https://paste.ubuntu.com/26170595/

summary: - bridge activaction error: bridge activation failed | unconfigured
- interfaces cause ifup failures
+ /usr/sbin/fanctl: arithmetic expression: expecting primary |
+ unconfigured interfaces cause ifup failures
Andy Whitcroft (apw) on 2017-12-12
Changed in ubuntu-fan (Ubuntu):
importance: Undecided → Critical
assignee: nobody → Stefan Bader (smb)
Dmitrii Shcherbakov (dmitriis) wrote :

Tested that on a "broken" environment (after a reboot with a broken package):

root@maas-xenial3:~# apt install ubuntu-fan
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  ubuntu-fan
1 upgraded, 0 newly installed, 0 to remove and 94 not upgraded.
Need to get 36.4 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ppa.launchpad.net/canonical-kernel-team/fan-fixes/ubuntu xenial/main amd64 ubuntu-fan all 0.12.8~16.04.2~rc1 [36.4 kB]
Fetched 36.4 kB in 0s (0 B/s)
(Reading database ... 60144 files and directories currently installed.)
Preparing to unpack .../ubuntu-fan_0.12.8~16.04.2~rc1_all.deb ...
Unpacking ubuntu-fan (0.12.8~16.04.2~rc1) over (0.12.8~16.04.1) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up ubuntu-fan (0.12.8~16.04.2~rc1) ...

# after a reboot with a broken package an address was present but the interface was not considered as "configured" by ifupdown - had to manually delete an address to avoid RTNETLINK answers: File exists on ifup

root@maas-xenial3:~# ip a d 10.10.101.104/24 dev ens3

root@maas-xenial3:~# ifdown ens3
ifdown: interface ens3 not configured

root@maas-xenial3:~# ifup ens3

root@maas-xenial3:~# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:82:7d:16 brd ff:ff:ff:ff:ff:ff
    inet 10.10.101.104/24 brd 10.10.101.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe82:7d16/64 scope link
       valid_lft forever preferred_lft forever

# all good
root@maas-xenial3:~# ifdown -a
root@maas-xenial3:~# ifup -a

Andy Whitcroft (apw) wrote :

Will upload this to xenial-proposed urgently.

tags: added: regression-update

Hello Jason, or anyone else affected,

Accepted ubuntu-fan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.8~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ubuntu-fan (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial

On xenial, deployed with maas 2.3.0. Node with two network interfaces: the first one (ens3) set to "auto" in maas, and the second one (ens8) left as "unconfigured" in MAAS:
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:58:04:57 brd ff:ff:ff:ff:ff:ff
    inet 10.0.5.10/24 brd 10.0.5.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe58:457/64 scope link
       valid_lft forever preferred_lft forever
3: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:66:6c:1c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe66:6c1c/64 scope link
       valid_lft forever preferred_lft forever

With ubuntu-fan 0.12.8~16.04.1 installed:

root@04-57:~# ifdown -a; ifup -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "
run-parts: /etc/network/if-up.d/ubuntu-fan exited with return code 2
Failed to bring up ens3.

That also leaves the networking system in a bit of an inconsistent state:
root@04-57:~# ifdown ens3
ifdown: interface ens3 not configured

Bug confirmed.

I chose to reboot now, and then install the updated ubuntu-fan package from proposed:

SKIP - package not available yet - will continue sometime later

Andreas Hasenack (ahasenack) wrote :

Ok, package available in proposed.
  Version table:
 *** 0.12.8~16.04.2 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages

Confusing result right after the upgrade:
root@04-57:~# ifdown -a; ifup -a
RTNETLINK answers: File exists
Failed to bring up ens3.
root@04-57:~# ifdown ens3
ifdown: interface ens3 not configured

rebooting

Now it works:root@04-57:~# ifdown -a; ifup -a
root@04-57:~#

If this test is acceptable, then consider this verified for xenial.

Frode Nordahl (fnordahl) wrote :

I can confirm that the proposed ubuntu-fan package resolves the issue.

If you are deploying with Juju and MaaS the Ubuntu proposed repository can be installed automatically on the nodes deployed by MaaS. This could be used as a temporary workaround for the issue.

Example for how it could be set up in MaaS:
https://gist.githubusercontent.com/fnordahl/f6e6b734484363e980a6e7fa2cfe51de/raw/b64d83225e0d11c1bd236e4f4625b0c466ad755f/maas-ubuntu-proposed.png

dann frazier (dannf) wrote :

Changing tag to verified as per comments #13 & #14.

tags: added: verification-done-xenial
removed: verification-needed-xenial
Jason Hobbs (jason-hobbs) wrote :

I just tested this also and can verify it fixed it in the environment/test where it was originally reported as broken.

james beedy (jamesbeedy) wrote :

thinking this is the same bug/duplicate I'm hitting here https://bugs.launchpad.net/juju/+bug/1736022

I'm deploying a few nodes to test the fix in proposed now

james beedy (jamesbeedy) wrote :

I can verify the fix works in my environment using whats in xenial-proposed http://paste.ubuntu.com/26178059/

Thanks!

Jason Hobbs (jason-hobbs) wrote :

Testing on arm64, the workaround of adding xenial-proposed via maas doesn't work - the newer ubuntu-fan package isn't being installed

http://paste.ubuntu.com/26178859/

I don't know how that can be, since the repo is being added (or should be added) before juju installs ubuntu-fan.

Michael Quiniola (mcl0v1n) wrote :

The workaround by adding the proposed repository into package repositories in MAAS worked for me. Thanks for the help!

Narinder Gupta (narindergupta) wrote :

The workaround by adding the proposed repository into package repositories in MAAS worked for me.

Changed in opnfv:
status: New → Fix Committed
importance: Undecided → Critical
assignee: nobody → Narinder Gupta (narindergupta)
Changed in juju:
status: Triaged → Invalid
tags: removed: verification-needed
tags: added: verification-done
Robie Basak (racb) wrote :

12:58 <rbasak> apw: no SRU paperwork in bug 1737640? I don't know how to check that the SRU verification minimises regression risk.

Robie Basak (racb) wrote :

Blocked on:

 * SRU paperwork
 * Fixes for future stable releases
 * Fix for Bionic

I understand that this was being treated as urgent. But it's been eight days, and so if this were still the case I'd have expected the other essential pieces to have been completed by now. I'm not sure it can be treated as urgent any more.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu-fan (Ubuntu Artful):
status: New → Confirmed
Changed in ubuntu-fan (Ubuntu Zesty):
status: New → Confirmed
John George (jog) wrote :

This bug certainly needs to be treated as urgent. It's being tracked as part of the critical SLA process and blocks deployments of foundation bundles in the QA lab and in the field.

Nobuto Murata (nobuto) wrote :

Is there anyone from ubuntu-fan developers willing to satisfy SRU requirements below?

> Blocked on:
>
> * SRU paperwork
> * Fixes for future stable releases
> * Fix for Bionic

I could try those, but it's not ideal since I have to go through sponsorship processes and guess the intention of the patch.

This bug is critical to me since the proposed workaround for MAAS deployments as adding -proposed in MAAS is not realistic in production cloud because:

- MAAS is not capable to add ubuntu-release aware apt line for -proposed using the custom repository management (trusty nodes will also have xenial-proposed apt line unexpectedly)
- MAAS cannot specify the pin-priority of xenial-proposed using the custom repository management so unnecessary kernels and grub will be installed from -proposed

I ended up using a little bit verbose late_commands in curtin_userdata which is not so handy:
https://paste.ubuntu.com/26319353/

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-fan - 0.12.8~16.04.2

---------------
ubuntu-fan (0.12.8~16.04.2) xenial; urgency=medium

  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Andy Whitcroft <email address hidden> Tue, 12 Dec 2017 15:34:26 +0000

Changed in ubuntu-fan (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for ubuntu-fan has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in ubuntu-fan (Ubuntu Xenial):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in ubuntu-fan (Ubuntu Artful):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in ubuntu-fan (Ubuntu Bionic):
assignee: Stefan Bader (smb) → Andreas Hasenack (ahasenack)

As per earlier arrangements, I have published the package as-is as per the provided test feedback in comments. It's a one-off exception, bionic and other series will be prepared ASAP.

Changed in ubuntu-fan (Ubuntu Zesty):
assignee: nobody → Andreas Hasenack (ahasenack)
Ante Karamatić (ivoks) on 2018-01-04
description: updated
summary: - /usr/sbin/fanctl: arithmetic expression: expecting primary |
+ [SRU] /usr/sbin/fanctl: arithmetic expression: expecting primary |
unconfigured interfaces cause ifup failures
description: updated
Andreas Hasenack (ahasenack) wrote :

I can see that ubuntu-fan in zesty, artful and bionic needs this patch, but I can't determine if it's actually hit by this bug.

artful, for example, will use netplan in a fresh install, not e-n-i. Are these releases really affected by this bug?

Andreas Hasenack (ahasenack) wrote :

I mean, I can see the error in netplan systems like artful, but maybe the impact there is different and local to ubuntu-fan only:

root@04-57:~# fanctl up -a
/usr/sbin/fanctl: 41: /usr/sbin/fanctl: arithmetic expression: expecting primary: " (32-)/4 "

Changed in ubuntu-fan (Ubuntu Artful):
status: Confirmed → Incomplete
Changed in ubuntu-fan (Ubuntu Bionic):
status: Confirmed → Incomplete
Andreas Hasenack (ahasenack) wrote :

Zesty will be EOL before we can push this fix out, marking it as "won't fix"

Changed in ubuntu-fan (Ubuntu Zesty):
status: Confirmed → Won't Fix
Changed in ubuntu-fan (Ubuntu Bionic):
importance: Critical → Undecided
Changed in ubuntu-fan (Ubuntu Artful):
status: Incomplete → In Progress
Changed in ubuntu-fan (Ubuntu Bionic):
status: Incomplete → In Progress
Andreas Hasenack (ahasenack) wrote :

The bug exists in artful and bionic, it's probably just not as critical as it was ifupdown systems like xenial and zesty.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-fan - 0.12.9

---------------
ubuntu-fan (0.12.9) bionic; urgency=medium

  [ Andy Whitcroft ]
  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Andreas Hasenack <email address hidden> Thu, 04 Jan 2018 15:41:32 -0200

Changed in ubuntu-fan (Ubuntu Bionic):
status: In Progress → Fix Released
Changed in opnfv:
status: Fix Committed → Invalid
Stefan Bader (smb) on 2018-01-31
description: updated

Hello Jason, or anyone else affected,

Accepted ubuntu-fan into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-fan/0.12.9~17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ubuntu-fan (Ubuntu Artful):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-artful
removed: verification-done
Jonathan Kamens (jik) wrote :

I did "systemctl restart NetworkManager" and observed errors from fanctl in my syslog. Then I upgraded ubuntu-fan to 0.12.9~17.10.1 and did "systemctl restart NetworkManager" again, and this time there were no such errors. However, I'm not certain that this is an adequate test, so I'm not changing the tags.

Stefan Bader (smb) wrote :

I think the previous test was as valid but to be completely sure I added an additional interface to a VM, installed ifupdown and assigned the additional interface to ifupdown control. Using 0.12.8~17.10.1 the error was observable and the status of the interface was incorrect. Upgrading to 0.12.9~17.10.1, ifup and ifdown worked correctly.

tags: added: verification-done verification-done-artful
removed: verification-needed verification-needed-artful
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ubuntu-fan - 0.12.9~17.10.1

---------------
ubuntu-fan (0.12.9~17.10.1) artful; urgency=medium

  [ Andy Whitcroft ]
  * fanctl: fix config_filename when the interface does not have a width
    (LP: #1737640)

 -- Stefan Bader <email address hidden> Tue, 30 Jan 2018 10:54:13 +0100

Changed in ubuntu-fan (Ubuntu Artful):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers