Random ethX interfaces are down when a node with Ubuntu restarts

Bug #1390122 reported by Dennis Dmitriev
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Aleksander Mogylchenko
5.1.x
Fix Released
High
Aleksander Mogylchenko
6.0.x
Fix Committed
High
Aleksander Mogylchenko
6.1.x
Invalid
High
Aleksander Mogylchenko

Bug Description

Faced on Jenkins CI test http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.ubuntu.thread_3/17 , system test 'ceph_multinode_restart' .

There are no 'eth1' and 'eth2' interfaces (public and management networks) on the compute 'node-1' after node reset and booted.
That caused the inaccessible RabbitMQ from the compute node and as result there is failed OSTF test "Create volume and attach it to instance".

======= dmesg: http://paste.openstack.org/show/130126/ ================
[ 6.040744] init: network-interface (eth2) pre-start process (1230) killed by TERM signal
[ 6.041678] init: network-interface (eth1) pre-start process (1231) killed by TERM signal

======= ifconfig: http://paste.openstack.org/show/130127/

The similar bug described on the following link: http://askubuntu.com/questions/382354/second-network-interface-fails-to-start-pre-start-killed-by-term :
" If VM has several network interfaces, then several instances of open-iscsi init script executed on system boot. Several of them try to load iscsi kernel modules and start iscsi daemon simultaneously. So one finished successfully, others - failed and stop process of setting IP on their network interfaces. "

As was suggested, I've removed open-iscsi from if-up.d and if-down.d (open-iscsi is already added to the correct runlevels):

rm /etc/network/if-up.d/open-iscsi
rm /etc/network/if-down.d/open-iscsi

After this, the service open-iscsi starts a bit later but without issues:

[ 2.854663] 8021q: adding VLAN 0 to HW filter on device eth2
[ 2.857242] 8021q: adding VLAN 0 to HW filter on device eth4
[ 2.858095] 8021q: adding VLAN 0 to HW filter on device eth1
[ 2.863143] 8021q: adding VLAN 0 to HW filter on device eth0
[ 3.053185] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input1
[ 3.255104] openvswitch: module verification failed: signature and/or required key missing - tainting kernel
[ 3.255496] openvswitch: Open vSwitch switching datapath 1.10.2, built Nov 5 2014 14:18:31
[ 3.314766] init: udev-fallback-graphics main process (1628) terminated with status 1
[ 123.015589] init: failsafe main process (1290) killed by TERM signal
[ 123.038598] Loading iSCSI transport class v2.0-870.
[ 123.045391] iscsi: registered transport (tcp)
[ 123.060854] iscsi: registered transport (iser)
[ 123.063824] iscsid (2043): /proc/2043/oom_adj is deprecated, please use /proc/2043/oom_score_adj instead.

Steps to reproduce:
     1. Deploy any cluster configuration based on Ubuntu with several network interfaces. Assign different networks to different interfaces.
     2. Reboot nodes.
     3. Check 'dmesg' output for 'network-interface .. killed by TERM signal'

When open-iscsi starts right after network interfaces started, it can cause termination of some interfaces.

Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

seems like this issue was reproduced again https://bugs.launchpad.net/fuel/+bug/1383319

Changed in fuel:
assignee: nobody → Fuel for Openstack (fuel)
importance: Undecided → High
Changed in fuel:
status: New → Confirmed
assignee: Fuel for Openstack (fuel) → Fuel Library Team (fuel-library)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → MOS Linux (mos-linux)
status: Confirmed → Triaged
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Andrey: if this is a duplicate of bug #1383319 it should be marked as such, and all useful information (btw kudos to Dennis on a very extensive bug description!) copied from here to the older bug.

Bogdan: why do you believe this is a MOS Linux issue?

Changed in fuel:
status: Triaged → Incomplete
Revision history for this message
Michael Semenov (msemenov) wrote :

Bogdan, please clarify.

Changed in fuel:
milestone: 6.0 → 6.1
assignee: MOS Linux (mos-linux) → Bogdan Dobrelya (bogdando)
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Looks like this bug has nothing to the system configuration provided by puppet manifests. It looks related to "open-iscsi" package installation. http://askubuntu.com/questions/382354/second-network-interface-fails-to-start-pre-start-killed-by-term

Changed in fuel:
assignee: Bogdan Dobrelya (bogdando) → MOS Linux (mos-linux)
Changed in fuel:
status: Incomplete → Confirmed
tags: added: customer-found
Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

It seems to be an upstream bug (reported 4 years ago):
https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/669142

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

This behavior is fixed in Trusty (although the bug is still open), with this commit:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/open-iscsi/trusty/revision/38

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Backported package is here:
http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-3050/ubuntu/

Waiting for test results.

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package open-iscsi has been built for project packages/precise/open-iscsi
Package version == 2.0.873, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3630
project: packages/precise/open-iscsi
branch: 6.0.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: open-iscsi packages from precise-backports
status: patchset-created

Files placed on repository:
open-iscsi-utils_2.0.873-ubuntu1_all.deb
open-iscsi_2.0.873-ubuntu1_amd64.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-3630/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package open-iscsi has been built for project packages/precise/open-iscsi
Package version == 2.0.873, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3050
project: packages/precise/open-iscsi
branch: 5.1.2
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: open-iscsi packages from precise-backports
status: change-merged

Files placed on repository:
open-iscsi-utils_2.0.873-ubuntu1_all.deb
open-iscsi_2.0.873-ubuntu1_amd64.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package open-iscsi has been built for project packages/precise/open-iscsi
Package version == 2.0.873, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3630
project: packages/precise/open-iscsi
branch: 6.0.1
author: Aleksandr Mogylchenko
committer: Aleksandr Mogylchenko
subject: open-iscsi packages from precise-backports
status: change-merged

Files placed on repository:
open-iscsi-utils_2.0.873-ubuntu1_all.deb
open-iscsi_2.0.873-ubuntu1_amd64.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

Packages appeared on public mirrors.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.