Kernel panic and nodes gets rebooted continuously

Bug #1603615 reported by Denis Kostryukov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
High
Albert Syriy

Bug Description

Found an issue when we try to get correct console on bootstrap images with the below command in MOS 9.0

cobbler profile edit --name bootstrap --kopts="console=tty0 console=ttyS0,115200 iommu=off" --in-place

It worked well in MOS 7.0 and MOS 8.0, but since in MOS 9.0 there is no profile with name bootstrap we replaced it with

cobbler profile edit --name ubuntu_bootstrap --kopts="console=tty0 console=ttyS0,115200 iommu=off" --in-place

Then the following issue is observed and nodes were not discovered.

During boot from bootstrap image we get the kernel panic

After inspecting the init procedure we realized that initrd.img tries the following:
mount -t devtmpfs -o mode=0755 udev /dev
mount -n -o move /dev ${rootmnt}/dev
rm -rf /dev
ln -s ${rootmnt}/dev /dev

The rm command deletes both /dev and /root/dev directories and the ln creates a recursive symlink to itself.
Also strange that the downloaded fetch parameter is not used anywhere.

information type: Public → Private
information type: Private → Public
tags: added: customer-found
description: updated
description: updated
description: updated
Changed in mos:
importance: Undecided → High
Revision history for this message
Aleksey Zvyagintsev (azvyagintsev) wrote :

Hello, could you please provide output from:
cobbler profile report --name ubuntu_bootstrap |grep -i 'Kernel Options'

If it was default bootstrap, you should find option:
 'panic': '60',
If you need to disable panic reboot- you should remove this option

@asiryi could you suggest something more ?

Anton Matveev (amatveev)
Changed in mos:
milestone: none → 9.1
tags: added: sla1
Revision history for this message
Denis Kostryukov (dkostryukov) wrote :

Here is the output

[root@fuel ~]# cobbler profile report --name ubuntu_bootstrap |grep -i 'Kernel Options'
Kernel Options : {'net.ifnames': '1', 'console': ['ttyS0,9600', 'tty0'], 'LOGLEVEL': '8', 'ip': 'frommedia', 'boot': 'live', 'log_buf_len': '10M', 'toram': '~', 'ignore_loglevel': '~', 'biosdevname': '0', 'url': 'http://192.168.0.11:8000/api', 'components': '~', 'debug': '~', 'mco_pass': 'iE0TnswgtnOqg8xlLVEkasr0', 'mco_user': 'mcollective', 'panic': '60', 'fetch': 'http://192.168.0.11:8080/bootstraps/active_bootstrap/root.squashfs', 'ethdevice-timeout': '120', 'print_fatal_signals': '1'}
Kernel Options (Post Install) : {}

Revision history for this message
Albert Syriy (asyriy) wrote :

Could we reproduce the bug somehow with your instructions or get the image initrd.img?
Is the variable ${rootmnt} empty?

Changed in mos:
assignee: nobody → MOS Linux (mos-linux)
status: New → Confirmed
Changed in mos:
assignee: MOS Linux (mos-linux) → Albert Syriy (asyriy)
Revision history for this message
Albert Syriy (asyriy) wrote :

The bug has been reproduced on MOS 9.0 with ordinary boostrap.
The most plausible cause of the issue is console parameters. See the bug https://bugs.launchpad.net/mos/+bug/1493767

Revision history for this message
Denis Kostryukov (dkostryukov) wrote :

Hello!
Here is initrd.img

Revision history for this message
Albert Syriy (asyriy) wrote :

Hello,

The actual culprit of the issue is the kernel parameter "toram", which is implicitly changing during applying your command:

cobbler profile edit --name ubuntu_bootstrap --kopts="console=tty0 console=ttyS0,115200 iommu=off" --in-place

Cobbler changes the console and iommu values _AND_ changes the 'toram' paramter from 'toram' to 'toram=~' as well. It triggers corruption of file system into the RAM.

Actually 'toram' could be a flag (without assigning) meaning that file system should be download into RAM or it could be a parameter with a value ('toram=filesystem.file') meaning that special filesystem file should be downloaded into RAM.

The workaround is to mention 'toram' parameter in the command "cobbler profile edit ..." each time when you update another parameters. In such case the 'toram' will not be implicitly changed.

Here is correct command:

cobbler profile edit --name ubuntu_bootstrap --kopts="console=tty0 console=ttyS0,115200 iommu=off toram" --in-place

Revision history for this message
Albert Syriy (asyriy) wrote :

The upstream bug has been reported for cobbler:

https://bugs.launchpad.net/ubuntu/+source/cobbler/+bug/1606285

Albert Syriy (asyriy)
Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/centos7/cobbler (9.0)

Fix proposed to branch: 9.0
Change author: Albert Syriy <email address hidden>
Review: https://review.fuel-infra.org/23714

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to packages/centos7/cobbler (9.0)

Reviewed: https://review.fuel-infra.org/23714
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 2d8a911821095cda480e3ef61846dd819d6e6a78
Author: Albert Syriy <email address hidden>
Date: Wed Jul 27 14:39:14 2016

Fix for mangling kernel options when the key --in-place is used. LP #1603615

Change-Id: I74b1f1b88516c4e303e0ae63025967d7601625d9
Closes-Bug: #1603615

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on packages/centos7/cobbler (master)

Change abandoned by Albert Syriy <email address hidden> on branch: master
Review: https://review.fuel-infra.org/23712
Reason: Will try commit the changes into cobbler upstream (for master).

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
tags: added: on-verification
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

Verified.

Changed in mos:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.