node failed to reboot

[Big Switch Networks]

We are trying to deploy fuel6.1 with build 521, trying for single node cluster.

1.) bootstrap did go through fine.
2.) While installing Ubuntu, reboot of the node failed, and stuck at below
kvm: exiting hardware virtualization
sd 0:0:0:0: [sda] Synchronizing SCSI cache

Looking at few online forums, suggestions made to update the bios, our bios was the latest from DELL. We are using Dell R220 HW and using 1.4.0 bios revision. In addition to this suggestion, there were few mentioning about setting the acpi to off. Not sure if that helps...I have copied /var/log/* from the node and attached here. Please let us know if you need more information.

Workaround we tried:

1.) Manually powering off/on the node, worked.

[Aleksander Mogylchenko (amogylchenko)]

Could you please clarify, how often this happens?

[Big Switch Networks (fuel-bugs-internal)]

It happens all the time on Dell R220 servers .

[Big Switch Networks (fuel-bugs-internal)]

Thanks , we have around 500 R220 servers to be brought online with Fuel 6.1 , Would appreciate any quick turnaround or patch.

[Albert Syriy (asyriy)]

Could we "borrow" the server for the bug investigation ?

[Albert Syriy (asyriy)]

Need access to the server to investigate the issue.

[Pavel Boldin (pboldin)]

The problem was that bnx2x device driver is not removing the devices during the shutdown, leaving some resources initialized (e.g. kernel timers and PCI resources).

Fixed that by backporting the following upstream changesets:

* Sun Jun 28 2015 Pavel Boldin <email address hidden> - 3.10.55-mira4
- Fix the Broadcom NetXtreme II reboot kernel hang by backporting the
  listed commits from the upstream:
    * b030ed2fdc8a396dba71e4d550236a0f1bb38b40
      bnx2x: Implement PCI shutdown
    * d9aee591b0f06bd44cd577b757d3f267bc35fe4d
      bnx2x: Don't release PCI bars on shutdown

Please see the https://review.fuel-infra.org/8839/ .

[Pavel Boldin (pboldin)]

Current status is:

1. Custom ISO built.
2. Custom ISO passes both CentOS and Ubuntu BVT.

[Michael Semenov (msemenov)]

Change request for CentOS - https://review.fuel-infra.org/#/c/9734/

[Michael Semenov (msemenov)]

Fix prepared, so assigning to maintenance team.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/9734
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: bcff343db585abf90256dff41aecdeeed1aa4246
Author: Pavel Boldin <email address hidden>
Date: Wed Jul 22 13:46:32 2015

Bump version to match kernel-lt

Bump openvswitch-kmod version so it matches the kernel-lt version after
https://review.fuel-infra.org/#q,I8648f704a942883479c66ba62068870a70135ccd,n,z

Change-Id: I8fcfa1b56d845e73e5f0e0760f47dc72bcba6c73
Related-Bug: #1467671

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/8839
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: 22cde41df6e5a831d910b1654cbd4d03548e6acf
Author: Pavel Boldin <email address hidden>
Date: Mon Jun 29 11:29:56 2015

Fix Broadcast NetXtreme II reboot hung

Broadcast NetXtreme II driver was lacking the shutdown handler required
to disable the devices. This led to a hang during the reboot because the
enabled driver code was attempting to use the freed resources.

Change-Id: I8648f704a942883479c66ba62068870a70135ccd
Closes-Bug: #1467671

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 6.1
Change author: Vitaly Sedelnik <email address hidden>
Review: https://review.fuel-infra.org/10208

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10208
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: e52257594df40fc567131e145464421168fc9ed3
Author: Vitaly Sedelnik <email address hidden>
Date: Fri Aug 7 10:32:58 2015

Empty commit to rebuild packages for change I8648f704a942883479c66ba62068870a70135ccd

Related-Bug: #1467671
Closes-Bug: #1482562

Change-Id: Ie2cfffecd6bc15c630f5bd0dca8a85fa11a9a7e0

[Vadim Rovachev (vrovachev)]

Verified on 6.1.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 7.0
Change author: Pavel Boldin <email address hidden>
Review: https://review.fuel-infra.org/14372

[Ivan Suzdal (isuzdal)]

In newer kernels (ubuntu - 3.13.0/centos - 3.10.0) this patch already included. So it's probably should be moved to "won't fix" state.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14372
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: 7.0

Commit: 1fcb648d4f49c71d7bb134898f3019dcd259f6f0
Author: Pavel Boldin <email address hidden>
Date: Wed Dec 2 13:27:03 2015

Fix Broadcast NetXtreme II reboot hung

Broadcast NetXtreme II driver was lacking the shutdown handler required
to disable the devices. This led to a hang during the reboot because the
enabled driver code was attempting to use the freed resources.

Kernel version has been bumped to mos6.

Change-Id: I8648f704a942883479c66ba62068870a70135ccd
Closes-Bug: #1467671
(cherry picked from commit 22cde41df6e5a831d910b1654cbd4d03548e6acf)

[Dmitry (dtsapikov)]

Verified on 7.0 release