mount: special device /dev/sdb does not exist

Bug #1464239 reported by Dan Prince on 2015-06-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Unassigned
tripleo
Low
Dan Prince

Bug Description

As of today it looks like all jobs fail due to a missing Ephemeral partition:

mount: special device /dev/sdb does not exist

----

This Nova commit looks suspicious: 7f8128f87f5a2fa93c857295fb7e4163986eda25
"Add the swap and ephemeral BDMs if needed"

Dan Prince (dan-prince) on 2015-06-11
Changed in tripleo:
assignee: nobody → Dan Prince (dan-prince)
status: New → In Progress
Changed in nova:
assignee: nobody → Dan Prince (dan-prince)
Changed in tripleo:
importance: Undecided → Critical
Changed in nova:
status: New → Triaged

Fix proposed to branch: master
Review: https://review.openstack.org/190622

Changed in nova:
status: Triaged → In Progress
Dan Prince (dan-prince) on 2015-06-11
Changed in nova:
importance: Undecided → Critical
Matt Riedemann (mriedem) wrote :

http://logs.openstack.org/02/190602/1/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/166e85a/logs/overcloud-controller0_logs/os-collect-config.txt.gz

Jun 11 12:59:21 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:21.325 1343 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1
Jun 11 12:59:21 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:21.325 1343 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.090 1343 WARNING os_collect_config.cfn [-] No metadata_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os_collect_config.heat [-] No auth_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os_collect_config.request [-] No metadata_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os-collect-config [-] Source [request] Unavailable.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.092 1343 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.092 1343 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data'])
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,125] (os-refresh-config) [INFO] Starting phase pre-configure
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: dib-run-parts Thu Jun 11 12:59:52 UTC 2015 Running /usr/libexec/os-refresh-config/pre-configure.d/00-fix-ephemeral-mount
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: ++ os-apply-config --key block-device-mapping.ephemeral0 --type raw --key-default ''
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + ephemeraldisk=/dev/sdb
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + '[' -n /dev/sdb ']'
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + grep -q '^/dev/sdb ' /proc/mounts
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + mount /dev/sdb /mnt
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: mount: special device /dev/sdb does not exist
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,445] (os-refresh-config) [ERROR] during pre-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/pre-configure.d']' returned non-zero exit status 32]
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config:
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,445] (os-refresh-config) [ERROR] Aborting...

John Garbutt (johngarbutt) wrote :

This can't be critical in nova, as it doesn't break any of our tests, and only affects a subset of users

Changed in nova:
importance: Critical → High

Fix proposed to branch: master
Review: https://review.openstack.org/205271

Changed in nova:
assignee: Dan Prince (dan-prince) → Ben Nemec (bnemec)
Ben Nemec (bnemec) wrote :

This was caused by the commit mentioned in the description changing the block device mapping when ephemeral partitions are in use. The nova metadata server started to return /dev/sdb as the ephemeral0 path, whereas before it returned /dev/sda1, which was the default set in the Ironic driver. The patch caused this default to be overridden.

My proposed patch only fixes the regression, but there's still an issue here because if the ephemeral partition doesn't end up on /dev/sda1 the metadata will still be wrong. I've opened https://bugs.launchpad.net/nova/+bug/1478065 to track that.

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/190622
Reason: Yeah, no response to this in nearly 2 months. Also, bnemec reported a new bug to follow up on (bug 1478065) so let's just track that for now.

Matt Riedemann (mriedem) wrote :

The workaround in tripleo is here: https://review.openstack.org/#/c/190629/

Changed in tripleo:
status: In Progress → Fix Committed
Changed in nova:
status: In Progress → Invalid
Matt Riedemann (mriedem) on 2015-07-29
Changed in nova:
status: Invalid → In Progress
Ben Nemec (bnemec) wrote :

Dropping the importance for tripleo since we don't use ephemeral partitions anymore.

Changed in tripleo:
importance: Critical → Low
Steven Hardy (shardy) on 2016-04-21
Changed in tripleo:
status: Fix Committed → Fix Released

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/205271
Reason: Yeah, I keep telling myself I'll come back to this, but at this point I think that's pretty clearly a lie. :-)

I'll restore if I get a chance to revisit it.

Cleanup
=======

There are no open reviews for this bug report since more than 2 weeks.
To signal that to other contributors which might provide patches for
this bug, I switch the status from "In Progress" to "Confirmed" and
remove the assignee.
Feel free to add yourself as assignee and to push a review for it.

Changed in nova:
status: In Progress → Confirmed
assignee: Ben Nemec (bnemec) → nobody
Yuvaraj (yuva4591) on 2016-08-18
Changed in nova:
assignee: nobody → Yuvaraj (yuva4591)
assignee: Yuvaraj (yuva4591) → nobody
Jay Pipes (jaypipes) wrote :

What's the deal with this? Can we close the Nova bug on it? Looks like this was something that was "fixed" in Triple-O, no?

Changed in nova:
status: Confirmed → Incomplete
Ben Nemec (bnemec) wrote :

I'm pretty sure the bug still exists in Nova, but it doesn't seem like anyone particularly cares anymore. TripleO "fixed" it by not using ephemeral partitions anymore so it doesn't affect us.

I guess this one could be closed against Nova though as there are two other bugs open related to it that are more general, so this one is redundant now.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers