mount: special device /dev/sdb does not exist

[Dan Prince]

As of today it looks like all jobs fail due to a missing Ephemeral partition:

mount: special device /dev/sdb does not exist

----

This Nova commit looks suspicious: 7f8128f87f5a2fa93c857295fb7e4163986eda25
"Add the swap and ephemeral BDMs if needed"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/190622

[Matt Riedemann (mriedem)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwibW91bnQ6IHNwZWNpYWwgZGV2aWNlIC9kZXYvc2RiIGRvZXMgbm90IGV4aXN0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzQwMzc5ODAyODd9

[Matt Riedemann (mriedem)]

http://logs.openstack.org/02/190602/1/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/166e85a/logs/overcloud-controller0_logs/os-collect-config.txt.gz

Jun 11 12:59:21 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:21.325 1343 ERROR os-collect-config [-] Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 1
Jun 11 12:59:21 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:21.325 1343 WARNING os-collect-config [-] Sleeping 30.00 seconds before re-exec.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.090 1343 WARNING os_collect_config.cfn [-] No metadata_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os_collect_config.heat [-] No auth_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os_collect_config.request [-] No metadata_url configured.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.091 1343 WARNING os-collect-config [-] Source [request] Unavailable.
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.092 1343 WARNING os_collect_config.local [-] /var/lib/os-collect-config/local-data not found. Skipping
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: 2015-06-11 12:59:52.092 1343 WARNING os_collect_config.local [-] No local metadata found (['/var/lib/os-collect-config/local-data'])
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,125] (os-refresh-config) [INFO] Starting phase pre-configure
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: dib-run-parts Thu Jun 11 12:59:52 UTC 2015 Running /usr/libexec/os-refresh-config/pre-configure.d/00-fix-ephemeral-mount
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: ++ os-apply-config --key block-device-mapping.ephemeral0 --type raw --key-default ''
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + ephemeraldisk=/dev/sdb
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + '[' -n /dev/sdb ']'
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + grep -q '^/dev/sdb ' /proc/mounts
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: + mount /dev/sdb /mnt
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: mount: special device /dev/sdb does not exist
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,445] (os-refresh-config) [ERROR] during pre-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/pre-configure.d']' returned non-zero exit status 32]
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config:
Jun 11 12:59:52 overcloud-controller0-echfxrozz2sa os-collect-config: [2015-06-11 12:59:52,445] (os-refresh-config) [ERROR] Aborting...

[John Garbutt (johngarbutt)]

This can't be critical in nova, as it doesn't break any of our tests, and only affects a subset of users

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/205271

[Ben Nemec (bnemec)]

This was caused by the commit mentioned in the description changing the block device mapping when ephemeral partitions are in use. The nova metadata server started to return /dev/sdb as the ephemeral0 path, whereas before it returned /dev/sda1, which was the default set in the Ironic driver. The patch caused this default to be overridden.

My proposed patch only fixes the regression, but there's still an issue here because if the ephemeral partition doesn't end up on /dev/sda1 the metadata will still be wrong. I've opened https://bugs.launchpad.net/nova/+bug/1478065 to track that.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/190622
Reason: Yeah, no response to this in nearly 2 months. Also, bnemec reported a new bug to follow up on (bug 1478065) so let's just track that for now.

[Matt Riedemann (mriedem)]

The workaround in tripleo is here: https://review.openstack.org/#/c/190629/

[Ben Nemec (bnemec)]

Dropping the importance for tripleo since we don't use ephemeral partitions anymore.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ben Nemec (<email address hidden>) on branch: master
Review: https://review.openstack.org/205271
Reason: Yeah, I keep telling myself I'll come back to this, but at this point I think that's pretty clearly a lie. :-)

I'll restore if I get a chance to revisit it.

[Markus Zoeller (markus_z) (mzoeller)]

Cleanup
=======

There are no open reviews for this bug report since more than 2 weeks.
To signal that to other contributors which might provide patches for
this bug, I switch the status from "In Progress" to "Confirmed" and
remove the assignee.
Feel free to add yourself as assignee and to push a review for it.

[Jay Pipes (jaypipes)]

What's the deal with this? Can we close the Nova bug on it? Looks like this was something that was "fixed" in Triple-O, no?

[Ben Nemec (bnemec)]

I'm pretty sure the bug still exists in Nova, but it doesn't seem like anyone particularly cares anymore. TripleO "fixed" it by not using ephemeral partitions anymore so it doesn't affect us.

I guess this one could be closed against Nova though as there are two other bugs open related to it that are more general, so this one is redundant now.