WrongPartitionSchemeError: /boot partition has not been created for some reasons

Bug #1465971 reported by magicboiz
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Kozhukalov
6.1.x
Fix Released
Critical
Vladimir Kozhukalov
7.0.x
Fix Released
Critical
Vladimir Kozhukalov

Bug Description

Hi

when trying to deploy Ubuntu/Qemu/NeutroGRE/Ceph/Murano/Ceilometer in a VM cluster of 3 controller + 3 ceph nodes + 1 compute, deployment fails almost immediatly: http://pastebin.com/qMpWB6Vf

Every VM runs under vmware esx, has 5 nics (eth0=admin/pxe, eth1=public, eth2=storage, eth3=private, eth4=mngt), network verification from fuel is OK, but the deployment fails. I've tried with Centos deployment, and it fails too.

Regards
J

Revision history for this message
magicboiz (magicboiz) wrote :

FUEL VERSION:
[root@fuel ~]# fuel --fuel-version
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-16_05-40-59
build_number: '277'
feature_groups:
- experimental
fuel-library_sha: 686b83ab9b1c4c6448c6b239472a49f46c46a4fc
fuel-ostf_sha: cd26f5e19af208322fb1e789688c7267c4b99294
fuelmain_sha: cd1adb0dc4a456a539f4399616f2936c7d80babf
nailgun_sha: e102638712d6391bb7e96ae6bdb928631b9a85ae
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-16_05-40-59
      build_number: '277'
      feature_groups:
      - experimental
      fuel-library_sha: 686b83ab9b1c4c6448c6b239472a49f46c46a4fc
      fuel-ostf_sha: cd26f5e19af208322fb1e789688c7267c4b99294
      fuelmain_sha: cd1adb0dc4a456a539f4399616f2936c7d80babf
      nailgun_sha: e102638712d6391bb7e96ae6bdb928631b9a85ae
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

Revision history for this message
magicboiz (magicboiz) wrote :
Download full text (17.4 KiB)

FUEL ENV SETTINGS:
[root@fuel ~]# fuel settings --env 1 --download
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
Settings configuration for environment with id=1 downloaded to /root/settings_1.yaml

[root@fuel ~]# cat settings_1.yaml
editable:
  access:
    email:
      description: Email address for Administrator
      label: Email
      regex:
        error: Invalid email
        source: ^\S+@\S+$
      type: text
      value: admin@localhost
      weight: 40
    metadata:
      label: Access
      weight: 10
    password:
      description: Password for Administrator
      label: Password
      regex:
        error: Empty password
        source: \S
      type: password
      value: admin
      weight: 20
    tenant:
      description: Tenant (project) name for Administrator
      label: Tenant
      regex:
        error: Invalid tenant name
        source: ^(?!services$)(?!nova$)(?!glance$)(?!keystone$)(?!neutron$)(?!cinder$)(?!swift$)(?!ceph$)(?![Gg]uest$)(?!.*
          +.*$).+
      type: text
      value: admin
      weight: 30
    user:
      description: Username for Administrator
      label: Username
      regex:
        error: Invalid username
        source: ^(?!services$)(?!nova$)(?!glance$)(?!keystone$)(?!neutron$)(?!cinder$)(?!swift$)(?!ceph$)(?![Gg]uest$)(?!.*
          +.*$).+
      type: text
      value: admin
      weight: 10
  additional_components:
    ceilometer:
      description: If selected, Ceilometer component will be installed
      label: Install Ceilometer
      type: checkbox
      value: true
      weight: 40
    heat:
      description: ''
      label: ''
      type: hidden
      value: true
      weight: 30
    metadata:
      label: Additional Components
      weight: 20
    mongo:
      description: If selected, You can use external Mongo DB as ceilometer backend
      label: Use external Mongo DB
      restrictions:
      - settings:additional_components.ceilometer.value == false
      type: checkbox
      value: false
      weight: 40
    murano:
      description: If selected, Murano component will be installed
      label: Install Murano
      restrictions:
      - cluster:net_provider != 'neutron'
      type: checkbox
      value: true
      weight: 20
    sahara:
      description: If selected, Sahara component will be installed
      label: Install Sahara
      type: checkbox
      value: false
      weight: 10
  common:
    auth_key:
      description: Public key(s) to include in authorized_keys on deployed nodes
      label: Public Key
      type: textarea
      value: ''
      weight: 70
    auto_assign_floating_ip:
      description: If selected, OpenStack will automatically assign a floating IP
        to a new instance
      label: Auto assign floating IP
      restrictions:
      - action: hide
        condition: cluster:net_provider == 'neutron'
      type: checkbox
      value: false
      weight: 40
    debug:
      description: Debug logging mode provides more information, but requires more
     ...

Changed in fuel:
importance: Undecided → High
milestone: none → 7.0
assignee: nobody → Fuel provisioning team (fuel-provisioning)
Revision history for this message
magicboiz (magicboiz) wrote :
Download full text (7.6 KiB)

FUEL ENV DOCKER-LOGS:
/var/log/docker-logs/fuel-agent-env-1.log: http://pastebin.com/T2dsunVA

notice:
2015-06-17 07:11:13.472 607 WARNING fuel_agent.utils.fs_utils [-] /tmp/tmp11oBMn.fuel-agent-image/boot is not a mountpoint, skipping umount
2015-06-17 07:11:13.473 607 DEBUG fuel_agent.utils.utils [-] Trying to execute command: mountpoint -q /tmp/tmp11oBMn.fuel-agent-image/
2015-06-17 07:11:13.490 607 WARNING fuel_agent.utils.utils [-] Failed to execute command: Unexpected error while running command.
Command: mountpoint -q /tmp/tmp11oBMn.fuel-agent-image/
Exit code: 1
Stdout: ''
Stderr: ''

NODES DOCKER-LOGS:

 /var/log/docker-logs/remote/node-1.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/JHUb1Fq1
 /var/log/docker-logs/remote/node-2.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/u26FFFDF
 /var/log/docker-logs/remote/node-3.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/NMCDYSiF
 /var/log/docker-logs/remote/node-4.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/AVQ7Xqye
/var/log/docker-logs/remote/node-5.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/FMFdG20D
/var/log/docker-logs/remote/node-6.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/iTP6Xwfx
/var/log/docker-logs/remote/node-7.domain.tld/bootstrap/fuel-agent.log: http://pastebin.com/qSnzHUNS

all nodes show the same warning.....

2015-06-17T07:12:11.186619+00:00 info: 2015-06-17 07:12:10.876 8684 DEBUG fuel_agent.drivers.nailgun [-] --- Preparing partition scheme ---
2015-06-17T07:12:11.186813+00:00 info: 2015-06-17 07:12:10.897 8684 DEBUG fuel_agent.drivers.nailgun [-] Looping over all disks in provision data
2015-06-17T07:12:11.186984+00:00 info: 2015-06-17 07:12:10.897 8684 DEBUG fuel_agent.drivers.nailgun [-] Processing disk sda
2015-06-17T07:12:11.187174+00:00 info: 2015-06-17 07:12:10.897 8684 DEBUG fuel_agent.drivers.nailgun [-] Adding gpt table on disk sda
2015-06-17T07:12:11.187350+00:00 info: 2015-06-17 07:12:10.897 8684 DEBUG fuel_agent.utils.utils [-] Trying to execute command: udevadm info --export-db
2015-06-17T07:12:11.187542+00:00 info: 2015-06-17 07:12:10.973 8684 DEBUG fuel_agent.utils.utils [-] Trying to execute command: udevadm info --query=property --export --name=/dev/sda
2015-06-17T07:12:11.187711+00:00 info: 2015-06-17 07:12:10.996 8684 DEBUG fuel_agent.utils.utils [-] Trying to execute command: blockdev --getsz --getro --getss --getpbsz --getsize64 --getiomin --getioopt --getra --getalignoff --getmaxsect /dev/sda
2015-06-17T07:12:11.187896+00:00 info: 2015-06-17 07:12:11.018 8684 DEBUG fuel_agent.drivers.nailgun [-] Adding bootloader stage0 on disk sda
2015-06-17T07:12:11.188080+00:00 info: 2015-06-17 07:12:11.020 8684 DEBUG fuel_agent.drivers.nailgun [-] Adding bios_grub partition on disk sda: size=24
2015-06-17T07:12:11.188271+00:00 info: 2015-06-17 07:12:11.020 8684 DEBUG fuel_agent.drivers.nailgun [-] Adding UEFI partition on disk sda: size=200
2015-06-17T07:12:11.188433+00:00 info: 2015-06-17 07:12:11.020 8684 DEBUG fuel_agent.drivers.nailgun [-] Looping over all volumes on disk sda
2015-06-17T07:12:11.188635+00:00 info: 2015-06-17 07:12:11.020 8684 DEBUG fuel_agent.drivers.nailgun [-] Proc...

Read more...

magicboiz (magicboiz)
no longer affects: fuel
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

The priority changed to Critical because it affects all types of deployments and in fact we can't deploy nothing.

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Timur, we run a huge amount of tests and it is not affect all types of deployments.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Yes, looks like the issue related to the count of HDDs on nodes, fuel team are working on it, we need to fix the issue in MOS 6.1 release, because it is regression.

Revision history for this message
magicboiz (magicboiz) wrote :

besides, I'm trying to generate a "fuel snapshot", but bug "httplib.InvalidURL: nonnumeric port: '8000Dump is timed out'" (https://bugs.launchpad.net/fuel/+bug/1427237) is still there, on #277 build.

J

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

On fuel-qa bare-metal lab all nodes which have one SSD/HDD weren't provisioned:

fuel_agent.cmd.agent WrongPartitionSchemeError: /boot partition has not been created for some reasons

so raising priority to Critical.

@magicboiz, most probably your issue with diagnostic snapshot is related to bug #1466045.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

yes, the same issue for me, I described separate bug for this (because on my env I can see only one snapshot): https://bugs.launchpad.net/fuel/+bug/1466045 - fuel team will help to fix this too :)

We already have some fix for the original issue (with provisioning):
https://review.openstack.org/#/c/192596

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

We need to also fix the issue in MOS 7.x - but looks like I have no access to assign new series here. Could anybody do this, please?

Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

Artem,

> On fuel-qa bare-metal lab all nodes which have one SSD/HDD weren't provisioned:

That's an unrelated problem, please file a different bug.

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

@Alexei,

I got an issue which was reported by Timur in bug #1466017, but Vladimir marked it as duplicate of this bug. Also I see the same errors (like on my environment) in comments above: https://bugs.launchpad.net/fuel/6.1.x/+bug/1465971/comments/3

Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

Oops, I was too fast. The first fragment of log (http://pastebin.com/T2dsunVA) is confusing and does not contain the actual error message

summary: - fuel 6.1-community.all (#277) deployment failure
+ WrongPartitionSchemeError: /boot partition has not been created for some
+ reasons
Revision history for this message
Mike Scherbakov (mihgen) wrote :

This bug seems to be a regression. Did we analyze on what is the cause?

Revision history for this message
magicboiz (magicboiz) wrote :

Can you upload the fix to a new https://ci.fuel-infra.org/view/ISO/job/6.1-community.all/ build?

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/193096

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/6.1)

Reviewed: https://review.openstack.org/193096
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=9bc5ca3f5e169d52cd1f83eb428773f35119362b
Submitter: Jenkins
Branch: stable/6.1

commit 9bc5ca3f5e169d52cd1f83eb428773f35119362b
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Jun 17 13:25:05 2015 +0300

    IBP: Fixed provision fail on a node with one disk

    To address some bugs about large amount of disks,
    huge disks, etc. we introduced workarounds in
    nailgun data driver in fuel agent.

    The issue is that when only one disk on a node,
    nailgun places /boot not on raid but on plain parition
    and flag boot_done is set only inside 'if raid' clause,
    but not inside 'if partition' clause.

    Change-Id: I8a598ccde661fae375bb102f36550fb53f1abbe8
    Closes-Bug: #1465971

Maksym Strukov (unbelll)
tags: added: on-verification
Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 6.1-525

{"build_id": "2015-06-19_13-02-31", "build_number": "525", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-06-19_13-02-31", "build_number": "525", "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "dbd54158812033dd8cfd7e60c3f6650f18013a37", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "a3998372183468f56019c8ce21aa8bb81fee0c2f"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "2e7a08ad9792c700ebf08ce87f4867df36aa9fab", "nailgun_sha": "dbd54158812033dd8cfd7e60c3f6650f18013a37", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "4fc55db0265bbf39c369df398b9dc7d6469ba13b", "astute_sha": "1ea8017fe8889413706d543a5b9f557f5414beae", "fuel-ostf_sha": "8fefcf7c4649370f00847cc309c24f0b62de718d", "release": "6.1", "fuelmain_sha": "a3998372183468f56019c8ce21aa8bb81fee0c2f"}

Steps:
1. Create environment: ubuntu, ha, neutron vlan
2. Add node with one hdd as compute
3. Add node with 3 hdd as compute. Unallocate two hdd's
4. Add controller
5. Deploy cluster

Expected:
Both two computes deployed successfull.

Revision history for this message
Maksym Strukov (unbelll) wrote :

Verified as fixed in 7.0-15

{"build_id": "2015-06-24_16-12-23", "build_number": "15", "release_versions": {"2014.2.2-7.0": {"VERSION": {"build_id": "2015-06-24_16-12-23", "build_number": "15", "api": "1.0", "fuel-library_sha": "7d19bc3783177aebf64fa4c2ae20d845cbd5348f", "nailgun_sha": "b74f847ec89c4bff1addb830704206dc503125f0", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "1b8574a7c4ea884862763a15c636b066d51f49e7", "astute_sha": "776157f722b13aff5f59bc098cf948793e6498ef", "fuel-ostf_sha": "69e7fa120e8efa7ed74d98efc63079d2f5c69d7b", "release": "7.0", "fuelmain_sha": "3b866d2ff3091a60362327028085fa62fd16c5a0"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "7d19bc3783177aebf64fa4c2ae20d845cbd5348f", "nailgun_sha": "b74f847ec89c4bff1addb830704206dc503125f0", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-7.0", "production": "docker", "python-fuelclient_sha": "1b8574a7c4ea884862763a15c636b066d51f49e7", "astute_sha": "776157f722b13aff5f59bc098cf948793e6498ef", "fuel-ostf_sha": "69e7fa120e8efa7ed74d98efc63079d2f5c69d7b", "release": "7.0", "fuelmain_sha": "3b866d2ff3091a60362327028085fa62fd16c5a0"}

tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.