Boot sector signature not found in case with changed disks configuration for Ceph node

[Anastasia Palkina]

"build_id": "2015-04-10_22-54-31",
"ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423",
"build_number": "304",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-10_22-54-31", "ostf_sha": "c2a76a60ec4ebbd78e508216c2e12787bf25e423", "build_number": "304", "api": "1.0", "nailgun_sha": "69547a71abb4696df7e6f44b1f7864b0535f2df7", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c", "astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "8daac234aea6ac0a98f27871deec039f74f6fdab", "fuellib_sha": "867028fe78837dc2e4635a2cbb976782856964d0"}}}, "auth_required": true,
"api": "1.0",
"nailgun_sha": "69547a71abb4696df7e6f44b1f7864b0535f2df7",
 "openstack_version": "2014.2-6.1",
"production": "docker",
"python-fuelclient_sha": "9208ff4a08dcb674ce2df132399a5aa3ddfac21c",
"astute_sha": "d96a80b63198a578b2c159edbd76048819039eb0",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "8daac234aea6ac0a98f27871deec039f74f6fdab",
"fuellib_sha": "867028fe78837dc2e4635a2cbb976782856964d0"

1. Create new environment (Ubuntu)
2. Choose Neutron, Vlan
3. Choose Ceph for images and Ceph for volumes
4. Choose image based provisioning
5. Add 3 controllers, 1 compute and 2 ceph
6. For Ceph nodes change sda (see screen)
7. Start deployment
8. Ceph nodes hangs because: "Boot sector signature not found (unbootable disk/partition?) (see screen)

Logs are here: https://drive.google.com/a/mirantis.com/file/d/0B6SjzarTGFxaUlNsX2tWbzFTakU/view?usp=sharing

[root@fuel ~]# fuel node
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|--------------|------------------|---------|------------|-------------------|------------|---------------|--------|---------
13 | provisioning | Untitled (3a:b0) | 3 | 10.20.0.5 | 08:00:27:ba:3a:b0 | ceph-osd | | True | 3
9 | provisioned | Untitled (28:e8) | 3 | 10.20.0.16 | 08:00:27:f9:28:e8 | controller | | True | 3
10 | provisioned | Untitled (d3:dc) | 3 | 10.20.0.15 | 08:00:27:bd:d3:dc | controller | | True | 3
11 | provisioned | Untitled (d6:61) | 3 | 10.20.0.10 | 08:00:27:8b:d6:61 | compute | | True | 3
12 | provisioning | Untitled (49:82) | 3 | 10.20.0.9 | 08:00:27:02:49:82 | ceph-osd | | True | 3
8 | provisioned | Untitled (6e:28) | 3 | 10.20.0.13 | 08:00:27:3b:6e:28 | controller | | True | 3
17 | discover | Untitled (1a:89) | None | 10.20.0.14 | 08:00:27:7d:1a:89 | | | True | None
15 | discover | Untitled (9d:ec) | None | 10.20.0.11 | 08:00:27:0d:9d:ec | | | True | None
16 | discover | Untitled (14:b5) | None | 10.20.0.8 | 08:00:27:35:14:b5 | | | True | None
14 | discover | Untitled (fc:1a) | None | 10.20.0.12 | 08:00:27:c8:fc:1a | | | True | None
[root@fuel ~]#

[Leontii Istomin (listomin)]

the same issue for nodes with one role - controller.
I tried to deploy the following configuration:
Ubuntu,IBP,HA,Neutron-gre,Ceph-all,Nova-quotas,Nova-debug,6.1_309
Controllers:3 Computes:197

All computenodes (with ceph role) has been provisioned successfully, but all controllers - no (ipmi screenshot is attached).

[Dina Belova (dbelova)]

Wonder if we need to move this to critical. We're not able to deploy env with default IBP - I guess this looks as smth critical

[Ryan Moe (rmoe)]

The issue occurs when using a Ceph journal partition. The agent calculates how to subdivide the journal space [0] and this calculation results in a floating point number (e.g 1 journal volume of 10GB and 3 osds will result in 3 journal partitions of size 3333.33). This size gets used in a dd command which fails. See the fuel-agent log [1] for a failed node.

Leontiy, could you please attach the bootstrap/fuel-agent.log from one of your failed nodes?

[0] https://github.com/stackforge/fuel-web/blob/master/fuel_agent/fuel_agent/drivers/nailgun.py#L177-L181
[1] http://paste.openstack.org/show/204159/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/174662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174662
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=9f22be0096b55dd932527fb0ca84ab2484b4cb5c
Submitter: Jenkins
Branch: master

commit 9f22be0096b55dd932527fb0ca84ab2484b4cb5c
Author: Ryan Moe <email address hidden>
Date: Thu Apr 16 17:33:07 2015 -0700

    Ensure ratio calculated for Ceph journals is an integer

    When creating journal paritions an osd-to-journal ratio is
    calculated. This ratio is later used to determine the size
    of each individual journal partition. The ratio was a floating
    point number which would result in floating point partition
    sizes causing dd commands to fail.

    Change-Id: Ide1311de3dd3c4a859b5d9b00f3e86912489d761
    Closes-bug: #1444459

[Anastasia Palkina (apalkina)]

Verified on ISO #338 on VBox

"build_id": "2015-04-22_22-54-31", "build_number": "338", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-22_22-54-31", "build_number": "338", "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"

[Alexander Evseev (aevseev)]

I have this issue with Fuel 6.1 build 352.

# fuel --fuel-version
DEPRECATION WARNING: file /etc/fuel/client/config.yaml is found and will be used as a source for settings. However, it deprecated and will not be used by default in the ongoing version of python-fuelclient.
api: '1.0'
astute_sha: c1793f982fda7e3fc7b937ccaa613c649be6a144
auth_required: true
build_id: 2015-04-26_22-54-38
build_number: '352'
feature_groups:
- mirantis
fuel-library_sha: 6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7
fuel-ostf_sha: b38602c841deaa03ddffc95c02f319360462cbe3
fuelmain_sha: 01288380950bc89d572cf5902141c9a393ada950
nailgun_sha: f9f17fe395f3e38c27d3d484302728a37ad8012d
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 2b311b3b82a1e2df1dc3484a0f37e282273cd988
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: c1793f982fda7e3fc7b937ccaa613c649be6a144
      build_id: 2015-04-26_22-54-38
      build_number: '352'
      feature_groups:
      - mirantis
      fuel-library_sha: 6bdf783e2bffdce80ecffcca2915e6d32a8ccdd7
      fuel-ostf_sha: b38602c841deaa03ddffc95c02f319360462cbe3
      fuelmain_sha: 01288380950bc89d572cf5902141c9a393ada950
      nailgun_sha: f9f17fe395f3e38c27d3d484302728a37ad8012d
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 2b311b3b82a1e2df1dc3484a0f37e282273cd988
      release: '6.1'

Two virtual nodes: one controller + Ceph OSD and one compute + Ceph OSD, both not loading.
More info about partitioning:

# fuel node --node 11 --disk --download
Node attributes for disks were written to:
/root/node_11/disks.yaml

# cat /root/node_11/disks.yaml
- extra:
  - disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0
  id: disk/by-path/pci-0000:02:03.0-scsi-0:0:0:0
  name: sda
  size: 101836
  volumes:
  - name: os
    size: 28672
  - name: cephjournal
    size: 0
  - name: ceph
    size: 73164

# fuel node --node 12 --disk --download
Node attributes for disks were written to:
/root/node_12/disks.yaml

# cat /root/node_12/disks.yaml
- extra:
  - disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0
  id: disk/by-path/pci-0000:02:03.0-scsi-0:0:0:0
  name: sda
  size: 101772
  volumes:
  - name: os
    size: 28672
  - name: vm
    size: 36550
  - name: cephjournal
    size: 0
  - name: ceph
    size: 36550

[Ryan Moe (rmoe)]

Alexander, can you please provide a diagnostic snapshot for your environment?

[Alexander Evseev (aevseev)]

On second try result slightly different — both nodes hangs on "Booting..." with 100% CPU each

[Alexander Evseev (aevseev)]

Same result for classic installation (cobbler)

[Alexander Evseev (aevseev)]

Sorry about erroneous confirmation.
It was my mistake in VMs config.

All from message #11 is untruly.