Overcloud upgrade failing due to tftp issue on one of the overcloud nodes

Bug #1347514 reported by Cian O'Driscoll
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
New
Undecided
Unassigned

Bug Description

+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 2247b499-02ef-4001-b95e-f04a77cb7419 | overcloud | UPDATE_IN_PROGRESS | 2014-07-22T16:44:05Z |
+--------------------------------------+------------+--------------------+----------------------+
+ USER_IMG_NAME=user.qcow2
+ USE_CIRROS=0
+ '[' 0 '!=' 0 ']'
+ TEST_IMAGE_DIB_EXTRA_ARGS=
+ '[' '!' -e /root/.cache/tripleo/user.qcow2 -o 1 == 0 ']'
+ echo 'Waiting for the overcloud stack to be ready'
Waiting for the overcloud stack to be ready
+ wait_for_stack_ready 360 10 overcloud
Command output matched '(CREATE|UPDATE)_FAILED'. Exiting...
root@stratus5:~/.cache/tripleo# nova list
+--------------------------------------+-------------------------------------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------------------+--------+------------+-------------+---------------------+
| 429c8dd9-49ab-4967-a377-485cbbf0a30b | overcloud-NovaCompute0-5cuekfxlcndi | ACTIVE | - | Running | ctlplane=192.0.2.38 |
| 1f1523da-324a-42f0-8c0a-4a42a0f938e8 | overcloud-NovaCompute1-axzsysjpuim5 | ERROR | - | NOSTATE | ctlplane=192.0.2.35 |
| f85b092d-072b-411c-96d9-02a346fc3122 | overcloud-controller0-wtpuxd5zutjt | ACTIVE | - | Running | ctlplane=192.0.2.39 |
+--------------------------------------+-------------------------------------+--------+------------+-------------+---------------------+

nova-baremetal-deploy-helper.log on undercloud

2014-07-22 16:46:27.226 5495 ERROR nova.virt.baremetal.deploy_helper [-] deployment to node 9 failed
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper Traceback (most recent call last):
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 288, in run
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper deploy(**params)
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 247, in deploy
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper image_path, preserve_ephemeral)
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 209, in work_on_disk
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper if not is_block_device(dev):
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 112, in is_block_device
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper s = os.stat(dev)
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper OSError: [Errno 2] No such file or directory: '/dev/disk/by-path/ip-192.0.2.37:3260-iscsi-iqn-f85b092d-072b-411c-96d9-02a346fc3122-lun-1'
2014-07-22 16:46:27.226 5495 TRACE nova.virt.baremetal.deploy_helper
192.0.2.38 - - [22/Jul/2014 16:46:28] "POST / HTTP/1.1" 200 0
192.0.2.39 - - [22/Jul/2014 16:48:30] "POST / HTTP/1.1" 200 0
192.0.2.35 - - [23/Jul/2014 09:20:57] "POST / HTTP/1.1" 200 0
2014-07-23 09:21:33.720 5495 ERROR nova.virt.baremetal.deploy_helper [-] deployment to node 5 failed
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper Traceback (most recent call last):
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 288, in run
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper deploy(**params)
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 247, in deploy
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper image_path, preserve_ephemeral)
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 209, in work_on_disk
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper if not is_block_device(dev):
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/cmd/baremetal_deploy_helper.py", line 112, in is_block_device
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper s = os.stat(dev)
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper OSError: [Errno 2] No such file or directory: '/dev/disk/by-path/ip-192.0.2.35:3260-iscsi-iqn-1f1523da-324a-42f0-8c0a-4a42a0f938e8-lun-1'
2014-07-23 09:21:33.720 5495 TRACE nova.virt.baremetal.deploy_helper
192.0.2.39 - - [23/Jul/2014 09:21:45] "POST / HTTP/1.1" 200 0
192.0.2.38 - - [23/Jul/2014 09:22:24] "POST / HTTP/1.1" 200 0

Managed to grab this from the virt-manager (it's ocr converted)

- calibration: 2666.764 MHz 3.1411711 input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042 seriol/input/input4 ul 23 09:20:48 192 user.info kernel: 3.1411711 input: ImExPS/2 Generic Exp lorer Mouse as /devices/platform/i8042/seriol/input/input4 letwork ready .tart iSCSI target on /dev/sda aiting for tgtd socket...found ul 23 09:20:52 192 daemon.info tgtd: semkey Ox610232a5 ul 23 09:20:52 192 daemon.warn tgtd: tgtd daemon started, pid:194 ul 23 09:20:52 192 daemon.warn tgtd: tgtd logger started, pid:195 debug:0 ul 23 09:20:52 192 daemon.err tgtd: iser_ib_init(3351) Failed to initialize ROM load kernel modules? ul 23 09:20:52 192 daemon.err tgtd: work_timer_start(150) use signal based sche uler ul 23 09:20:52 192 daemon.err tgtd: bs_init_signalfd(271) could not open backin ..-store module directory /usr/lib/tgt/backing-store ul 23 09:20:52 192 daemon.err tgtd: bs_init(390) use signalfd notification request boot server to deploy image connecting to 192.0.2.3:10000 (192.0.2.3:10000) ul 23 09:20:53 192 daemon.err tgtd: device_mgmt(246) sz:14 params:path,/dev/sda ul 23 09:20:53 192 daemon.err tgtd: bs_thread_open(412) 16
aiting for notice of complete

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.