Cloud vm not pingable after overcloud upgrade

[Cian O'Driscoll]

Steps to repo:

1) Run devtest
2) Change overcloud control and compute image md5sum
3) Run devtest_overcloud

This will cause the overcloud to be rebuilt with the new images.

Guest vm will no longer be pingable after the overcloud nodes come back up

Logs:

devtest_overcloud

+ wait_for 30 10 ping -c 1 $'192.0.2.46\r'
Timing out after 300 seconds:
COMMAND=ping -c 1 192.0.2.46
 (192.0.2.46) 56(84) bytes of data.
From 192.0.2.46 icmp_seq=1 Destination Host Unreachable

 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

On the overcloud compute host

libvirt-bin.log

/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.
/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.
/usr/sbin/libvirtd: error: Unable to initialize network sockets. Check /var/log/messages or run without --daemon for more info.

neutron-openvswitch-agent.log

2014-05-01 10:37:06.041 2079 CRITICAL neutron [-] Unexpected boolean value ''
2014-05-01 10:37:06.894 2136 CRITICAL neutron [-] Unexpected boolean value ''
2014-05-01 10:37:07.908 2147 CRITICAL neutron [-] Unexpected boolean value ''
2014-05-01 10:38:21.526 2424 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file)
2014-05-01 10:38:22.090 2424 CRITICAL neutron [-] Trying to re-send() an already-triggered event.

nova-compute.log

2014-05-01 10:38:24.139 3405 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.0.2.36:5672
2014-05-01 10:38:54.715 3405 ERROR nova.virt.libvirt.driver [-] [instance: 2553ad5c-90fb-44ea-8736-9d5c7c29266c] During wait destroy, instance disappeared.
2014-05-01 10:38:57.306 3405 WARNING nova.compute.manager [req-da0cc3dd-0953-4bca-aa72-9f17b9fd93dd None] Found 1 in the database and 0 on the hypervisor.
2014-05-01 10:39:00.627 3405 ERROR nova.virt.libvirt.driver [-] [instance: 2553ad5c-90fb-44ea-8736-9d5c7c29266c] During wait destroy, instance disappeared.

[Cian O'Driscoll (dricco)]

From the console mountall is failing to run

......
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[ 14.618056] init: mountall main process (193) terminated with status 127
..........

[James Polley (tchaypo)]

Cian, I'm assuming that you were chasing this bug yourself, and the lack of updates probably means it hasn't been a problem for you lately?

If this isn't the case please update the bug with more details and assign to me.

[Derek Higgins (derekh)]

Closing due to lack of additional detail, if this is still a problem please feel free to reopen