I have been grinding up against this one for a couple days now also. Not sure if its the same or even not a storage issue but figured I would post details.

Hosts boot, things look great but Arps seemed to get snuffed out by the action, so went to noop.firewall. Then everything goes south. 

localrc: 
HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.0/24
FLOATING_RANGE=172.31.246.128/25
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.libvirt.firewall.IptablesFirewallDriver


Trying to get around Arps being dropped from controller to host. 
Controller = 172.24.17.220
VM = 172.24.17.3

From 2 differnet phy hosts w/same config:
in_port(0),eth(src=66:70:9c:83:be:42,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.24.17.220,tip=172.24.17.3,op=1,sha=66:70:9c:83:be:42,tha=00:00:00:00:00:00), packets:73, bytes:3066, used:0.840s, actions:drop

in_port(0),eth(src=e8:39:35:c4:70:f4,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=172.24.17.225,tip=172.24.17.130,op=1,sha=e8:39:35:c4:70:f4,tha=00:00:00:00:00:00), packets:1, bytes:42, used:0.825s, actions:2
in_port(2),eth(src=00:1e:c9:dd:bd:15,dst=01:80:c2:00:00:00), packets:95, bytes:5700, used:0.225s, actions:drop

======================================================================

Paste is with
HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.128/25
FLOATING_RANGE=x.x.139.40/29
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver

http://pastebin.com/4pGvuWma

===============================================================================
With this config. I noticed block error which may be due to simple scheduler being depreciated w/Cinder.

Oct  2 02:44:28 openstack2 kernel: [20294.298311] block nbd15: NBD_DISCONNECT
Oct  2 02:44:28 openstack2 kernel: [20294.298430] block nbd15: Receive control failed (result -32)
Oct  2 02:44:28 openstack2 kernel: [20294.298448] block nbd15: queue cleared

HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.128/25
FLOATING_RANGE=x.x.139.40/29
SYSLOG=True
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
NOVA_VIF_DRIVER=nova.virt.libvirt.vif.LibvirtBridgeDriver


libvirt.log
2012-10-02 06:39:11.719+0000: 31727: error : virNetDevGetIndex:656 : Unable to get index for interface vnet0: No such device
2012-10-02 06:42:06.644+0000: 31724: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error
2012-10-02 06:42:06.644+0000: 31724: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error
2012-10-02 06:44:31.558+0000: 43773: info : libvirt version: 0.9.8
2012-10-02 06:44:31.558+0000: 43773: error : virNetDevGetIndex:656 : Unable to get index for interface vnet0: No such device


http://pastebin.com/bX4u4RhZ


=====================================================

So I ran it again without SimpleScheduler, looked the same. May very well be a storage problem, going crosseyed.

HOST_IP=127.0.0.1
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=openstack
RABBIT_PASSWORD=openstack
SERVICE_PASSWORD=openstack
MYSQL_PASSWORD=password
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
ENABLED_SERVICES+=,-n-vol,c-api,c-sch,c-vol,-n-net,q-svc,q-agt,q-dhcp,quantum,horizon,mysql
FIXED_RANGE=172.24.17.128/25
FLOATING_RANGE=x.x.139.40/29
SYSLOG=True
#LIBVIRT_FIREWALL_DRIVER=nova.virt.libvirt.firewall.IptablesFirewallDriver
LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver
NOVA_VIF_DRIVER=nova.virt.libvirt.vif.LibvirtBridgeDriver

Same output in libvirt
2012-10-02 06:53:06.610+0000: 43768: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error
2012-10-02 06:53:06.610+0000: 43768: error : virNetSocketReadWire:996 : End of file while reading data: Input/output error
2012-10-02 06:56:12.151+0000: 9305: info : libvirt version: 0.9.8
2012-10-02 06:56:12.151+0000: 9305: error : virNetDevGetIndex:656 : Unable to get index for interface vnet0: No such device

http://pastebin.com/phAX0V2h


Dan, do you have a working devstack localrc or even a couple of scenarios  that you could share?

Thanks!
-Brent