There seem to be 17 occurrences of this error in the last 7 days [starting 3DEC2015]:
message: "libvirtError: cannot read header" AND tags: "screen-n-cpu.txt"
And, 195 occurrences of the below error message string when looked at logstash, :
message: "session 2 login rejected: Initiator failed authentication with target" AND tags: "syslog"
From:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/old/screen-n-cpu.txt.gz#_2015-12-02_10_57_35_812
[. . .]
2015-12-02 10:57:35.812 ERROR nova.compute.manager [req-b5b4691f-da88-4abd-bffd-a72d52242dae tempest-TestVolumeBootPattern-440462397 tempest-TestVolumeBootPattern-1264952841] [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Instance failed to spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] Traceback (most recent call last):
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/compute/manager.py", line 2155, in _build_resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] yield resources
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/compute/manager.py", line 2009, in _build_and_run_instance
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 2444, in spawn
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] block_device_info=block_device_info)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4519, in _create_domain_and_network
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] xml, pause=pause, power_on=power_on)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/virt/libvirt/driver.py", line 4449, in _create_domain
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] guest.launch(pause=pause)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 141, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] self._encoded_xml, errors='ignore')
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(self.type_, self.value, self.tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 136, in launch
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] return self._domain.createWithFlags(flags)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] rv = execute(f, *args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] six.reraise(c, e, tb)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] rv = meth(*args, **kwargs)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778] libvirtError: cannot read header '/dev/disk/by-path/ip-149.202.189.35:3260-iscsi-iqn.2010-10.org.openstack:volume-690bf429-192d-40f1-97bf-f558a8106208-lun-1': Input/output error
2015-12-02 10:57:35.812 25717 ERROR nova.compute.manager [instance: 4d176a3a-2678-4207-99c6-f84894a23778]
[. . .]
And, there are errors from 'iscsid' from syslog:
http://logs.openstack.org/80/251980/1/check/gate-grenade-dsvm-multinode/5716e36/logs/syslog.txt.gz#_Dec__2_10_55_36
[. . .]
Dec 2 10:55:36 ubuntu iscsid: Kernel reported iSCSI connection 2:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
Dec 2 10:55:36 ubuntu iscsid: Login failed to authenticate with target iqn.2010-10.org.openstack:volume-6fb73a19-26f9-4c76-95e0-ccfa57489593
Dec 2 10:55:36 ubuntu iscsid: session 1 login rejected: Initiator failed authentication with target
Dec 2 10:55:36 ubuntu object-auditor: Object audit (ALL) "forever" mode completed: 3.39s. Total quarantined: 0, Total errors: 0, Total files/sec: 7.66, Total bytes/sec: 9986657.72, Auditing time: 3.39, Rate: 1.00
Dec 2 10:55:36 ubuntu kernel: [ 2566.962683] type=1400 audit(1449053736.838:29): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-4ac13a03-c8c8-4c20-8cd1-2f2354131a67" pid=1807 comm="apparmor_parser"
Dec 2 10:55:38 ubuntu kernel: [ 2568.159143] connection2:0: detected conn error (1020)
[. . .]
I think this was fixed by an infra change, the two hosts had the same hostname so n-cpu was getting tripped up.
https:/