Failed to launch instance: could not file file or directory for instance console

[Michael Skalka]

During execution of a rally test on a Focal Ussuri openstack during this test run: https://solutions.qa.canonical.com/testruns/testRun/a9c7e6a9-6191-4827-a04a-a53f72eec38a
Crashdump can be found here: https://oil-jenkins.canonical.com/artifacts/a9c7e6a9-6191-4827-a04a-a53f72eec38a/generated/generated/openstack/juju-crashdump-openstack-2020-12-11-18.42.02.tar.gz

From the log output nova tried to launch the instance however somewhere in the process the console log file was never created. Upon creation of the PTY device virsh failed as a result.

From the crashdump tar:

1/baremetal/var/log/nova/nova-compute.log

2020-12-11 17:11:00.959 1530144 ERROR nova.virt.libvirt.guest [req-17d75220-8c15-44f5-9bec-1959a23cb487 9cdd5500d7404781aced2d6f1edad070 2cd4c52559474383be06edb09883b6e2 - 7659a60928684fe7845d6d29c833a529 7659a60928684fe7845d6d29c833a529] Error launching a defined domain with XML:
<domain>
...
    <serial type='pty'>
      <log file='/var/lib/nova/instances/34d24f53-6f23-4acf-8d36-34ca8d047afd/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <log file='/var/lib/nova/instances/34d24f53-6f23-4acf-8d36-34ca8d047afd/console.log' append='off'/>
      <target type='serial' port='0'/>
    </console>
...
</domain>
: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-12-11T17:10:59.646894Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: Failed to create PTY: No such file or directory
2020-12-11 17:11:00.959 1530144 ERROR nova.virt.libvirt.driver [req-17d75220-8c15-44f5-9bec-1959a23cb487 9cdd5500d7404781aced2d6f1edad070 2cd4c52559474383be06edb09883b6e2 - 7659a60928684fe7845d6d29c833a529 7659a60928684fe7845d6d29c833a529] [instance: 34d24f53-6f23-4acf-8d36-34ca8d047afd] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-12-11T17:10:59.646894Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on: Failed to create PTY: No such file or directory
2020-12-11 17:11:01.312 1530144 INFO os_vif [req-17d75220-8c15-44f5-9bec-1959a23cb487 9cdd5500d7404781aced2d6f1edad070 2cd4c52559474383be06edb09883b6e2 - 7659a60928684fe7845d6d29c833a529 7659a60928684fe7845d6d29c833a529] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:76:2b:48,bridge_name='br-int',has_traffic_filtering=True,id=379c6824-2843-4fdb-88b1-38fee30e95a0,network=Network(9b817be8-5860-483e-8683-50b95db2e56d),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap379c6824-28')
2020-12-11 17:11:01.313 1530144 INFO nova.virt.libvirt.driver [req-17d75220-8c15-44f5-9bec-1959a23cb487 9cdd5500d7404781aced2d6f1edad070 2cd4c52559474383be06edb09883b6e2 - 7659a60928684fe7845d6d29c833a529 7659a60928684fe7845d6d29c833a529] [instance: 34d24f53-6f23-4acf-8d36-34ca8d047afd] Deleting instance files /var/lib/nova/instances/34d24f53-6f23-4acf-8d36-34ca8d047afd_del
2020-12-11 17:11:01.314 1530144 INFO nova.virt.libvirt.driver [req-17d75220-8c15-44f5-9bec-1959a23cb487 9cdd5500d7404781aced2d6f1edad070 2cd4c52559474383be06edb09883b6e2 - 7659a60928684fe7845d6d29c833a529 7659a60928684fe7845d6d29c833a529] [instance: 34d24f53-6f23-4acf-8d36-34ca8d047afd] Deletion of /var/lib/nova/instances/34d24f53-6f23-4acf-8d36-34ca8d047afd_del complete

[Michael Skalka (mskalka)]

We've seen this again a few more times in the last month or so:

https://solutions.qa.canonical.com/bugs/bugs/bug/1908096

Including this test run: https://solutions.qa.canonical.com/testruns/testRun/b1588957-d3f4-44cd-8267-5f23363b907a

I imagine it's probably the root cause for this issue: https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1898581 but I need to go through the logs from the linked runs on that bug.