Containers: vm console log cannot be collected

[Yang Liu]

Brief Description
-----------------
'nova console-log <vm_id>' returns nothing.
Same issue when using horizon to view instance console log.

Severity
--------
Minor

Steps to Reproduce
------------------
- Launch a vm
- nova console-log <vm_id> OR view Instance Console Log from horizon

Expected Behavior
------------------
- vm console log is returned/displayed

Actual Behavior
----------------
- nothing returned/displayed

Reproducibility
---------------
Reproducible

System Configuration
--------------------
any

Branch/Pull Time/Commit
-----------------------
f/stein as of 2019-02-22

Timestamp/Logs
--------------
[2019-02-25 16:07:31,824] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne console-log 8749eadf-6929-4e59-933e-950430a03f4d'
controller-0:~$

[Ghada Khalil (gkhalil)]

Marking as release gating; the vm console is a key tool to help debug issues in the guest. Medium priority.

[Frank Miller (sensfan22)]

Assigning to Cindy to identify a prime to work on this issue.

[Lin Shuicheng (shuicheng)]

Issue is reproduced, begin to check it.

[Lin Shuicheng (shuicheng)]

From nova compute log, we could find below log, need check why nova/libvirt cannot get the log file.
{"log":"2019-05-13 02:27:46.823 2527442 INFO nova.compute.manager [req-20970eda-4e51-4b49-bf34-0175f567897d e9d3a26729064fb3b57174e588c2d48f bc278a0009864e538ce7b18b7bb8ca95 - default default] [instance: cdccdeb9-98b1-4758-b223-534717b5f02c] Get console output\n","stream":"stdout","time":"2019-05-13T02:27:46.825988609Z"}
{"log":"2019-05-13 02:27:46.823 2527442 INFO nova.compute.manager [req-20970eda-4e51-4b49-bf34-0175f567897d e9d3a26729064fb3b57174e588c2d48f bc278a0009864e538ce7b18b7bb8ca95 - default default] [instance: cdccdeb9-98b1-4758-b223-534717b5f02c] Get console output\n","stream":"stdout","time":"2019-05-13T02:27:46.8260216Z"}
{"log":"2019-05-13 02:27:46.830 2527442 INFO nova.virt.libvirt.driver [req-20970eda-4e51-4b49-bf34-0175f567897d e9d3a26729064fb3b57174e588c2d48f bc278a0009864e538ce7b18b7bb8ca95 - default default] [instance: cdccdeb9-98b1-4758-b223-534717b5f02c] console logfile for instance does not exist\n","stream":"stdout","time":"2019-05-13T02:27:46.831918789Z"}
{"log":"2019-05-13 02:27:46.830 2527442 INFO nova.virt.libvirt.driver [req-20970eda-4e51-4b49-bf34-0175f567897d e9d3a26729064fb3b57174e588c2d48f bc278a0009864e538ce7b18b7bb8ca95 - default default] [instance: cdccdeb9-98b1-4758-b223-534717b5f02c] console logfile for instance does not exist\n","stream":"stdout","time":"2019-05-13T02:27:46.831957786Z"}

[Lin Shuicheng (shuicheng)]

The reason is found, when libvirt/qemu create the VM, it will create a pty device (/dev/pts/x) for the vm also. Due to libvirt/nova is containerized now, and /dev is sperated for each container. So nova running in nova-compute container cannot find the pty device, and reports the "console logfile instance does not exist" error, and cause the console log cannot be shown in horizon.
The fix is try to map /dev in host to nova-compute container. Code for the change is under test now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/660268

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660268
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=4a88506bfdcb030b142cbf73b9f6766b61b9929a
Submitter: Zuul
Branch: master

commit 4a88506bfdcb030b142cbf73b9f6766b61b9929a
Author: Shuicheng Lin <email address hidden>
Date: Tue May 21 15:06:08 2019 +0800

    fix vm console log cannot be collected

    When the vm is created by libvirt/qemu, a pty device is created
    also under /dev/pts. And nova will check this device exist or not
    when get console log. After openstack containerization, libvirt and
    nova compute are run in container. Libvirt mounts /dev space to
    container directly, so the pty device could be seen in host also.
    But nova compute doesn't mount /dev space, and lead to the issue.
    Solution:
    mount /dev/pts space in host to nova compute container.
    Test:
    vm console log could be seen both in horizon and by openstack cmd.

    Closes-Bug: 1817618
    Change-Id: I39bdd8c50a3bbea3b7896bac258721d7e1d2ee44
    Signed-off-by: Shuicheng Lin <email address hidden>

[Yang Liu (yliu12)]

Verified passed on 0611 master load.