Collect is missing openstack details and maria database

[Jim Gauld]

Brief Description
-----------------
Collect is not capturing openstack related information.
Collect is capturing postgres database, openstack now uses mariadb, we are missing mariadb.

Full mariadb database dump is required for debugging many openstack nova related issues, especially when there are currently limited logs coming out for typical VM lifecycle and failure to schedule code paths.

Want a wrapper script to gain access to mariadb CLI to run mysql commands, eg, similar to helmv2-cli.

Severity
--------
Major - needed for debugging.

Steps to Reproduce
------------------
collect

Expected Behavior
------------------
Easy access to mariadb via CLI to run mysql commands.
Output from mariadb database.
Output from standard openstack list commands.

Actual Behavior
----------------
No data.

Reproducibility
---------------
100 percent

System Configuration
--------------------
All configurations.

Branch/Pull Time/Commit
-----------------------
- not applicable

Last Pass
---------
- not applicable

Timestamp/Logs
--------------
- not applicable

Test Activity
-------------
Developer Testing, Evaluation.

Workaround
----------
Can get individual tables or specific fields like this :
kubectl exec -it -n openstack mariadb-server-0 – bash -c "mysql --password=\$MYSQL_DBADMIN_PASSWORD --user=root nova -e 'select host,pci_stats from compute_nodes;'"

Example of brute force collection of important tables here:
/folk/cgts/users/gkopec/updates/Kubernetes_Helm/scripts/get*tables
./get_cinder_tables
./get_nova_api_tables
./get_nova_cell0_tables
./get_nova_tables
./get_placement_tables

For the 'collect', suggest just dump entire database as we did for postgres, instead of trying to capture individual queries for each table.

Should write a wrapper script similar to helmv2-cli, but just for mariadb CLI mysql commands.

Should capture a few key openstack related outputs.
Cannot count on having openrc.admin source file for credentials,so instead of 'source ./openrc.admin', need alternate way to get credentials, and possibly add manually those options like the following:
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne \
<command>

Want generic openstack commands like:
openstack user list --long
openstack router list --long
openstack network list --long
openstack subnet list --long
openstack image list --long
openstack volume list --all-projects --long
openstack availability zone list --long --all-projects
openstack server group list --all-projects --long
openstack server list --all-projects --long
openstack stack list --long --all-projects
openstack security group list --all-projects
openstack security group rule list --all-projects --long
openstack keypair list
openstack availability zone list --long --all-projects
openstack configuration show
openstack host list
openstack hypervisor list --long
openstack hypervisor stats show
openstack hypervisor show controller-0 -f yaml
(ideally show per hypervisor)

[Frank Miller (sensfan22)]

Since there is not a solution yet for this LP, re-assigning to a designer to get a solution proposed this week.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/750425

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750425
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=dd902cf3865f2147559b8c8baeecff70ffebe3cb
Submitter: Zuul
Branch: master

commit dd902cf3865f2147559b8c8baeecff70ffebe3cb
Author: Jim Gauld <email address hidden>
Date: Tue Sep 8 11:13:24 2020 -0400

    Add mariadb-cli, kubectl and openstack commands to collect

    This adds containerized related commands to collect:
    - containerized wrapper script mariadb-cli to access MariaDB mysql
    - various kubectl commands to understand resource usage
    - various containerized openstack commands to understand resource usage

    mariadb-cli is used to dump contents of all mariadb databases.

    Change-Id: I70a08e38adbba247152509a22a8b9beac9128ff9
    Closes-Bug: 1889678
    Closes-Bug: 1894103
    Signed-off-by: Jim Gauld <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/utilities/+/792213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/utilities/+/792213
Committed: https://opendev.org/starlingx/utilities/commit/c4d042615e6fe8944a4628fa1a29e86e012a9bf5
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 557cada006fd5a3bd81ad5af387c37657801f8c5
Author: Fernando Theirs <email address hidden>
Date: Thu May 13 16:21:47 2021 -0300

    Collect is missing etcdctl output

    When the collect tool is run, it does not include the contents
    of the etcd database. Fixes have been made for this to dump the
    contents in "etcd_database.dump" file.

    Verify if etcd access is secured. In that case, certificates
    will be used.

    Closes-Bug: 1911935

    Signed-off-by: Fernando Theirs <email address hidden>
    Change-Id: Idbc60edffa978a7a6bead939a4eb54f4abae29a6

commit 6045b1b8a0d8ed6a94d06cdfc994bf1a5fa9dbb5
Author: Jim Gauld <email address hidden>
Date: Thu May 6 11:58:34 2021 -0400

    Provide utility script is-rootdisk-device.sh

    This provides a utility script to determine which disk contains the root
    filesystem. This can also be used as a helper function for io-scheduler
    udev rules that require specific configuration for root disk.

    Example usage:
    /usr/local/bin/is-rootdisk-device.sh
    ROOTDISK_DEVICE=sda

    /usr/local/bin/is-rootdisk-device.sh /dev/sda
    ROOTDISK_DEVICE=sda

    /usr/local/bin/is-rootdisk-device.sh /dev/sdb
    (i.e., no output)

    Partial-Bug: 1927515
    Signed-off-by: Jim Gauld <email address hidden>
    Change-Id: Ib0d4a161a407b08d294c5ff9aa0b7590961e18c9

commit 88a678f142cfe86c58b6405aae6babbc08de0e8f
Author: Chen, Haochuan Z <email address hidden>
Date: Fri Mar 26 09:09:41 2021 +0800

    Add packages to stx-ceph-manager image

    This update installs ceph-mgr, ceph-mon, ceph-osd packages as part
    of stx-ceph-manager image.

    Partial-Bug: 1920882

    Change-Id: I4afde8b1476e14453fac8561f1edde7360b8ee96
    Signed-off-by: Chen, Haochuan Z <email address hidden>

commit 09b3542fcc6cc0300a9cae0d302225e6977780f3
Author: Scott Little <email address hidden>
Date: Thu Mar 25 11:49:49 2021 -0400

    Set SW_VERSION 21.05

    Prep for the StarlingX 5.0 release.
    SW_VERSION, also known as PLATFORM_RELEASE, uses YY.MM format.

    Story: 2008055
    Task: 42115
    Signed-off-by: Scott Little <email address hidden>
    Change-Id: If7c91a2b523358269ae4850961cf4189ffcd7a75

commit ae4cefd0e2a0001476782c31e1003810da2b4838
Author: Chris Friesen <email address hidden>
Date: Thu Mar 4 18:04:12 2021 -0500

    add dcmanager-audit-worker to patch restart script

    Need to add the new process to the patch restart script.

    Story: 2007267
    Task: 41999
    Signed-off-by: Chris Friesen <email address hidden>
    Change-Id: If5faa806bd0d52ddbf1343b064959f4207cf975a

commit 27fce5a52321f3014fa8ae9181d344bc774289da
Author: Enzo Candotti <email address hidden>
Date: Mon Feb 1 12:47:38 2021 -0300

    Add resource CPU and memory info in collect

    This adds commands to collect more data to debug
    resource allocations and...