Reviewed: https://review.opendev.org/c/starlingx/utilities/+/792213 Committed: https://opendev.org/starlingx/utilities/commit/c4d042615e6fe8944a4628fa1a29e86e012a9bf5 Submitter: "Zuul (22348)" Branch: f/centos8 commit 557cada006fd5a3bd81ad5af387c37657801f8c5 Author: Fernando Theirs Date: Thu May 13 16:21:47 2021 -0300 Collect is missing etcdctl output When the collect tool is run, it does not include the contents of the etcd database. Fixes have been made for this to dump the contents in "etcd_database.dump" file. Verify if etcd access is secured. In that case, certificates will be used. Closes-Bug: 1911935 Signed-off-by: Fernando Theirs Change-Id: Idbc60edffa978a7a6bead939a4eb54f4abae29a6 commit 6045b1b8a0d8ed6a94d06cdfc994bf1a5fa9dbb5 Author: Jim Gauld Date: Thu May 6 11:58:34 2021 -0400 Provide utility script is-rootdisk-device.sh This provides a utility script to determine which disk contains the root filesystem. This can also be used as a helper function for io-scheduler udev rules that require specific configuration for root disk. Example usage: /usr/local/bin/is-rootdisk-device.sh ROOTDISK_DEVICE=sda /usr/local/bin/is-rootdisk-device.sh /dev/sda ROOTDISK_DEVICE=sda /usr/local/bin/is-rootdisk-device.sh /dev/sdb (i.e., no output) Partial-Bug: 1927515 Signed-off-by: Jim Gauld Change-Id: Ib0d4a161a407b08d294c5ff9aa0b7590961e18c9 commit 88a678f142cfe86c58b6405aae6babbc08de0e8f Author: Chen, Haochuan Z Date: Fri Mar 26 09:09:41 2021 +0800 Add packages to stx-ceph-manager image This update installs ceph-mgr, ceph-mon, ceph-osd packages as part of stx-ceph-manager image. Partial-Bug: 1920882 Change-Id: I4afde8b1476e14453fac8561f1edde7360b8ee96 Signed-off-by: Chen, Haochuan Z commit 09b3542fcc6cc0300a9cae0d302225e6977780f3 Author: Scott Little Date: Thu Mar 25 11:49:49 2021 -0400 Set SW_VERSION 21.05 Prep for the StarlingX 5.0 release. SW_VERSION, also known as PLATFORM_RELEASE, uses YY.MM format. Story: 2008055 Task: 42115 Signed-off-by: Scott Little Change-Id: If7c91a2b523358269ae4850961cf4189ffcd7a75 commit ae4cefd0e2a0001476782c31e1003810da2b4838 Author: Chris Friesen Date: Thu Mar 4 18:04:12 2021 -0500 add dcmanager-audit-worker to patch restart script Need to add the new process to the patch restart script. Story: 2007267 Task: 41999 Signed-off-by: Chris Friesen Change-Id: If5faa806bd0d52ddbf1343b064959f4207cf975a commit 27fce5a52321f3014fa8ae9181d344bc774289da Author: Enzo Candotti Date: Mon Feb 1 12:47:38 2021 -0300 Add resource CPU and memory info in collect This adds commands to collect more data to debug resource allocations and usage for the system and containerization. The following command outputs were added: system host-cpu-list ctr -n k8s.io images list kubectl describe nodes | grep -e Capacity: -B1 -A40 | grep -e 'System Info:' -B13 | grep -v 'System info:' lscpu lscpu -e cat /sys/devices/system/cpu/isolated Partial-Bug: 1886868 Signed-off-by: Enzo Candotti Change-Id: Ib26b4bb22d787451cc96820b3a413b410f79d49d commit a4365bf1b2124c9587d78eff82594ce798fb7e06 Author: Enzo Candotti Date: Mon Jan 25 13:16:55 2021 -0300 Fix clear passwords presented in collected log files The --relative option used by rsync resulted in changes to the paths of some collected files in the tarball, which in turn resulted in failures to mask the passwords in those files (as they were no longer in the expected location). This update removes the --relative option, restoring the location to that expected by the masking. Closes-Bug: 1906524 Signed-off-by: Enzo Candotti Change-Id: I001d3e2b1d2ec0ff88a129fe7b31bee21928686b commit 5c8c8434389d67a24758ea36b9439ef946c46dfd Author: Jose Infanzon Date: Wed Dec 16 17:25:18 2020 -0300 Add new logs to the inventory info files Now collect_tool implements a new param '-i' when inventory info is required to be retrieved. By default it will not be executed. The following command outputs were added: system show system host-show system host-port-list system host-if-list system interface-network-list system network-list system host-memory-list system host-label-list system host-disk-list system host-stor-list system host-lvg-list system host-pv-list the execution of: -system host-show -system host-port-list -system host-if-list -system interface-network-list -system host-ethernet-port-list -system host-memory-list -system host-label-list -system host-disk-list -system host-stor-list -system host-lvg-list -system host-pv-list on a simplex lab, took 12 seconds to complete Story: 2008452 Task: 41428 Signed-off-by: Jose Infanzon Change-Id: I223a3ef239a00a1e9dddb86d04874f13c33163e9 commit 538d47cb81317976a7ed31024a05d2fc6bf373bb Author: Jim Gauld Date: Fri Jan 15 16:06:16 2021 -0500 collect is missing helmv2 output This adds helm v2 outputs that we used to have before helm v3 upversion. This gives 'helm status' and 'helm get values' per application using helmv2-cli. This changes helm v3 commands to run as user sysadmin since helm is configured for sysadmin and not root. Otherwise there is missing environment and helm files when run via collect sudo. This corrects one complex expression that does not properly evaluate since it requires being run through command line first. Expressions like CMD="command1 | command2" needs to be run as: eval ${CMD} instead of just: ${CMD} e.g., eval "cat filename | python -m json.tool" Closes-Bug: 1911933 Signed-off-by: Jim Gauld Change-Id: Ie13c78ae325ce0c2f506b320db3e9b51df3f74f2 commit cb00a00c15896b8ab493354e14287cfec29bf2d8 Author: Nicolas Alvarez Date: Fri Dec 18 18:40:49 2020 -0300 Add new log to the device and interface info files. collector.spec: modify to add the new scripts with permissions. collector_disk: new bash script with smartctl command. collector_interfaces: new bash script with ethtool command. Story: 2008452 Task: 41434 Signed-off-by: Nicolas Alvarez Change-Id: I4c7b6e6e3d3fe990750c1fb40b1ab555a63edf83 commit a0736f3dc3f35c70e21eaf62fdb96675ac36ba5d Author: Don Penney Date: Wed Jan 6 14:28:30 2021 -0500 Remove empty package from stx-extensions Packages defined in a spec with no files do not result in an RPM produced by the build. On a rebuild, the build tools scan the spec and sees the package defined but does not find a corresponding RPM, and so flags the package for a rebuild as a result. This commit removes the empty package definition from the spec. Change-Id: I48e725bee837cd27352d38732d6f7155827cfe32 Partial-Bug: 1910439 Signed-off-by: Don Penney commit 676df72f5bbd66e9e8d02cf23ce6721ca169e9f7 Author: pablo bovina Date: Fri Dec 18 17:43:57 2020 -0300 Add parameters for list open files command. This is for performance improvements and usability issues for debugging and testing. Closes-Bug: 1906537 Change-Id: I2e703882219fb986c16fc32e01865fa774094543 Signed-off-by: Pablo Bovina commit 312c87a86fd84534c5acd4447efebfda99e16016 Author: Pablo Bovina Date: Fri Dec 18 17:55:14 2020 +0000 Revert "Collect tool times out due to long running lsof" This reverts commit bdd8c02d84de416ab407cb2cbd39e70b216dc524. Reason for revert: < The following command: "grep awk '($3 !~ /^[0-9]+$/ && /\/mnt\/huge/) || NR==1 {print $0;}'" after the command "lsof -lwX" doesn't adds value for debugging and testing purposes. > Story: 2008452 Task: 41483 Change-Id: I2f81ff6ca4daa6956ad6d0dd210262df2f9e00e8 Signed-off-by: Pablo Bovina commit dbd5dbd8e0b2153ffa04f24649d4bc946ef226d0 Author: albailey Date: Sun Dec 13 19:09:00 2020 -0600 Turn off legacy resolver workaround in pip The legacy resolver was enabled when the new pip was released. The requirements have been cleaned up, and sysinv has been updated. pkgconfig is required to be able to install libvirt from pip. A bindep file has been added to ensure the components required to build libvirt-python are available on the zuul host. The upper constraints have been removed from tox/zuul since the headers for components installed by bindep are newer than what will work when compiling the older libvirt Depends-On: https://review.opendev.org/c/starlingx/config/+/766645 Partial-Bug: #1907678 Signed-off-by: albailey Change-Id: I5905f6cd0bb9bbe379835008ccf3f70102cfc17f commit bdd8c02d84de416ab407cb2cbd39e70b216dc524 Author: pablo bovina Date: Thu Dec 17 15:56:04 2020 -0300 Collect tool times out due to long running lsof lsof without these options: -l : conversion of user ID numbers to login names -w : suppress warning messages -X : skip the reporting of information on all open TCP, UDP and UDPLITE IPv4 and IPv6 files. can lead to timeout failure. Story: 2008452 Task: 41465 Change-Id: I688e024c39d8a56ad30e0e944aeb3e3a16aad2fc Signed-off-by: Pablo Bovina commit 467851e8d5128df6060399207843b78a9325c951 Author: Don Penney Date: Thu Dec 17 13:20:49 2020 -0500 Add auto-version for remaining stx/utilities packages Update remaining StarlingX packages with hardcoded TIS_PATCH_VER to use PKG_GITREVCOUNT where possible, with offsets as needed to ensure the version is incremented above the hardcoded version. Change-Id: Ia1ade77e12b948040f06806a5417d005488e3bb9 Story: 2008455 Task: 41460 Signed-off-by: Don Penney commit dd97a2c3bb0f4c4dcc8f283b9e20cfedcc181737 Author: Lu Yao Chen Date: Wed Dec 16 15:05:08 2020 -0500 Fixing collect tool issue Fixing error found in patch 763859, missing \ character in line 104. Review: https://review.opendev.org/c/starlingx/utilities/+/763859 Closes-Bug: 1896116 Signed-off-by: Lu Yao Chen Change-Id: I80ee960a2b8dab19443a45f8539a5cb602af4efb commit e7e0ecfab90e1f8843a04dd6dc00e7cf00c2adb1 Author: pablo bovina Date: Mon Dec 14 11:12:39 2020 -0300 Add new logs to the alarm info file Add new log to the alarm.info generated by the output of: fm event-list --nopaging Story: 2008452 Task: 41435 Change-Id: Ic7ea6c33e79b846058723dc1023797526ceeb7af Signed-off-by: Pablo Bovina commit b9b8ef2dc99009c18966c91f05bbe81a78b575f7 Author: pablo bovina Date: Mon Dec 14 15:51:42 2020 -0300 Add new logs to the crash info file Add new log to the crash.info generated by the output of: ls -lrtd /var/crash/* md5sum /var/crash/* Story: 2008452 Task: 41433 Change-Id: I19a532c7cead5b972a9c143860478ea128bfab3d Signed-off-by: Pablo Bovina commit 8c44a5a1065edc24e528c2d44b93c5bb7b5dd858 Author: albailey Date: Thu Dec 10 08:53:23 2020 -0600 Enable legacy resolver for pip until requirements are updated pylint zuul jobs are failing due to incompatible dependencies that cause the new version of pip to abort. Enabling the legacy resolver (for now) so zuul can pass, while we fix all the requirements across the different repos. Related-Bug: 1907125 Signed-off-by: albailey Change-Id: I59e29aadb291574307e8278c9351f961bfc0277f commit 17c62bd5aa24459bde2894a876d170803f52db3e Author: Andy Ning Date: Thu Dec 3 09:57:12 2020 -0500 Remove secure hieradata files from collect Supporting controller puppet manifests apply following DOR introduces cached hieradata which will be included in log collect. This change updated collect to remove the secure hieradata files in the cache as they contain clear text passwords. Change-Id: I17542c9fd778107f065531d02c53c59581fc179e Partial-Bug: 1904739 Depends-On: https://review.opendev.org/c/starlingx/config/+/765373 Signed-off-by: Andy Ning commit e3f78a9c0ad948b01fdd73a77604601e46289181 Author: Jackie Huang Date: Tue Oct 27 10:17:31 2020 -0600 cpumap_functions.sh: fix perl experimental feature issue An experimental feature added in Perl 5.14 allowed each, keys, push, pop, shift, splice, unshift, and values to be called with a scalar argument. This experiment is considered unsuccessful, and has been removed in 5.23 and later releases. So don't use this feature to avoid failure: localhost:~# platform_expanded_cpu_list Experimental keys on scalar is now forbidden at -e line 13. Closes-Bug: 1901642 Change-Id: I5898145151e25538c745572da51af26b9251c285 Signed-off-by: Jackie Huang commit b58d9d20d3d1c5307f12706900645e1ce969b2e4 Author: Martin, Chen Date: Fri Oct 30 14:30:00 2020 +0800 Build image for ceph-manager For host-based ceph cluster, serivce manager launch service mgr-restful-plugin which launch ceph-mgr and daemon ceph-manager. Ceph-manager daemon polls ceph cluster status by ceph-mgr restful module and raise or clean alarm to to fault manager. For rook, build a image named stx-ceph-manager, use this image to make a deployment, which will take the same task polling containerized ceph cluster status and raise or clean alarm. Story: 2005527 Task: 41338 Change-Id: Iaaedfc0c7198e102eb4b8c94ab759e9b209e6bfd Signed-off-by: Martin, Chen commit 43cd10d392e64abd810b8032a01bcbf4a29524b6 Author: Lu Yao Chen Date: Mon Nov 23 13:58:11 2020 -0500 Masking passwords with collect script Using collect script to mask cleartext password incidents in /var/log/user.log, done by grepping for -password, password: prefixes and headers and redacting password with xxxxxx string, used user.log with cleartext passwords to test Partial-Bug: 1896116 Signed-off-by: Lu Yao Chen Change-Id: I3a3c02b61994d53589d673b2335d0eb023adfac6 commit ddd40f08f2086690e8439c9ce8183ead7d41761b Author: Carmen Rata Date: Thu Oct 22 23:05:08 2020 -0400 Fix shared libraries file permissions Updated shared library files permission from /usr/lib/systemd/system/ to be non group-writable, to fix openscap security violation. Verified installation is successful in AIO-SX and Standard 2+2 system configurations. Ran successfully "taskset" command to check current affinity to platforms CPUs. Story: 2008037 Task: 40694 Change-Id: If8d7d3becba073ee827e988f1e651a9c8d31d773 Signed-off-by: Carmen Rata commit 0f0bcf58446cc854397e67c0f6577288f98decf6 Author: Eric MacDonald Date: Fri Oct 16 21:44:10 2020 -0400 Exclude /var/log/crash from collect This update adds support to exclude /var/log/crash content from a collect operation. Additional /var/log content exclusions can be added to the new /etc/collect/varlog.exclude file added by this update. Change-Id: I657ce2552e36cc3ac296f11ecd7ed1331ec0f13e Partial-Fix: 1898602 Signed-off-by: Eric MacDonald commit db2156eaddeafec0523aeccda45447b3069eb059 Author: Jim Gauld Date: Fri Oct 16 15:41:34 2020 -0400 Affine kswapd* kernel threads to platform cores The kswapd* kernel tasks are per NUMA node and have floating cpu affinity masks spanning those nodes. On AIO low-latency systems, this affines the kswapd* kernel tasks to platform cores. This is a performance improvement for low-latency sensitive applications. Partial-Bug: 1900174 Change-Id: I20db19978362997b23a69bf591b8e7c23096f492 Signed-off-by: Jim Gauld commit dd5002831818c1a43b90cc3ca704648e6d3e889d Author: Scott Little Date: Thu Sep 17 16:10:31 2020 -0400 Set SW_VERSION/PLATFORM_RELEASE to 20.12 Story: 2008055 Task: 40919 Change-Id: If065e2967b3fbe0a5fce41b6cef3a99e5ffd17f3 Signed-off-by: Scott Little commit 5bc7a33773e947eef8caeb3f371afa816fbcb774 Author: Don Penney Date: Wed Sep 9 17:12:45 2020 -0400 Use newer flake8 to run on ubuntu-focal Zuul machines flake8 2.5.5 fails on ubuntu-focal zuul machines running python3.8 with the following error: AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT' The update removes the hacking constraint to use newer flake8. This also ignores new warnings/errors, which should be addressed in a future update to remove the ignores. Change-Id: Ib24639adeea4da3063fb403a8e8484937f9e1a9f Partial-Bug: 1895054 Signed-off-by: Don Penney commit dd902cf3865f2147559b8c8baeecff70ffebe3cb Author: Jim Gauld Date: Tue Sep 8 11:13:24 2020 -0400 Add mariadb-cli, kubectl and openstack commands to collect This adds containerized related commands to collect: - containerized wrapper script mariadb-cli to access MariaDB mysql - various kubectl commands to understand resource usage - various containerized openstack commands to understand resource usage mariadb-cli is used to dump contents of all mariadb databases. Change-Id: I70a08e38adbba247152509a22a8b9beac9128ff9 Closes-Bug: 1889678 Closes-Bug: 1894103 Signed-off-by: Jim Gauld commit 6573fbe80c04e7cf85c46238c508aff1f2a04454 Author: Don Penney Date: Wed Sep 9 15:27:51 2020 -0400 Fix missing log_error in update-iso.sh The update-iso.sh utility references an undefined log_error function. As a result, error messages are not properly reported, instead resulting in messages such as: stx-iso-utils.sh: line 53: log_error: command not found This error was introduced by code restructuring in a previous update: https://review.opendev.org/707673 This update defines a log_error function for update-iso.sh that writes the message to stderr, as this is a user-run utility, so no need to log to syslog. Change-Id: Ifdc2a25c24cd86e61ccc12601a0bcbe6888ac349 Closes-Bug: 1895042 Signed-off-by: Don Penney commit f43e37b4ea5ed58a55a660cab0b76a3ce0b6532e Author: Don Penney Date: Wed Sep 2 22:42:41 2020 -0400 Fix gen-bootloader-iso.sh error on unpatched controller On a system that has never been patched, the updates repository may not yet have a Packages directory. This update fixes the patch setup in gen-bootloader-iso.sh to check for this case and handle it without failing. This update also improves the platform-kickstarts-pxeboot check to look for a pxeboot kickstart, to ensure there isn't a false-positive match against an extracted pxe-network-installer patch, which would also have a /pxeboot directory. Change-Id: I8e24a3ff123f4c2649be52b1a0ce77d830e342f3 Closes-Bug: 1893990 Signed-off-by: Don Penney commit c0ab7c29c1e44bf448fcd89dbc6a8fa9e814818b Author: Jessica Castelino Date: Wed Sep 2 10:29:08 2020 -0400 Adding dcmanager-orchestrator process to patch restart script Process dcmanager-orchestrator has been added to the restart script to simplify no-reboot patches that updates that process. Story: 2007267 Task: 40813 Change-Id: I5d634f769e86ac4c79170bfd8476bd65a1ac2280 Signed-off-by: Jessica Castelino commit fb70729a9e13f6b12d0a5376c637723701578721 Author: Don Penney Date: Mon Aug 31 14:37:59 2020 -0400 Improve DNF cache handling in gen-bootloader-iso.sh The "dnf repoquery" command is using and updating the cache for ad-hoc repos created when using the repofrompath option. This can result in gen-bootloader-iso.sh using stale cached information when looking for specific RPMs in the patch repo when generating bootloader ISOs for subcloud installs. In order to avoid such issues, this update asks DNF to mark the cache as expired after copying the patch repo for generating the new ISOs, as well as on deletion. This ensures the repoquery uses up-to-date information. Additionally, this update removes a duplicated patch repo setup step, which was not cleaned up during dev testing on the previous commit. Change-Id: I664edb346692d79cd13b23772686ea88f0aaaf9f Story: 2007994 Task: 40798 Signed-off-by: Don Penney commit 584b23bd1527bb2666952c6427013ce4e7728934 Author: Michel Thebeau Date: Wed Aug 26 16:06:12 2020 -0400 stx-extensions: use the host's PID for coredump systemd-coredump running as root on the host wants to get process information using the process' PID, but %p refers to "the PID namespace in which the process resides" (the container). systemd-coredump coredumps. Dmesg shows "Failed to get EXE" and the segfault (sic 'systemd-coredum'). This segfault of systemd-coredump is intermittent, and so on occasion a core file may actually be dumped for the containerized process. This happens when the %p does not match a process from the host's perspective. Dmesg shows "Failed to get COMM..." and "Failed to get EXE". This becomes more likely as the PIDs in container's namespace become larger - the host's PIDs are more sparse as the numbers increase. Use %P instead, "as seen in the initial PID namespace" (host). Convert the package to use PKG_GITREVCOUNT for release increment. Closes-Bug: 1892951 Change-Id: Ifa5017d5997d12891893fc97fac4487ddfbbbbb8 Signed-off-by: Michel Thebeau commit a69c15f4065017bcacd1d8cefba40bd1e4c956ba Author: Don Penney Date: Fri Aug 7 10:56:25 2020 -0400 Use host patches for subcloud install setup This update enhances gen-bootloader-iso.sh to use patches applied on the host controller when setting up the subcloud installation bootloader. This checks the release version from the ISO to look for patches for that release only, copying applied and committed patches to the installation setup. This update also adds cleanup of partial setup on failure. Change-Id: I4eeb0141b75e48ce53c132b89e19e07c68cc49d2 Story: 2007994 Task: 40631 Signed-off-by: Don Penney commit 6fec9e9ec55af9c62ca03558766806710d4da0b6 Author: Carmen Rata Date: Thu Jul 16 15:08:50 2020 -0400 Verify-license plugin support Current StarlingX implementation does not support a proper license validation. Used the Stevedore plugin pattern to allow override of the verify-license implementation to add enhanced validation rules if necessary. Story: 2007403 Task: 39644 Change-Id: I6fa6626feabff06b832dd41a2778804a28956131 Signed-off-by: Carmen Rata commit 71314e4af1af0c9590908656882c7a4c68172399 Author: albailey Date: Thu Jul 9 14:01:52 2020 -0500 Adding missing distributed cloud processes to patch restart script In order to simplify no-reboot patches that update any of these DC processes, they have been added to the restart script. - dcdbsync-api - dcorch-identity-api-proxy - dcmanager-audit Closes-Bug: 1887002 Change-Id: Ida2cee5b3ca873888820e3de485e9877b64166b9 Signed-off-by: albailey commit 7c076a390f99cb72623da7168ae64e2947a25080 Author: Eric MacDonald Date: Mon Jul 6 11:24:10 2020 -0400 Exclude temporary subcloud install iso files from collect The collect tool is unnecessarily including temp subcloud install iso files and related other large tmp files. This update changes the raw 'cp ...' to 'rsync ...' with the --exclude option to filter out iso files from the the collect_dc and collect_sysinv collect scripts. Change-Id: I3575c3193a24f376dcd006c3e5015c551023c69a Closes-Bug: 1885778 Signed-off-by: Eric MacDonald commit 7ed511a6a0b861573bd9224a8a2a2ff6a031f463 Author: Don Penney Date: Tue Jun 30 17:19:14 2020 -0400 Protect against parallel exec of gen-bootloader-iso.sh The bulk of the work done by gen-bootloader-iso.sh is not safe to be executed by two callers in parallel. If the initial execution is setting up shared files when a subsequent call starts, the subsequent call can end up with an incomplete set of symlinks to shared files. This, in turn, can result in installation failures. To protect against this, a file lock is added to gen-bootloader-iso.sh to prevent subsequent calls from accessing shared files before they're completely setup. Change-Id: Id6def4527b226b8746b2b8ff74059c0d3937e130 Closes-Bug: 1885779 Signed-off-by: Don Penney commit a10381203d920c55732109eb07aae2cf6d3345e8 Author: Jim Gauld Date: Wed Jun 17 09:20:47 2020 -0400 Add kube-cpusets to collect This adds kube-cpusets tool to collect_hosts. This displays cpuset and numa node information per kubernetes pod on a given host. Change-Id: I1bd862cc0b1cef3b7997c032108c02b09f2885a1 Story: 2006999 Task: 40106 Depends-On: https://review.opendev.org/723808 Signed-off-by: Jim Gauld