| 2022-01-06 11:57:16 |
Bernardo Decco de Siqueira |
description |
While the WRO application is applied to the system, the backup will exclude its images. But when the WRO application is removed, the WRO images are going to be included in the backup. This can make the backup too large to fit in the /opt/platform-backup filesystem that is used to hold the backups. This is especially important in the case of AIO-SX subcloud upgrades, as the backup must fit in /opt/platform-backup.
Load Info / Patch Line-Up
upgrade from below load to WRCP 2021-03-30_00-00-07 build
sw-patch query
Patch ID RR Release Patch State
================================== == ======= ===========
PATCH.ENABLE_DEV_CERTIFICATE-20.06 N 20.06 Applied
WRCP_20.06_PATCH_0001 Y 20.06 Committed
WRCP_20.06_PATCH_0002 Y 20.06 Committed
WRCP_20.06_PATCH_0003 N 20.06 Committed
WRCP_20.06_PATCH_0004 N 20.06 Committed
WRCP_20.06_PATCH_0005 Y 20.06 Committed
WRCP_20.06_PATCH_0006 N 20.06 Committed
WRCP_20.06_PATCH_0007 Y 20.06 Committed
WRCP_20.06_PATCH_0008 Y 20.06 Committed
WRCP_20.06_PATCH_0009 Y 20.06 Committed
WRCP_20.06_PATCH_0010 Y 20.06 Applied
WRCP_20.06_PATCH_0011 Y 20.06 Applied
WRCP_20.06_PATCH_0012 Y 20.06 Applied
WRCP_20.06_UPGRADES_3_31_A N 20.06 Applied
System Config
AIO-SX
Description of failure
Simplex upgrade was failing when WRO docker images are applied prior to upgrade. When WRO images are there it was trying to backup huge docker image space where there not enough space to backup .
TASK [backup/backup-system : Fail if there is not enough free space to create docker images backup archive] ***
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Not enough free space for /opt/platform-backup/upgrade_images_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz. Free space available is 4294148KiB. Estimation shows it needs at least 20976036KiB."}ontroller-0:/home/sysadmin# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.local:9001/docker.io/wind-river/tis-networking-avs-neutron WRO.20.06-15.3.1.dev26-wrs.1 866f53a00197 3 months ago 910MB
registry.local:9001/docker.io/starlingx/stx-keystone-api-proxy stx.4.0-1.0.0-wrs.1 2d297e55a9bd 6 months ago 965MB
registry.local:9001/docker.io/wind-river/tis-libvirt WRO.20.06-4.7.0-wrs.1 11e5286b1f7a 7 months ago 685MB
registry.local:9001/docker.io/wind-river/tis-networking-avs-nova WRO.20.06-20.3.1.dev4-wrs.1 473b3b3bd44c 8 months ago 1.26GB
registry.local:9001/docker.io/starlingx/stx-placement stx.4.0-2.0.0 3ae8e8c07aae 9 months ago 823MB
registry.local:9001/docker.io/starlingx/stx-keystone stx.4.0-16.0.2.dev6 b194ae9b9343 9 months ago 858MB
registry.local:9001/docker.io/starlingx/stx-horizon stx.4.0-16.2.0 b5fa59ded7c9 9 months ago 1.07GB
registry.local:9001/docker.io/starlingx/stx-heat stx.4.0-13.0.2 586f1f8d9ee5 9 months ago 900MB
registry.local:9001/docker.io/starlingx/stx-gnocchi stx.4.0-4.3.2 109b7de17b90 9 months ago 894MB
registry.local:9001/docker.io/starlingx/stx-glance stx.4.0-19.0.3 34ea58feea61 9 months ago 876MB
registry.local:9001/docker.io/starlingx/stx-cinder stx.4.0-15.2.1.dev1 b1375529abce 9 months ago 934MB
registry.local:9001/docker.io/starlingx/stx-ceilometer stx.4.0-13.1.2.dev1 0b7830001204 9 months ago 915MB
registry.local:9001/docker.io/starlingx/stx-barbican stx.4.0-9.0.1 719efe0597ef 9 months ago 798MB
registry.local:9001/docker.io/starlingx/stx-panko stx.4.0-7.0.1.dev2 06d5a35bcb29 9 months ago 859MB
registry.local:9001/docker.io/starlingx/stx-ironic stx.4.0-13.0.4.dev44 566afb4e0b38 9 months ago 893MB
registry.local:9001/docker.io/starlingx/stx-aodh stx.4.0-9.0.1 2d60a0eff387 9 months ago 861MB
registry.local:9001/docker.io/starlingx/stx-nova-api-proxy stx.4.0-1.0.0 4c236592b15c 9 months ago 753MB
registry.local:9001/docker.io/starlingx/stx-ovs stx.4.0-2.11.0 cc8c8224f14f 9 months ago 754MB
registry.local:9001/docker.io/wind-river/tis-networking-avs-heat WRO.20.06-13.0.2 79fe2463ecd1 9 months ago 901MB
registry.local:9001/docker.io/starlingx/stx-mariadb stx.4.0-10.2.18 ec64d8896fb9 9 months ago 522MB
registry.local:9001/docker.io/starlingx/stx-fm-rest-api stx.4.0-1.0.0 e83b925dd17a 9 months ago 805MB
registry.local:9001/docker.io/rabbitmq 3.7-management f31ea3fef6e6 10 months ago 180MB
registry.local:9001/docker.io/rabbitmq 3.7.24-management 8d22195e21ff 11 months ago 182MB
registry.local:9001/docker.io/rabbitmq 3.7.24 a635b62c006b 11 months ago 151MB
registry.local:9001/docker.io/starlingx/n3000-opae stx.4.0-v1.0.0 fb95693fe5c6 13 months ago 506MB
tis-lab-registry.cumulus.wrs.com:9001/wrcp-staging/docker.io/starlingx/n3000-opae stx.4.0-v1.0.0 fb95693fe5c6 13 months ago 506MB
registry.local:9001/quay.io/airshipit/armada 8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic 3061a8a540ac 20 months ago 458MB
tis-lab-registry.cumulus.wrs.com:9001/wrcp-staging/quay.io/airshipit/armada 8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic 3061a8a540ac 20 months ago 458MB
registry.local:9001/docker.io/openstackhelm/mariadb 10.2.18 ad585bf77f99 2 years ago 447MB
registry.local:9001/quay.io/external_storage/cephfs-provisioner v2.1.0-k8s1.11 cac658c0a096 2 years ago 401MB
registry.local:9001/quay.io/stackanetes/kubernetes-entrypoint v0.3.1 7fb3c2364b87 2 years ago 97.7MB
registry.local:9001/docker.io/mariadb 10.2.13 ea5e726062ce 3 years ago 396MB
registry.local:9001/docker.io/memcached 1.5.5 9a7e8440a999 3 years ago 58.6MB
registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress-controller 0.9.0 e56d6a14283a 3 years ago 190MB
registry.local:9001/docker.io/nginx 1.13.3 b8efb18f159b 3 years ago 107MB
registry.local:9001/gcr.io/google_containers/defaultbackend 1.0 137a07dfd084 5 years ago 7.51MB
Timestamp when failure occurred
AY RECAP *********************************************************************
localhost : ok=105 changed=48 unreachable=0 failed=1
sysinv 2021-03-31 17:51:56.524 2928 INFO sysinv.agent.manager [-] Exception during simplex upgrade data collection
sysinv 2021-03-31 17:51:56.525 2928 ERROR sysinv.agent.manager [-] Command '['ansible-playbook', '-e', 'platform_backup_file=upgrade_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz docker_local_registry_backup_file=upgrade_images_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz backup_user_local_registry=true backup_dir=/opt/platform-backup', '/usr/share/ansible/stx-ansible/playbooks/backup.yml']' returned non-zero exit status 2: CalledProcessError: Command '['ansible-playbook', '-e', 'platform_backup_file=upgrade_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz docker_local_registry_backup_file=upgrade_images_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz backup_user_local_registry=true backup_dir=/opt/platform-backup', '/usr/share/ansible/stx-ansible/playbooks/backup.yml']' returned non-zero exit
Issue intermittent (Frequency of occurrence) or 100% Reproducible?
Reproduceable 100
Steps to Reproduce OR Events leading up to failure
Prior to upgrade apply WRO and remove
Follow upgrade steps as per procedure . upgrade start failed
Impact of Failure
(ex: Critical, Severe, Standard)
Impact on Customer
(ex: Lab or Deployment? Deadlines and/or deliverables impacted? Target date(s) which customer needs to be addressed? Workaround lower severity of issue? Did the system automatically recover?
Unable to use WRO during the upgrade
Log/File location
(Collect all required logs, scripts, or any relevant files)
(In Distributed Cloud Environment - please collect system controller as well as subcloud logs)
TRIAGE
Was the issue reproduced internally
yes
Time-line based on log analysis
AY RECAP *********************************************************************
localhost : ok=105 changed=48 unreachable=0 failed=1
sysinv 2021-03-31 17:51:56.524 2928 INFO sysinv.agent.manager [-] Exception during simplex upgrade data collection
sysinv 2021-03-31 17:51:56.525 2928 ERROR sysinv.agent.manager [-] Command '['ansible-playbook', '-e', 'platform_backup_file=upgrade_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz docker_local_registry_backup_file=upgrade_images_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz backup_user_local_registry=true backup_dir=/opt/platform-backup', '/usr/share/ansible/stx-ansible/playbooks/backup.yml']' returned non-zero exit status 2: CalledProcessError: Command '['ansible-playbook', '-e', 'platform_backup_file=upgrade_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz docker_local_registry_backup_file=upgrade_images_data_2021-03-31T175022_b4970c68-eec6-4ae7-9975-7743452ea1f4.tgz backup_user_local_registry=true backup_dir=/opt/platform-backup', '/usr/share/ansible/stx-ansible/playbooks/backup.yml']' returned non-zero exit
Key failure logs
(explain why logs are of interest)
disk required for backup and failed |
While the WRO application is applied to the system, the backup will exclude its images. But when the WRO application is removed, the WRO images are going to be included in the backup. This can make the backup too large to fit in the /opt/platform-backup filesystem that is used to hold the backups. This is especially important in the case of AIO-SX subcloud upgrades, as the backup must fit in /opt/platform-backup. |
|
| 2022-01-06 15:23:19 |
Ghada Khalil |
description |
While the WRO application is applied to the system, the backup will exclude its images. But when the WRO application is removed, the WRO images are going to be included in the backup. This can make the backup too large to fit in the /opt/platform-backup filesystem that is used to hold the backups. This is especially important in the case of AIO-SX subcloud upgrades, as the backup must fit in /opt/platform-backup. |
While the openstack application is applied to the system, the backup will exclude its images. But when the application is removed, the images are going to be included in the backup. This can make the backup too large to fit in the /opt/platform-backup filesystem that is used to hold the backups. This is especially important in the case of AIO-SX subcloud upgrades, as the backup must fit in /opt/platform-backup. |
|
