Activity log for bug #1886003

Date Who What changed Old value New value Message
2020-07-02 07:22:50 ruediger stock bug added bug
2020-07-02 07:26:30 ruediger stock description Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. Severity -------- Major: Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m 2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m 2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m 2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m 2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m 2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m 2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m 2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m 2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNKNOWN details = "release osh-openstack-mariadb failed: timed out waiting for the condition" debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Test Activity ------------- Regression Testing Workaround ---------- None Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. Severity -------- Major: Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m  2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m  2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m  2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m  2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m  2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m  2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:    status = StatusCode.UNKNOWN    details = "release osh-openstack-mariadb failed: timed out waiting for the condition"    debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Logfiles available at https://files.starlingx.kube.cengn.ca/launchpad/1886003 Test Activity ------------- Regression Testing Workaround ---------- None
2020-07-02 07:42:12 ruediger stock description Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. Severity -------- Major: Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m  2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m  2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m  2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m  2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m  2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m  2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:    status = StatusCode.UNKNOWN    details = "release osh-openstack-mariadb failed: timed out waiting for the condition"    debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Logfiles available at https://files.starlingx.kube.cengn.ca/launchpad/1886003 Test Activity ------------- Regression Testing Workaround ---------- None Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. Severity -------- Major Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m  2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m  2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m  2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m  2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m  2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m  2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:    status = StatusCode.UNKNOWN    details = "release osh-openstack-mariadb failed: timed out waiting for the condition"    debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Logfiles available at https://files.starlingx.kube.cengn.ca/launchpad/1886003 Test Activity ------------- Regression Testing Workaround ---------- None
2020-07-02 14:30:02 zhipeng liu starlingx: assignee zhipeng liu (zhipengs)
2020-07-02 18:05:03 Ghada Khalil tags stx.4.0 stx.4.0 stx.distro.openstack
2020-07-02 18:05:10 Ghada Khalil starlingx: importance Undecided Critical
2020-07-02 18:08:07 Ghada Khalil description Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. Severity -------- Major Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m  2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m  2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m  2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m  2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m  2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m  2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:    status = StatusCode.UNKNOWN    details = "release osh-openstack-mariadb failed: timed out waiting for the condition"    debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Logfiles available at https://files.starlingx.kube.cengn.ca/launchpad/1886003 Test Activity ------------- Regression Testing Workaround ---------- None Brief Description ----------------- The load from https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z/outputs/iso/ only installs successfully on simplex. It fails for duplex and multinode setups. This is also an issue with the master builds as Ussuri was merged on 2020-06-28: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20200701T054008Z/outputs/iso/ Severity -------- Major Steps to Reproduce ------------------ Setup fails in provisioning - it runs into a timeout when trying to install mariadb. Expected Behavior ------------------ Provisioning finishes successfully. Actual Behavior ---------------- Provisioning failes Reproducibility --------------- Reproducible for all setup types except Simplex System Configuration -------------------- virtual multi node Branch/Pull Time/Commit ----------------------- https://mirror.starlingx.cengn.ca/mirror/starlingx/ussuri/centos/monolithic/20200625T130609Z Last Pass --------- This was the first "officially" available build containing ussuri. Tested with a developer load from 05/29 before. Timestamp/Logs -------------- ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/stx-openstack/1.0-42-centos-stable-versioned/stx-openstack-stx-openstack.yaml with exit code 1. See /var/log/armada/stx-openstack-apply_2020-06-30-12-10-08.log for details. From /var/log/armada/stx*log: 2020-06-30 12:10:11.307 726 INFO armada.handlers.armada [-] Processing ChartGroup: openstack-mariadb (Mariadb), sequenced=True^[[00m  2020-06-30 12:10:11.307 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Processing Chart, release=osh-openstack-mariadb^[[00m  2020-06-30 12:10:11.307 726 DEBUG armada.handlers.wait [-] [chart=openstack-mariadb]: Resolved `wait.resources` list: [{'type': 'job', 'required': False, 'labels': {'release_group': 'osh-openstack-mariadb'}}, {'type': 'pod', 'labels': {'release_group': 'osh-openstack-mariadb'}}] __init__ /usr/local/lib/python3.6/dist-packages/armada/handlers/wait.py:89^[[00m  2020-06-30 12:10:11.308 726 INFO armada.handlers.chartbuilder [-] [chart=openstack-mariadb]: Building dependency chart helm-toolkit for release openstack-mariadb.^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Purging release osh-openstack-mariadb with status FAILED^[[00m  2020-06-30 12:10:11.328 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Delete osh-openstack-mariadb release with disable_hooks=False, purge=True, timeout=300 flags^[[00m  2020-06-30 12:10:12.018 726 INFO armada.handlers.chart_deploy [-] [chart=openstack-mariadb]: Installing release osh-openstack-mariadb in namespace openstack, wait=True, timeout=1799s^[[00m  2020-06-30 12:10:12.027 726 INFO armada.handlers.tiller [-] [chart=openstack-mariadb]: Helm install release: wait=True, timeout=1799^[[00m  2020-06-30 12:40:11.799 726 ERROR armada.handlers.tiller [-] [chart=openstack-mariadb]: Error while installing release osh-openstack-mariadb: grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:    status = StatusCode.UNKNOWN    details = "release osh-openstack-mariadb failed: timed out waiting for the condition"    debug_error_string = "{"created":"@1593520811.799184138","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"release osh-openstack-mariadb failed: timed out waiting for the condition","grpc_status":2}" Logfiles available at https://files.starlingx.kube.cengn.ca/launchpad/1886003 Test Activity ------------- Regression Testing Workaround ---------- None
2020-07-06 02:08:21 zhipeng liu starlingx: status In Progress Fix Committed
2020-07-06 02:08:40 zhipeng liu starlingx: status Fix Committed Confirmed
2020-07-07 00:47:51 OpenStack Infra starlingx: status Confirmed In Progress
2020-07-07 22:13:19 OpenStack Infra starlingx: status In Progress Fix Released