StarlingX

SX after BnR, platform-integ-apps apply-failed

Bug #1866704 reported by Peng Peng on 2020-03-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	High	Mihnea Saracin

Bug Description

Brief Description
-----------------
platform-integ-apps applied on system. After backup and Restore, platform-integ-apps app applu-failed.
/var/log/armada/platform-integ-apps-apply_2020-03-09-09-28-48.log shows:
Unable to extract /tmp/armadacwbivapw

Severity
--------
Major

Steps to Reproduce
------------------
platform-integ-apps applied on system.
Executing backup and Restore,
After BnR, check platform-integ-apps app status.

TC-name: Sanity after BnR

Expected Behavior
------------------
app applied

Actual Behavior
----------------
apply-failed

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
One node system

Lab-name: WCP_112

Branch/Pull Time/Commit
-----------------------
2020-03-08_04-10-00

Last Pass
---------
First time run

Timestamp/Logs
--------------
[2020-03-09 08:31:57,752] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-03-09 08:31:58,815] 436 DEBUG MainThread ssh.expect :: Output:
+---------------------+---------+-------------------------------+---------------+----------+-----------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+---------+-------------------------------+---------------+----------+-----------+
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | uploaded | completed |
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | applied | completed |
+---------------------+---------+-------------------------------+---------------+----------+-----------+

Backup and Restore ...

[2020-03-09 17:28:42,296] 314 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://[abcd:204::1]:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2020-03-09 17:28:43,399] 436 DEBUG MainThread ssh.expect :: Output:
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+
| oidc-auth-apps | 1.0-0 | oidc-auth-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |
| platform-integ-apps | 1.0-8 | platform-integration-manifest | manifest.yaml | apply-failed | operation aborted, check logs for detail |
+---------------------+---------+-------------------------------+---------------+--------------+------------------------------------------+

From /var/log/armada/platform-integ-apps-apply_2020-03-09-09-28-48.log
2020-03-09 09:28:49.743 11 WARNING armada.handlers.armada [-] Disabling server validation certs to extract charts^[[00m
2020-03-09 09:28:50.731 11 INFO armada.handlers.lock [-] Releasing lock^[[00m
2020-03-09 09:28:50.737 11 ERROR armada.cli [-] Caught internal exception: armada.exceptions.source_exceptions.TarballExtractException: Unable to extract /tmp/armadacwbivapw
2020-03-09 09:28:50.737 11 ERROR armada.cli Traceback (most recent call last):
2020-03-09 09:28:50.737 11 ERROR armada.cli File "/usr/local/lib/python3.6/dist-packages/armada/utils/source.py", line 150, in extract_tarball
2020-03-09 09:28:50.737 11 ERROR armada.cli file = tarfile.open(tarball_path)
2020-03-09 09:28:50.737 11 ERROR armada.cli File "/usr/lib/python3.6/tarfile.py", line 1576, in open
2020-03-09 09:28:50.737 11 ERROR armada.cli raise ReadError("file could not be opened successfully")
2020-03-09 09:28:50.737 11 ERROR armada.cli tarfile.ReadError: file could not be opened successfully

Test Activity
-------------
Sanity

Tags:

Peng Peng (ppeng) on 2020-03-09

tags:

added: stx.retestneeded

Revision history for this message

Peng Peng (ppeng) wrote on 2020-03-09:

log @
https://files.starlingx.kube.cengn.ca/launchpad/1866704

Frank Miller (sensfan22) on 2020-03-12

Changed in starlingx:
assignee:	nobody → Mihnea Saracin (msaracin)

Revision history for this message

Mihnea Saracin (msaracin) wrote on 2020-03-12:

It seems that the helm charts are not correctly restored when the wipe_ceph_osds flag is set to true.

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-03-13:

stx.4.0 / high priority - this issue was introduced by recent code changes related to B&R

tags:	added: stx.4.0 stx.update
Changed in starlingx:
importance:	Undecided → High
status:	New → Triaged

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-16: Fix proposed to ansible-playbooks (master)

Fix proposed to branch: master
Review: https://review.opendev.org/713332

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-17: Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/713332
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=3441d046eaafcf20a463aaf4bd8c11172014f1e9
Submitter: Zuul
Branch: master

commit 3441d046eaafcf20a463aaf4bd8c11172014f1e9
Author: Mihnea Saracin <email address hidden>
Date: Mon Mar 16 23:11:27 2020 +0200

Fix B&R when wipe_ceph_osds is set to true

The helm charts are restored only when the ceph osds are not wiped,
but they also need to be restored when the ceph osds are wiped.

    Closes-Bug: 1866704
    Change-Id: I7d9bf8cbfbf26a0edd44c6af6f0d98d95abfb682
    Signed-off-by: Mihnea Saracin <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

Yang Liu (yliu12) wrote on 2020-03-25:

This issue is no longer seen since following load: 2020-03-19_04-10-00

tags:

removed: stx.retestneeded

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-31: Fix proposed to ansible-playbooks (f/centos8)

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716133

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-31: Fix merged to ansible-playbooks (f/centos8)

Download full text (12.5 KiB)

Reviewed: https://review.opendev.org/716133
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=ddcb11f4b773f4b3190663defe3ba0f3ec4201c8
Submitter: Zuul
Branch: f/centos8

commit bf103f3c54eb45c26d52a43c35339d1d863a42de
Author: Mihnea Saracin <email address hidden>
Date: Fri Mar 27 18:19:02 2020 +0200

Fix B&R when the controller needs to be unlocked

    After running the restore playbook, all the applications
    should be in an uploaded state. But they are in an
    applied state instead, making the controller-0
    unable to unlock.

    Closes-Bug: 1869403
    Change-Id: I8bd9c51e250969cc334d52b78c616f9ad082afd8
    Signed-off-by: Mihnea Saracin <email address hidden>

commit 6e875971afeaf1378c2c8aeb845359459838ce30
Author: Stefan Dinescu <email address hidden>
Date: Sat Mar 21 16:57:57 2020 +0200

Fix Netapp port conflict

    By default, the Trident Netapp service opens port 8443 for
    HTTPS REST api usage. This conflicts with the port the
    Horizon dashboard uses on an HTTPS enabled setup (the port
    is also 8443).

    In order to fix this, we change the default port from 8443
    to 8678, but also make it configurable through ansible
    overrides.

    The Trident service also opens port 8001 for metrics usage.
    While that doesn't currently conflict with any other service
    on the system, I also made that configurable through
    ansible overrides, in case such a conflict appears in the
    future.

    Change-Id: I08db939acac6082f82b9e12e932d8289c7cecdeb
    Closes-bug: 1868382
    Signed-off-by: Stefan Dinescu <email address hidden>

commit 5a9ba6786e393f2cd93bfae8c3a8f09f0cf9eb26
Author: Robert Church <email address hidden>
Date: Thu Mar 19 19:08:17 2020 -0400

Upversion Multus to 3.4

Updates the Multus configuration to align with version 3.4

    Change-Id: Ifc236ccbbe4e559987d7ef522902f638062348ca
    Depends-On: https://review.opendev.org/#/c/714024/
    Story: 2006999
    Task: 39110
    Signed-off-by: Robert Church <email address hidden>

commit 6a261463f9ac0f81d9c7f054dd3cb10a51934d4a
Author: Robert Church <email address hidden>
Date: Wed Mar 18 22:01:03 2020 -0400

Upversion Calico from 3.6 to 3.12

    Updates the Calico configuration to align with version 3.12. This
    introduces support for a Flex Volume Driver which requires enabling the
    --volume-plugin-dir option for kubelet, the --flex-volume-plugin-dir
    option for kube-controller-manager, and pulling the pod2daemon-flexvol
    image used by calico-node pods.

    Change-Id: I74bc5c53ffcb16c8e3c06cebf20eac296b9ccc65
    Story: 2006999
    Task: 39109
    Depends-On: https://review.opendev.org/#/c/714023
    Signed-off-by: Robert Church <email address hidden>

commit b35387f8bc40714e9633e6191267284b8af8ccee
Author: Stefan Dinescu <email address hidden>
Date: Thu Mar 19 18:13:26 2020 +0200

Netapp: Fix handling of IPv6 addresses

    Using bash process subtitution to pass the file parameter
    to the "create backend" command doesn't work as the bash
    variable expansion...

Reviewed:  https://review.opendev.org/716133
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=ddcb11f4b773f4b3190663defe3ba0f3ec4201c8
Submitter: Zuul
Branch:    f/centos8

commit bf103f3c54eb45c26d52a43c35339d1d863a42de
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Fri Mar 27 18:19:02 2020 +0200

Fix B&R when the controller needs to be unlocked
    
    After running the restore playbook, all the applications
    should be in an uploaded state. But they are in an
    applied state instead, making the controller-0
    unable to unlock.
    
    Closes-Bug: 1869403
    Change-Id: I8bd9c51e250969cc334d52b78c616f9ad082afd8
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>

commit 6e875971afeaf1378c2c8aeb845359459838ce30
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Sat Mar 21 16:57:57 2020 +0200

Fix Netapp port conflict
    
    By default, the Trident Netapp service opens port 8443 for
    HTTPS REST api usage. This conflicts with the port the
    Horizon dashboard uses on an HTTPS enabled setup (the port
    is also 8443).
    
    In order to fix this, we change the default port from 8443
    to 8678, but also make it configurable through ansible
    overrides.
    
    The Trident service also opens port 8001 for metrics usage.
    While that doesn't currently conflict with any other service
    on the system, I also made that configurable through
    ansible overrides, in case such a conflict appears in the
    future.
    
    Change-Id: I08db939acac6082f82b9e12e932d8289c7cecdeb
    Closes-bug: 1868382
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit 5a9ba6786e393f2cd93bfae8c3a8f09f0cf9eb26
Author: Robert Church <robert.church@windriver.com>
Date:   Thu Mar 19 19:08:17 2020 -0400

Upversion Multus to 3.4
    
    Updates the Multus configuration to align with version 3.4
    
    Change-Id: Ifc236ccbbe4e559987d7ef522902f638062348ca
    Depends-On: https://review.opendev.org/#/c/714024/
    Story: 2006999
    Task: 39110
    Signed-off-by: Robert Church <robert.church@windriver.com>

commit 6a261463f9ac0f81d9c7f054dd3cb10a51934d4a
Author: Robert Church <robert.church@windriver.com>
Date:   Wed Mar 18 22:01:03 2020 -0400

Upversion Calico from 3.6 to 3.12
    
    Updates the Calico configuration to align with version 3.12. This
    introduces support for a Flex Volume Driver which requires enabling the
    --volume-plugin-dir option for kubelet, the --flex-volume-plugin-dir
    option for kube-controller-manager, and pulling the pod2daemon-flexvol
    image used by calico-node pods.
    
    Change-Id: I74bc5c53ffcb16c8e3c06cebf20eac296b9ccc65
    Story: 2006999
    Task: 39109
    Depends-On: https://review.opendev.org/#/c/714023
    Signed-off-by: Robert Church <robert.church@windriver.com>

commit b35387f8bc40714e9633e6191267284b8af8ccee
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Thu Mar 19 18:13:26 2020 +0200

Netapp: Fix handling of IPv6 addresses
    
    Using bash process subtitution to pass the file parameter
    to the "create backend" command doesn't work as the bash
    variable expansion breaks the IPv6 format, resulting in
    an invalid JSON file.
    
    In order to fix this, we create a temporary file where we
    store the desired information. This file is then cleaned
    (along with the whole staging folder) once the playbook
    finishes exection.
    
    Also improved handling of IPv6 by the install command by
    supporing extra parameters for the "tridentctl install"
    command. In order to force IPv6 compatibility, you need to
    add to the "trident_install_extra_params" variable the
    "--use-ipv6" option
    
    Change-Id: I5ecd98307d40cfdfe277d72acd7b095ff0b66aad
    Closes-bug: 1868103
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit 8c6cf0e629cfb62ff654653e866e9a894b1bb806
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Wed Mar 4 14:12:57 2020 +0200

Add playbook for installing Trident Netapp backend
    
    Changes included:
    - add separate playbook for configuring the netapp storage
      backend for kubernetes
    - templates for trident are generated using the "tridentctl
      -n trident install --generate-custom-yaml"
    - change to the push-docker-images role to support being
      imported for the trident playbook
    - the YAML files generated from these templates are stored
      in /etc/kubernetes/trident/setup. This location can be
      customized using ansible overrides.
    - merged the "upgrade_k8s_networking=true" and
      "upgrade_kubernetes=true" variables into the
      "mode" variable.
    
    Changes made to the generated templates:
    - changed default REST api port from 8000 to 8677; this was
      done due to port 8000 already being used by armada in
      StarlingX. The port can be changed at any time through
      ansible overrides
    - allow configuration of any namespace for the pods to be
      launched in; the default is still "trident"
    - images pulled by the trident pods are pulled from the
      local registry. They are pre-pulled to the local registry
      by the push-docker-images role
    - add an imagePullSecret to the service account so images
      can be pulled from the local registry
    - add additional nodeSelector in order to restrict the
      main trident service to launching only on controller
      nodes
    
    Change-Id: I5ca9db9a17ba2cf8d1a000ab0b4e36656d8b290c
    Co-Authored-By: Stefan Dinescu <stefan.dinescu@windriver.com>
    Co-Authored-By: Ovidiu Poncea <ovidiu.poncea@windriver.com>
    Depends-On: https://review.opendev.org/#/c/711907/
    Story: 2007391
    Task: 38985
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit 3441d046eaafcf20a463aaf4bd8c11172014f1e9
Author: Mihnea Saracin <Mihnea.Saracin@windriver.com>
Date:   Mon Mar 16 23:11:27 2020 +0200

Fix B&R when wipe_ceph_osds is set to true
    
    The helm charts are restored only when the ceph osds are not wiped,
    but they also need to be restored when the ceph osds are wiped.
    
    Closes-Bug: 1866704
    Change-Id: I7d9bf8cbfbf26a0edd44c6af6f0d98d95abfb682
    Signed-off-by: Mihnea Saracin <Mihnea.Saracin@windriver.com>

commit c03dc5f3fc7dd1a5646cfc5b83d6c6fc08241bc1
Author: Andy Ning <andy.ning@windriver.com>
Date:   Thu Mar 5 16:57:29 2020 -0500

Create ssl_ca dir to support multiple CA certificate
    
    This update creates ssl_ca directory in the drbd shared fs during
    ansible bootstrap. The directory is used to store the CA certificates
    as individual files. CA certificate install and uninstall operations
    are manipulating these certificate files.
    
    Change-Id: Idfca7220320a5de1cc274e077fc7ce8d2778a76b
    Closes-Bug: 1861438
    Closes-Bug: 1860995
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit 208df05af590ab1cbdac16c94f65b29d4fac3e90
Author: David Sullivan <david.sullivan@windriver.com>
Date:   Sun Mar 1 21:16:04 2020 -0500

Add OAM IP records to API server certSANs
    
    Add the OAM IP records to the apiserver_cert_sans. This will allow for
    remote access to the kubernetes API server.
    
    Change-Id: I344f59fa0b5a24633f1341e2c2fd54748d88c0af
    Closes-Bug: 1863798
    Signed-off-by: David Sullivan <david.sullivan@windriver.com>

commit fea7aa9c64f4a84216e241921d5b202a137c2c04
Author: Jerry Sun <jerry.sun@windriver.com>
Date:   Wed Mar 4 16:08:36 2020 -0500

Support post-bootstrap config of kube-apiserver parameters
    
    Save the openid connect parameters for kube-apiserver in system
    service parameters. This keeps them consistent if the user changes
    them after bootstrap
    
    Story: 2006711
    Task: 38944
    
    Depends-On: https://review.opendev.org/711337
    
    Change-Id: I1757afecb1ff222b16feee464fa486d5ac02f40d
    Signed-off-by: Jerry Sun <jerry.sun@windriver.com>

commit 3a21623f6b7285ac9e822a8a9b23d5d0fd0f4e38
Author: hutianhao <hu.tianhao@99cloud.net>
Date:   Wed Feb 19 17:14:44 2020 +0800

Select another way to get device information
    
    If terminal language is Chinese(maybe other language will also have
    this problem), it can't get exact root disk size because the word
    "Disk" is not in English. So drdb config will fail after unlock.
    Select another way to get device information to avoid this problem.
    
    Closes-Bug: 1859951
    Change-Id: Ie144f7f302b73854c8407681a5dd0981797d4c5f
    Signed-off-by: hutianhao <hu.tianhao@99cloud.net>

commit 7e0f62e8480197d1518e890eb47ec7ac18986e46
Author: Ovidiu Poncea <ovidiu.poncea@windriver.com>
Date:   Thu Dec 19 11:15:45 2019 +0200

B&R: Add support for backing up and restoring StarlingX apps
    
    This works when ceph storage backend is kept intact (i.e. when
    restore is run with wipe_ceph_osds=false). To accomplish this the
    commit enhances platform B&R to include the etcd database. Ceph
    PVC are kept intact.
    
    Details:
     o Backup etcd db
     o Restore etcd db
     o Allow k8s to start with the restored db
     o Fix helm to start with the restored db
    
    Change-Id: I80f36dbedbd322bd3129aaa7740704f97da7750c
    Story: 2006770
    Task: 37780
    Signed-off-by: Ovidiu Poncea <ovidiu.poncea@windriver.com>

commit 37471539878e6861cec4983cd8d89ba154acd7f3
Author: Tao Liu <tao.liu@windriver.com>
Date:   Wed Feb 19 19:06:26 2020 -0500

Increase activeDeadlineSeconds in the rvmc template
    
    While investigating a remote install failure, it was observed
    that the rvmc job/pod was terminating after 10 mins prior
    to entering an error or complete state hence, the error
    logs were not captured.
    
    This update increases activeDeadlineSeconds from 600 seconds
    to 1200 seconds.
    
    Change-Id: Ibb9240c7153b7dbead20ffb0a7ac4fc606b64fcd
    Story: 2006980
    Task: 38465
    Signed-off-by: Tao Liu <tao.liu@windriver.com>

commit f4baa947f7643d0d437e63c83725c4c9c24f27c8
Author: Don Penney <don.penney@windriver.com>
Date:   Wed Feb 12 14:48:15 2020 -0500

Replace image stable-latest tags with static versions
    
    In order to avoid compatiblity issues from using a moving
    stable-latest tag on referenced images, the playbooks are updated to
    use static version tags for those images built by CENGN.
    
    Change-Id: I336ec16ec6c8d7581f547efa77bd75f2c1b4726b
    Partial-Bug: 1854869
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit e364e4b11ca9977a37e3b3d05924846b0fbe7e0d
Author: Stefan Dinescu <stefan.dinescu@windriver.com>
Date:   Fri Jan 31 17:37:23 2020 +0200

Remove default provisioning of ceph
    
    Ceph is no longer a default service, so we remove
    the provisioning from ansible.
    
    Ceph will be configured using the following command
        "system storage-backend-add ceph --confirmed"
    This configuration option will be available after
    running the ansible bootstrap.
    
    Change-Id: I8cac1ef3ab97a036f7e77289b8fb7807cf37ddd7
    Depends-On: https://review.opendev.org/705235
    Story: 2007064
    Task: 38590
    Signed-off-by: Stefan Dinescu <stefan.dinescu@windriver.com>

commit ee0b46522c106832004eed7e332fbe93f8de385b
Author: Andy Ning <andy.ning@windriver.com>
Date:   Tue Feb 4 15:18:01 2020 -0500

Enable kubernetes secret encryption at rest
    
    This update enabled kubernetes secret resource encryption during
    ansible play system deployment. It contains the following changes:
    
    - An extraArgs "encryption-provider-config:
      /etc/kubernetes/encryption-provider.yaml" is added to kubeadm.yaml
      by template. This argument will be fed into kube-apiserver static
      pod manifest so that kube-apiserver will be started with this
      encryption enabling flag.
    - A template file is added to generate the actual encryption provider
      configuration file. This generated file is hostPath mounted to
      kube-apiserver pod as kube-apiserver's encryption provider configuration.
    - The generated encryption provider configuration file is transfered to
      the second controller by the shared fs for kube-apiserver pod running
      on the second controller.
    
    Change-Id: If1c1e1887e024923e5d876bf589a5bfd312d5da9
    Story: 2007243
    Task: 38626
    Depends-On: https://review.opendev.org/#/c/705975
    Signed-off-by: Andy Ning <andy.ning@windriver.com>

commit 5471cd1ae5ed92af0d160e04ce0983dee85cf716
Author: Bin Qian <bin.qian@windriver.com>
Date:   Wed Feb 5 08:27:51 2020 -0500

Adding job to upload commits to GitHub
    Add job to publish ansible-playbooks repo to GitHub
    
    Change-Id: Ica6f28f34882ec574bd91f6aa26933e0f1b4f6fa
    Story: 2007252
    Task: 38655
    Signed-off-by: Bin Qian <bin.qian@windriver.com>

tags:

added: in-f-centos8