stx-openstack re-apply fails; tiller not recording openstack-neutron release

[austin gillmann]

Brief Description
-----------------
Upon attempting to reapply the openstack application, the process will abort at the placement chart. Logs from the tiller pod show this to be the result of the generated configmaps being too long.

Severity
--------
Major - Essential services available with the exception of a few.

Steps to Reproduce
------------------
1. Apply stx-openstack
2. Watch for successful apply
3. Reapply a configuration change

Expected Behavior
------------------
Application reapply success

Actual Behavior
----------------
Apply aborted.

Reproducibility
---------------
Reproducible on my cluster persistently across reinstalls.

System Configuration
--------------------
2 Controllers, 12 workers, 2 dedicated storage

Branch/Pull Time/Commit
-----------------------
stx3.0

Timestamp/Logs
--------------
[tiller] 2020/03/17 04:14:38 preparing install for osh-openstack-neutron
[storage] 2020/03/17 04:14:38 getting release history for "osh-openstack-neutron"
[tiller] 2020/03/17 04:14:38 rendering neutron chart using values
2020/03/17 04:16:39 info: manifest "neutron/daemonset-bagpipe-bgp.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/job-bootstrap.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/daemonset-lb-agent.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/daemonset-l2gw-agent.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/secret-ingress-tls.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/job-image-repo-sync.yaml" is empty. Skipping.
2020/03/17 04:16:39 info: manifest "neutron/job-db-drop.yaml" is empty. Skipping.
[tiller] 2020/03/17 04:16:40 performing install for osh-openstack-neutron
[tiller] 2020/03/17 04:16:40 executing 1 crd-install hooks for osh-openstack-neutron
[tiller] 2020/03/17 04:16:40 hooks complete for crd-install osh-openstack-neutron
[tiller] 2020/03/17 04:16:40 executing 1 pre-install hooks for osh-openstack-neutron
[tiller] 2020/03/17 04:16:40 hooks complete for pre-install osh-openstack-neutron
[storage] 2020/03/17 04:16:40 getting release history for "osh-openstack-neutron"
[storage] 2020/03/17 04:16:40 creating release "osh-openstack-neutron.v1"
[storage/driver] 2020/03/17 04:16:41 create: failed to create: ConfigMap "osh-openstack-neutron.v1" is invalid: []: Too long: must have at most 1048576 characters
[tiller] 2020/03/17 04:16:41 warning: Failed to record release osh-openstack-neutron: ConfigMap "osh-openstack-neutron.v1" is invalid: []: Too long: must have at most 1048576 characters
[kube] 2020/03/17 04:16:41 building resources from manifest
[kube] 2020/03/17 04:16:41 creating 201 resource(s)
[tiller] 2020/03/17 04:16:41 warning: Release "osh-openstack-neutron" failed: poddisruptionbudgets.policy "neutron-server" already exists
[storage] 2020/03/17 04:16:41 updating release "osh-openstack-neutron.v1"
[storage/driver] 2020/03/17 04:16:42 update: failed to update: configmaps "osh-openstack-neutron.v1" not found
[tiller] 2020/03/17 04:16:42 warning: Failed to update release osh-openstack-neutron: configmaps "osh-openstack-neutron.v1" not found
[tiller] 2020/03/17 04:16:42 failed install perform step: release osh-openstack-neutron failed: poddisruptionbudgets.policy "neutron-server" already exists
[storage] 2020/03/17 04:16:42 getting last revision of "osh-openstack-neutron"
[storage] 2020/03/17 04:16:42 getting release history for "osh-openstack-neutron"

[Ghada Khalil (gkhalil)]

What is the config change done at step (3) in the Steps to Reproduce?

[austin gillmann (austingillmann)]

@Ghada

Can be anything, in my case it was originally to enable the swift charts but my workaround was to just delete the whole app and data and apply it with the overrides. The configmap for neutron will still fail to write even then.

[austin gillmann (austingillmann)]

I suppose to be more specific when I first discovered this it was changing the helm endpoint domain as described in the docs.

[Ghada Khalil (gkhalil)]

Adding the release that this was reported in

[yong hu (yhu6)]

we are heading to 4.0 release, so this LP has to be held on for a while and will come back later.

[Ghada Khalil (gkhalil)]

Closing as stx.3.0 is EOL as of Dec 2020