stx-openstack apply-failed

Bug #1889023 reported by admin@zhangqi.net on 2020-07-27
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
StarlingX
Undecided
Unassigned

Bug Description

Brief Description
-----------------
I am tring to install starlingx R3.0 AIO virutalized.
application-apply stx-openstack failed finally.

Severity
--------

Steps to Reproduce
------------------
1. install ubuntu 16.04 on baremetal
2. git clone https://opendev.org/starlingx/tools.git and setup virutal machine via setup_configuration.sh -i simplex
3. run ansible-playbook
4. upload stx-openstack and apply

Expected Behavior
------------------
expect all k8s container in running or complete, openstack setup complete and ready to use.

Actual Behavior
----------------
apply failed at 54%.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
<One node system, Two node system, Multi-node system, Dedicated storage, https, IPv4, IPv6 etc.>
one node system, all in one, virtualized environment

Branch/Pull Time/Commit
-----------------------
controller-0:~# docker images | grep ^starlingx
starlingx/stx-placement rc-3.0-centos-stable-latest e273740d28c4 5 days ago 861MB
starlingx/stx-nova rc-3.0-centos-stable-latest a64d25525301 5 days ago 1.3GB
starlingx/stx-neutron rc-3.0-centos-stable-latest 23ccbedea543 5 days ago 944MB
starlingx/stx-keystone rc-3.0-centos-stable-latest 2d1d06bcb3fd 5 days ago 896MB
starlingx/stx-horizon rc-3.0-centos-stable-latest 99c3209b44fd 5 days ago 1.11GB
starlingx/stx-heat rc-3.0-centos-stable-latest 5ea3a26624d1 5 days ago 938MB
starlingx/stx-gnocchi rc-3.0-centos-stable-latest 8ab74fec318b 5 days ago 932MB
starlingx/stx-glance rc-3.0-centos-stable-latest 5f8a852ce1d5 5 days ago 914MB
starlingx/stx-cinder rc-3.0-centos-stable-latest 688337608d98 5 days ago 972MB
starlingx/stx-ceilometer rc-3.0-centos-stable-latest e48a281f9b57 5 days ago 953MB
starlingx/stx-panko rc-3.0-centos-stable-latest 39aca91c109b 5 days ago 897MB
starlingx/stx-ironic rc-3.0-centos-stable-latest dfe55bd7527c 5 days ago 931MB
starlingx/stx-aodh rc-3.0-centos-stable-latest 5de9ace955ce 5 days ago 899MB
starlingx/stx-nova-api-proxy rc-3.0-centos-stable-latest d867d8969c4f 5 days ago 791MB
starlingx/stx-libvirt rc-3.0-centos-stable-latest 67d7c4cf814e 5 days ago 758MB
starlingx/stx-ovs rc-3.0-centos-stable-latest c606795ff805 5 days ago 799MB
starlingx/stx-mariadb rc-3.0-centos-stable-latest f27522485de4 5 days ago 522MB
starlingx/stx-fm-rest-api rc-3.0-centos-stable-latest 10cce1323fe8 5 days ago 842MB
starlingx/stx-keystone-api-proxy rc-3.0-centos-stable-latest 887497308b5d 5 days ago 994MB
starlingx/k8s-cni-sriov master-centos-stable-latest 49f3250071d9 6 days ago 9.81MB
starlingx/k8s-plugins-sriov-network-device master-centos-stable-latest ab8eb0c6b3bc 3 weeks ago 25.6MB
starlingx/multus v3.2.16 4c3da64245c6 10 months ago 308MB
starlingx/ceph-config-helper v1.15.0 e9e28af77f24 12 months ago 820MB

Last Pass
---------

Timestamp/Logs
--------------

Test Activity
-------------
Feature Testing

Workaround
----------
Describe workaround if available

admin@zhangqi.net (r-sdmin-0) wrote :
admin@zhangqi.net (r-sdmin-0) wrote :
Download full text (12.2 KiB)

During apply, I found another error in neutron-ovs-agent pod.

controller-0:~# kubectl get pods -n openstack
NAME READY STATUS RESTARTS AGE
cinder-api-dbc984dd5-lz829 1/1 Running 0 25m
cinder-backup-794588786b-2hpzh 1/1 Running 0 25m
cinder-backup-storage-init-pv4qn 0/1 Completed 0 25m
cinder-bootstrap-blrzz 0/1 Completed 0 25m
cinder-create-internal-tenant-qpbch 0/1 Completed 0 25m
cinder-db-init-z9ghz 0/1 Completed 0 25m
cinder-db-sync-w8tss 0/1 Completed 0 25m
cinder-ks-endpoints-bx4hd 0/9 Completed 0 25m
cinder-ks-service-jbh68 0/3 Completed 0 25m
cinder-ks-user-4fs99 0/1 Completed 0 25m
cinder-rabbit-init-25kzm 0/1 Completed 0 25m
cinder-scheduler-84f6549-d5qzn 1/1 Running 0 25m
cinder-storage-init-jsnv7 0/1 Completed 0 25m
cinder-volume-5d9bbc778f-xl9mj 1/1 Running 0 25m
cinder-volume-usage-audit-1595814300-c7xxs 0/1 Completed 0 13m
cinder-volume-usage-audit-1595814600-d6r9n 0/1 Completed 0 8m30s
cinder-volume-usage-audit-1595814900-dp6tj 0/1 Completed 0 3m27s
glance-api-66c8f55d47-5fmhz 1/1 Running 0 28m
glance-db-init-cf8z5 0/1 Completed 0 28m
glance-db-sync-m79jq 0/1 Completed 0 28m
glance-ks-endpoints-47nq2 0/3 Completed 0 28m
glance-ks-service-lmq56 0/1 Completed 0 28m
glance-ks-user-8hbvq 0/1 Completed 0 28m
glance-rabbit-init-fh22j 0/1 Completed 0 28m
glance-storage-init-twmg4 0/1 Completed 0 28m
ingress-c74489c96-mbglp 1/1 Running 0 32m
ingress-error-pages-5889bfcf6c-29ks2 1/1 Running 0 32m
keystone-api-7865c6cb67-8rjkv 1/1 Running 0 29m
keystone-bootstrap-82psp 0/1 Completed 0 29m
keystone-credential-setup-6jnqc 0/1 Completed 0 29m
keystone-db-init-6s2jx 0/1 Completed 0 29m
keystone-db-sync-thvw6 0/1 Completed 0 29m
keystone-domain-manage-jfg4x 0/1 Completed 0 29m
keystone-fernet-setup-7m4xv 0/1 ...

admin@zhangqi.net (r-sdmin-0) wrote :

Finally solve this problem by the following steps:

1. docker run -it registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest bash
vi /var/lib/openstack/lib/python2.7/site-packages/neutron/cmd/sanity_check.py

comment line 347:
347 #cfg.CONF.set_default('vf_management', True)

2. docker commit -p XXXX NEW_IMAGE_TAG
3. docker tag NEW_IMAGE_TAG registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest
4. kubectl delete pod -n openstack neutron-ovs-agent-controller-0-937646f6-bgtnf

then, if lucky, k8s can start neutron-ovs-agent again and we finally make openstack running.

Ghada Khalil (gkhalil) on 2020-07-27
tags: added: stx.3.0 stx.distro.openstack
Pratik M. (pvmpublic) wrote :

I hit the same issue, and so did another user here:
http://lists.starlingx.io/pipermail/starlingx-discuss/2020-July/009242.html

In my case, I have a newly installed AIO-DX and then try to apply stx-openstack. The relevant logs snippets are attached.

Download full text (4.4 KiB)

Yes I did exactly the same thing to solve this problem.

On Thu, Aug 6, 2020 at 9:21 PM Steven Webster <email address hidden>
wrote:

> Relevant:
>
> http://lists.starlingx.io/pipermail/starlingx-
> discuss/2020-July/009270.html
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1889023
>
> Title:
> stx-openstack apply-failed
>
> Status in StarlingX:
> New
>
> Bug description:
> Brief Description
> -----------------
> I am tring to install starlingx R3.0 AIO virutalized.
> application-apply stx-openstack failed finally.
>
>
> Severity
> --------
>
>
> Steps to Reproduce
> ------------------
> 1. install ubuntu 16.04 on baremetal
> 2. git clone https://opendev.org/starlingx/tools.git and setup virutal
> machine via setup_configuration.sh -i simplex
> 3. run ansible-playbook
> 4. upload stx-openstack and apply
>
>
> Expected Behavior
> ------------------
> expect all k8s container in running or complete, openstack setup
> complete and ready to use.
>
>
> Actual Behavior
> ----------------
> apply failed at 54%.
>
>
>
> Reproducibility
> ---------------
> 100% reproducible
>
> System Configuration
> --------------------
> <One node system, Two node system, Multi-node system, Dedicated storage,
> https, IPv4, IPv6 etc.>
> one node system, all in one, virtualized environment
>
>
> Branch/Pull Time/Commit
> -----------------------
> controller-0:~# docker images | grep ^starlingx
> starlingx/stx-placement
> rc-3.0-centos-stable-latest
> e273740d28c4 5 days ago 861MB
> starlingx/stx-nova
> rc-3.0-centos-stable-latest
> a64d25525301 5 days ago 1.3GB
> starlingx/stx-neutron
> rc-3.0-centos-stable-latest
> 23ccbedea543 5 days ago 944MB
> starlingx/stx-keystone
> rc-3.0-centos-stable-latest
> 2d1d06bcb3fd 5 days ago 896MB
> starlingx/stx-horizon
> rc-3.0-centos-stable-latest
> 99c3209b44fd 5 days ago 1.11GB
> starlingx/stx-heat
> rc-3.0-centos-stable-latest
> 5ea3a26624d1 5 days ago 938MB
> starlingx/stx-gnocchi
> rc-3.0-centos-stable-latest
> 8ab74fec318b 5 days ago 932MB
> starlingx/stx-glance
> rc-3.0-centos-stable-latest
> 5f8a852ce1d5 5 days ago 914MB
> starlingx/stx-cinder
> rc-3.0-centos-stable-latest
> 688337608d98 5 days ago 972MB
> starlingx/stx-ceilometer
> rc-3.0-centos-stable-latest
> e48a281f9b57 5 days ago 953MB
> starlingx/stx-panko
> rc-3.0-centos-stable-latest
> 39aca91c109b 5 days ago 897MB
> starlingx/stx-ironic
> rc-3.0-centos-stable-latest
> dfe55bd7527c 5 days ago 931MB
> starlingx/stx-aodh
> rc-3.0-centos-stable-latest
> 5de9ace955ce 5 days ago 899MB
> starlingx/stx-nova-api-proxy
> rc-3.0-centos-stable-latest
> d867d8969c4f 5 days ago 791MB
> starlingx/stx-libvirt
> ...

Read more...

Pratik M. (pvmpublic) wrote :

The OpenStack install will work if you use the stable-versioned helm charts instead of stable-latest.

http://mirror.starlingx.cengn.ca/mirror/starlingx/release/3.0.0/centos/outputs/helm-charts/stx-openstack-1.0-19-centos-stable-versioned.tgz

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers