stx-openstack apply-failed

Bug #1889023 reported by admin@zhangqi.net
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
StarlingX
Won't Fix
Low
Unassigned

Bug Description

Brief Description
-----------------
I am tring to install starlingx R3.0 AIO virutalized.
application-apply stx-openstack failed finally.

Severity
--------

Steps to Reproduce
------------------
1. install ubuntu 16.04 on baremetal
2. git clone https://opendev.org/starlingx/tools.git and setup virutal machine via setup_configuration.sh -i simplex
3. run ansible-playbook
4. upload stx-openstack and apply

Expected Behavior
------------------
expect all k8s container in running or complete, openstack setup complete and ready to use.

Actual Behavior
----------------
apply failed at 54%.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
<One node system, Two node system, Multi-node system, Dedicated storage, https, IPv4, IPv6 etc.>
one node system, all in one, virtualized environment

Branch/Pull Time/Commit
-----------------------
controller-0:~# docker images | grep ^starlingx
starlingx/stx-placement rc-3.0-centos-stable-latest e273740d28c4 5 days ago 861MB
starlingx/stx-nova rc-3.0-centos-stable-latest a64d25525301 5 days ago 1.3GB
starlingx/stx-neutron rc-3.0-centos-stable-latest 23ccbedea543 5 days ago 944MB
starlingx/stx-keystone rc-3.0-centos-stable-latest 2d1d06bcb3fd 5 days ago 896MB
starlingx/stx-horizon rc-3.0-centos-stable-latest 99c3209b44fd 5 days ago 1.11GB
starlingx/stx-heat rc-3.0-centos-stable-latest 5ea3a26624d1 5 days ago 938MB
starlingx/stx-gnocchi rc-3.0-centos-stable-latest 8ab74fec318b 5 days ago 932MB
starlingx/stx-glance rc-3.0-centos-stable-latest 5f8a852ce1d5 5 days ago 914MB
starlingx/stx-cinder rc-3.0-centos-stable-latest 688337608d98 5 days ago 972MB
starlingx/stx-ceilometer rc-3.0-centos-stable-latest e48a281f9b57 5 days ago 953MB
starlingx/stx-panko rc-3.0-centos-stable-latest 39aca91c109b 5 days ago 897MB
starlingx/stx-ironic rc-3.0-centos-stable-latest dfe55bd7527c 5 days ago 931MB
starlingx/stx-aodh rc-3.0-centos-stable-latest 5de9ace955ce 5 days ago 899MB
starlingx/stx-nova-api-proxy rc-3.0-centos-stable-latest d867d8969c4f 5 days ago 791MB
starlingx/stx-libvirt rc-3.0-centos-stable-latest 67d7c4cf814e 5 days ago 758MB
starlingx/stx-ovs rc-3.0-centos-stable-latest c606795ff805 5 days ago 799MB
starlingx/stx-mariadb rc-3.0-centos-stable-latest f27522485de4 5 days ago 522MB
starlingx/stx-fm-rest-api rc-3.0-centos-stable-latest 10cce1323fe8 5 days ago 842MB
starlingx/stx-keystone-api-proxy rc-3.0-centos-stable-latest 887497308b5d 5 days ago 994MB
starlingx/k8s-cni-sriov master-centos-stable-latest 49f3250071d9 6 days ago 9.81MB
starlingx/k8s-plugins-sriov-network-device master-centos-stable-latest ab8eb0c6b3bc 3 weeks ago 25.6MB
starlingx/multus v3.2.16 4c3da64245c6 10 months ago 308MB
starlingx/ceph-config-helper v1.15.0 e9e28af77f24 12 months ago 820MB

Last Pass
---------

Timestamp/Logs
--------------

Test Activity
-------------
Feature Testing

Workaround
----------
Describe workaround if available

Revision history for this message
admin@zhangqi.net (r-sdmin-0) wrote :
Revision history for this message
admin@zhangqi.net (r-sdmin-0) wrote :
Revision history for this message
admin@zhangqi.net (r-sdmin-0) wrote :
Download full text (12.2 KiB)

During apply, I found another error in neutron-ovs-agent pod.

controller-0:~# kubectl get pods -n openstack
NAME READY STATUS RESTARTS AGE
cinder-api-dbc984dd5-lz829 1/1 Running 0 25m
cinder-backup-794588786b-2hpzh 1/1 Running 0 25m
cinder-backup-storage-init-pv4qn 0/1 Completed 0 25m
cinder-bootstrap-blrzz 0/1 Completed 0 25m
cinder-create-internal-tenant-qpbch 0/1 Completed 0 25m
cinder-db-init-z9ghz 0/1 Completed 0 25m
cinder-db-sync-w8tss 0/1 Completed 0 25m
cinder-ks-endpoints-bx4hd 0/9 Completed 0 25m
cinder-ks-service-jbh68 0/3 Completed 0 25m
cinder-ks-user-4fs99 0/1 Completed 0 25m
cinder-rabbit-init-25kzm 0/1 Completed 0 25m
cinder-scheduler-84f6549-d5qzn 1/1 Running 0 25m
cinder-storage-init-jsnv7 0/1 Completed 0 25m
cinder-volume-5d9bbc778f-xl9mj 1/1 Running 0 25m
cinder-volume-usage-audit-1595814300-c7xxs 0/1 Completed 0 13m
cinder-volume-usage-audit-1595814600-d6r9n 0/1 Completed 0 8m30s
cinder-volume-usage-audit-1595814900-dp6tj 0/1 Completed 0 3m27s
glance-api-66c8f55d47-5fmhz 1/1 Running 0 28m
glance-db-init-cf8z5 0/1 Completed 0 28m
glance-db-sync-m79jq 0/1 Completed 0 28m
glance-ks-endpoints-47nq2 0/3 Completed 0 28m
glance-ks-service-lmq56 0/1 Completed 0 28m
glance-ks-user-8hbvq 0/1 Completed 0 28m
glance-rabbit-init-fh22j 0/1 Completed 0 28m
glance-storage-init-twmg4 0/1 Completed 0 28m
ingress-c74489c96-mbglp 1/1 Running 0 32m
ingress-error-pages-5889bfcf6c-29ks2 1/1 Running 0 32m
keystone-api-7865c6cb67-8rjkv 1/1 Running 0 29m
keystone-bootstrap-82psp 0/1 Completed 0 29m
keystone-credential-setup-6jnqc 0/1 Completed 0 29m
keystone-db-init-6s2jx 0/1 Completed 0 29m
keystone-db-sync-thvw6 0/1 Completed 0 29m
keystone-domain-manage-jfg4x 0/1 Completed 0 29m
keystone-fernet-setup-7m4xv 0/1 ...

Revision history for this message
admin@zhangqi.net (r-sdmin-0) wrote :

Finally solve this problem by the following steps:

1. docker run -it registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest bash
vi /var/lib/openstack/lib/python2.7/site-packages/neutron/cmd/sanity_check.py

comment line 347:
347 #cfg.CONF.set_default('vf_management', True)

2. docker commit -p XXXX NEW_IMAGE_TAG
3. docker tag NEW_IMAGE_TAG registry.local:9001/docker.io/starlingx/stx-neutron:rc-3.0-centos-stable-latest
4. kubectl delete pod -n openstack neutron-ovs-agent-controller-0-937646f6-bgtnf

then, if lucky, k8s can start neutron-ovs-agent again and we finally make openstack running.

Ghada Khalil (gkhalil)
tags: added: stx.3.0 stx.distro.openstack
Revision history for this message
Pratik M. (pvmpublic) wrote :

I hit the same issue, and so did another user here:
http://lists.starlingx.io/pipermail/starlingx-discuss/2020-July/009242.html

In my case, I have a newly installed AIO-DX and then try to apply stx-openstack. The relevant logs snippets are attached.

Revision history for this message
Steven Webster (swebster-wr) wrote :
Revision history for this message
admin@zhangqi.net (r-sdmin-0) wrote : Re: [Bug 1889023] Re: stx-openstack apply-failed
Download full text (4.4 KiB)

Yes I did exactly the same thing to solve this problem.

On Thu, Aug 6, 2020 at 9:21 PM Steven Webster <email address hidden>
wrote:

> Relevant:
>
> http://lists.starlingx.io/pipermail/starlingx-
> discuss/2020-July/009270.html
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1889023
>
> Title:
> stx-openstack apply-failed
>
> Status in StarlingX:
> New
>
> Bug description:
> Brief Description
> -----------------
> I am tring to install starlingx R3.0 AIO virutalized.
> application-apply stx-openstack failed finally.
>
>
> Severity
> --------
>
>
> Steps to Reproduce
> ------------------
> 1. install ubuntu 16.04 on baremetal
> 2. git clone https://opendev.org/starlingx/tools.git and setup virutal
> machine via setup_configuration.sh -i simplex
> 3. run ansible-playbook
> 4. upload stx-openstack and apply
>
>
> Expected Behavior
> ------------------
> expect all k8s container in running or complete, openstack setup
> complete and ready to use.
>
>
> Actual Behavior
> ----------------
> apply failed at 54%.
>
>
>
> Reproducibility
> ---------------
> 100% reproducible
>
> System Configuration
> --------------------
> <One node system, Two node system, Multi-node system, Dedicated storage,
> https, IPv4, IPv6 etc.>
> one node system, all in one, virtualized environment
>
>
> Branch/Pull Time/Commit
> -----------------------
> controller-0:~# docker images | grep ^starlingx
> starlingx/stx-placement
> rc-3.0-centos-stable-latest
> e273740d28c4 5 days ago 861MB
> starlingx/stx-nova
> rc-3.0-centos-stable-latest
> a64d25525301 5 days ago 1.3GB
> starlingx/stx-neutron
> rc-3.0-centos-stable-latest
> 23ccbedea543 5 days ago 944MB
> starlingx/stx-keystone
> rc-3.0-centos-stable-latest
> 2d1d06bcb3fd 5 days ago 896MB
> starlingx/stx-horizon
> rc-3.0-centos-stable-latest
> 99c3209b44fd 5 days ago 1.11GB
> starlingx/stx-heat
> rc-3.0-centos-stable-latest
> 5ea3a26624d1 5 days ago 938MB
> starlingx/stx-gnocchi
> rc-3.0-centos-stable-latest
> 8ab74fec318b 5 days ago 932MB
> starlingx/stx-glance
> rc-3.0-centos-stable-latest
> 5f8a852ce1d5 5 days ago 914MB
> starlingx/stx-cinder
> rc-3.0-centos-stable-latest
> 688337608d98 5 days ago 972MB
> starlingx/stx-ceilometer
> rc-3.0-centos-stable-latest
> e48a281f9b57 5 days ago 953MB
> starlingx/stx-panko
> rc-3.0-centos-stable-latest
> 39aca91c109b 5 days ago 897MB
> starlingx/stx-ironic
> rc-3.0-centos-stable-latest
> dfe55bd7527c 5 days ago 931MB
> starlingx/stx-aodh
> rc-3.0-centos-stable-latest
> 5de9ace955ce 5 days ago 899MB
> starlingx/stx-nova-api-proxy
> rc-3.0-centos-stable-latest
> d867d8969c4f 5 days ago 791MB
> starlingx/stx-libvirt
> ...

Read more...

Revision history for this message
Pratik M. (pvmpublic) wrote :

The OpenStack install will work if you use the stable-versioned helm charts instead of stable-latest.

http://mirror.starlingx.cengn.ca/mirror/starlingx/release/3.0.0/centos/outputs/helm-charts/stx-openstack-1.0-19-centos-stable-versioned.tgz

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Closing as stx.3.0 is EOL as of Dec 2020

Changed in starlingx:
importance: Undecided → Low
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.