Train standalone deploy failed with Error: cannot exec into container that is not running: container state improper while applying neutron::ovn_metadata' config

Bug #1855055 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Numan Siddique

Bug Description

All Train standalone deploy job got failed while applying ovn config to cinder container with following error at step4.
http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-train/64fb014/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

2019-12-03 23:42:31 | "+ STEP=4",
2019-12-03 23:42:31 | "+ TAGS=exec",
2019-12-03 23:42:31 | "+ CONFIG='include ::tripleo::profile::base::neutron::ovn_metadata'",
2019-12-03 23:42:31 | "+ EXTRA_ARGS=",
2019-12-03 23:42:31 | "+ '[' -d /tmp/puppet-etc ']'",
2019-12-03 23:42:31 | "+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet",
2019-12-03 23:42:31 | "+ echo '{\"step\": 4}'",
2019-12-03 23:42:31 | "+ export FACTER_deployment_type=containers",
2019-12-03 23:42:31 | "+ FACTER_deployment_type=containers",
2019-12-03 23:42:31 | "+ set +e",
2019-12-03 23:42:31 | "+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules --tags exec -e 'noop_resource('\\''package'\\''); include ::tripleo::profile::base::neutron::ovn_metadata'",
2019-12-03 23:42:31 | "Warning: Support for ruby version 2.0.0 is deprecated and will be removed in a future release. See https://puppet.com/docs/puppet/latest/system_requirements.html for a list of supported ruby versions.",
2019-12-03 23:42:31 | " (location: /usr/share/ruby/vendor_ruby/puppet.rb:130:in `<module:Puppet>')",
2019-12-03 23:42:31 | "Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
2019-12-03 23:42:31 | "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5",
2019-12-03 23:42:31 | " (file: /etc/puppet/hiera.yaml)",
2019-12-03 23:42:31 | "Warning: Undefined variable '::deploy_config_name'; \\n (file & line not available)",
2019-12-03 23:42:31 | "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not available)",
2019-12-03 23:42:31 | "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
2019-12-03 23:42:31 | "Warning: Unknown variable: 'dhcp_agents_per_net'. (file: /etc/puppet/modules/tripleo/manifests/profile/base/neutron.pp, line: 154, column: 37)",
2019-12-03 23:42:31 | "Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
2019-12-03 23:42:31 | "Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/neutron/manifests/init.pp\", 468]:[\"/etc/puppet/modules/tripleo/manifests/profile/base/neutron/ovn_metadata.pp\", 62]",
2019-12-03 23:42:31 | " (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')",
2019-12-03 23:42:31 | "Warning: ModuleLoader: module 'oslo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\\n (file & line not available)",
2019-12-03 23:42:31 | "+ rc=2",
2019-12-03 23:42:31 | "+ set -e",
2019-12-03 23:42:31 | "+ set +ux",
2019-12-03 23:42:31 | "WARNING: The same type, major and minor should not be used for multiple devices.",
2019-12-03 23:42:31 | "Error: cannot exec into container that is not running: container state improper",
2019-12-03 23:42:31 | "+ command -v python3",
2019-12-03 23:42:31 | "+ command -v python2",
2019-12-03 23:42:31 | "+ python2 /container-config-scripts/placement_wait_for_service.py",
2019-12-03 23:42:31 | "+ python2 /container-config-scripts/nova_wait_for_api_service.py"
2019-12-03 23:42:31 | ]
2019-12-03 23:42:31 | }

Reason: https://review.opendev.org/#/c/694453/ [Use ovn-ctl script to start OVN ovsdb-servers] landed in kolla and in order to fix that https://review.opendev.org/#/c/690105/ in tht needs to be also backported to stable/train

Some more logs:

http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/c40d9fa/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario001-standalone-train/ac1fa39/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-train/92ab7a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

Changed in tripleo:
assignee: nobody → Numan Siddique (numansiddique)
status: Confirmed → In Progress
Revision history for this message
Marios Andreou (marios-b) wrote :

 "ovn standalone: Start OVN ovsdb-servers using the kolla startup script Change-Id: I1fbfaf43af17b558497fd2b46fc4278b4703ec74 * https://review.opendev.org/#/c/697237/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/697285

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/697285
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0f1ffe9a10ea27b404a638fc11f12ad8cb42609d
Submitter: Zuul
Branch: stable/train

commit 0f1ffe9a10ea27b404a638fc11f12ad8cb42609d
Author: Numan Siddique <email address hidden>
Date: Fri Nov 15 19:40:44 2019 +0530

    ovn-db standalone: Create ovndbs before starting the OVN ovsdb servers

    The kolla commit [1] deleted the extend_start script for ovn-nb-db-server/ovn-sb-db-server
    kolla images. This script created the ovn dbs from the schema. After this commit
    tripleo CI is broken as the ovn dbs are not present.

    To fix this issue, this patch first creates OVN dbs before starting the OVN ovsdb-servers.

    [1] - I31a372d801dace92950cd85b29b308431f78bbdb
    Closes-bug: #1852682
    Closes-bug: #1855055

    (cherry-picked from master commit e8ecbecb6a36c1222a5ff4459ceaa6e7e3b9a680)
    Change-Id: I9640529455e1d14c98c3b53993a71d77de8d7a25
    Signed-off-by: Numan Siddique <email address hidden>

tags: added: in-stable-train
Revision history for this message
Marios Andreou (marios-b) wrote :
Changed in tripleo:
status: In Progress → Confirmed
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.