CI: containers jobs fail in pingtest because volume error

Bug #1700333 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

Containers jobs gate-tripleo-ci-centos-7-ovb-containers-oooq and gate-tripleo-ci-centos-7-containers-multinode fail in pingtest:
(it's gate jobs so gates are broken atm)

http://logs.openstack.org/27/476427/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/50db285/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz#_2017-06-25_03_01_20

Resource CREATE failed: ClientException: resources.volume1: Service Unavailable (HTTP 503)

Errors in cinder:
http://logs.openstack.org/27/476427/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/50db285/logs/overcloud-controller-0/var/log/containers/cinder/cinder-scheduler.log.txt.gz

2017-06-25 03:01:42.213 6 INFO cinder.rpc [req-8f71aa40-b052-47e7-ae75-3f38410d7a18 - - - - -] Automatically selected cinder-volume objects version 1.26 as minimum service version.
2017-06-25 03:01:42.214 6 CRITICAL cinder [req-8f71aa40-b052-47e7-ae75-3f38410d7a18 - - - - -] Unhandled error: CappedVersionUnknown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.26. Most likely your environment contains only new services and you're trying to start an older one. Use `cinder-manage service list` to check that and upgrade this service.
2017-06-25 03:01:42.214 6 ERROR cinder Traceback (most recent call last):
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/bin/cinder-scheduler", line 10, in <module>
2017-06-25 03:01:42.214 6 ERROR cinder sys.exit(main())
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 53, in main
2017-06-25 03:01:42.214 6 ERROR cinder server = service.Service.create(binary='cinder-scheduler')
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 426, in create
2017-06-25 03:01:42.214 6 ERROR cinder cluster=cluster)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 152, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder *args, **kwargs)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/scheduler/manager.py", line 75, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder self.driver = importutils.import_object(scheduler_driver)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_utils/importutils.py", line 44, in import_object
2017-06-25 03:01:42.214 6 ERROR cinder return import_class(import_str)(*args, **kwargs)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 40, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder super(FilterScheduler, self).__init__(*args, **kwargs)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/scheduler/driver.py", line 77, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder self.volume_rpcapi = volume_rpcapi.VolumeAPI()
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/rpc.py", line 208, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder serializer = base.CinderObjectSerializer(obj_version_cap)
2017-06-25 03:01:42.214 6 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/objects/base.py", line 499, in __init__
2017-06-25 03:01:42.214 6 ERROR cinder raise exception.CappedVersionUnknown(version=version_cap)
2017-06-25 03:01:42.214 6 ERROR cinder CappedVersionUnknown: Unrecoverable Error: Versioned Objects in DB are capped to unknown version 1.26. Most likely your environment contains only new services and you're trying to start an older one. Use `cinder-manage service list` to check that and upgrade this service.
2017-06-25 03:01:42.214 6 ERROR cinder

LVM errors:
http://logs.openstack.org/27/476427/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq/50db285/logs/overcloud-controller-0/var/log/cinder/volume.log.txt.gz#_2017-06-25_02_46_52_785

2017-06-25 02:46:48.152 41738 INFO cinder.volume.manager [req-ebe4c4cf-523d-4132-b86a-7009031d68e3 - - - - -] Starting volume driver LVMVolumeDriver (3.0.0)
2017-06-25 02:46:52.785 41738 INFO cinder.brick.local_dev.lvm [req-ebe4c4cf-523d-4132-b86a-7009031d68e3 - - - - -] Logical Volume not found when querying LVM info. (vg_name=cinder-volumes, lv_name=cinder-volumes-pool: ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix cinder-volumes/cinder-volumes-pool
Exit code: 5
Stdout: u''
Stderr: u'File descriptor 20 (/dev/urandom) leaked on lvs invocation. Parent PID 41880: /usr/bin/python2\n Failed to find logical volume "cinder-volumes/cinder-volumes-pool"\n'
2017-06-25 02:46:53.427 41738 INFO cinder.volume.drivers.lvm [req-ebe4c4cf-523d-4132-b86a-7009031d68e3 - - - - -] Enabling LVM thin provisioning by default because no LVs exist.
2017-06-25 02:46:54.677 41738 INFO cinder.brick.local_dev.lvm [req-ebe4c4cf-523d-4132-b86a-7009031d68e3 - - - - -] Logical Volume not found when querying LVM info. (vg_name=cinder-volumes, lv_name=cinder-volumes-pool: ProcessExecutionError: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C lvs --noheadings --unit=g -o vg_name,name,size --nosuffix cinder-volumes/cinder-volumes-pool
Exit code: 5
Stdout: u''
Stderr: u'File descriptor 20 (/dev/urandom) leaked on lvs invocation. Parent PID 41896: /usr/bin/python2\n Failed to find logical volume "cinder-volumes/cinder-volumes-pool"\n'

Tags: containers
tags: removed: ci
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/477234

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/477234
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=35f189f70047f1904d4f2cef73598e56dad175bc
Submitter: Jenkins
Branch: master

commit 35f189f70047f1904d4f2cef73598e56dad175bc
Author: Martin André <email address hidden>
Date: Sun Jun 25 18:54:55 2017 +0200

    Temporarily disable containerized cinder

    The containerized cinder service was merged a bit too soon and it
    caused several issues in CI. Disable it temporarily to unblock CI until
    it matures.

    Change-Id: I8c6c0ce0011fddfec1e2de798d4fc6f34ae78de2
    Related-Bug: #1700333

tags: removed: alert
Changed in tripleo:
status: Triaged → Fix Released
milestone: none → pike-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/484872

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by Christian Schwede (<email address hidden>) on branch: master
Review: https://review.openstack.org/484872
Reason: Duplicate of https://review.openstack.org/#/c/479001/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.