devstack-gate failure due to dpkg lock

Bug #1097592 reported by James E. Blair
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
Fix Released
High
Clark Boylan

Bug Description

06:26:03 2013-01-09 06:26:03 + sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install bridge-utils pep8 pylint python-pip screen unzip wget psmisc git lsof openssh-server vim-nox locate python-virtualenv python-unittest2 iputils-ping wget curl tcpdump euca2ools tar python-cmd2 python-netaddr gcc libxml2-dev python-dev python-eventlet python-routes python-greenlet python-sqlalchemy python-wsgiref python-pastedeploy python-xattr python-iso8601 python-setuptools python-dev python-lxml python-pastescript python-pastedeploy python-paste sqlite3 python-pysqlite2 python-sqlalchemy python-mysqldb python-webob python-greenlet python-routes libldap2-dev libsasl2-dev python-bcrypt gcc python-dateutil dnsmasq-base dnsmasq-utils kpartx parted python-mysqldb python-xattr python-lxml kvm gawk iptables ebtables sqlite3 sudo kvm libjs-jquery-tablesorter vlan curl socat python-mox python-paste python-migrate python-gflags python-greenlet python-libvirt python-libxml2 python-routes python-netaddr python-pastedeploy python-eventlet python-cheetah python-carrot python-tempita python-sqlalchemy python-suds python-lockfile python-m2crypto python-boto python-kombu python-feedparser python-iso8601 python-qpid lvm2 open-iscsi open-iscsi-utils genisoimage python-beautifulsoup python-dateutil python-paste python-pastedeploy python-anyjson python-routes python-xattr python-sqlalchemy python-webob python-kombu pylint pep8 python-eventlet python-nose python-sphinx python-mox python-kombu python-coverage python-cherrypy3 python-migrate nodejs python-netaddr curl gcc memcached python-configobj python-coverage python-dev python-eventlet python-greenlet python-netifaces python-nose python-pastedeploy python-setuptools python-simplejson python-webob python-xattr sqlite3 xfsprogs tgt lvm2

06:26:03 2013-01-09 06:26:03 E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
06:26:03 2013-01-09 06:26:03 E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

Tags: gate-failure
James E. Blair (corvus)
summary: - devstack failure due to dpkg lock
+ devstack-gate failure due to dpkg lock
Revision history for this message
Eric Harney (eharney) wrote :
James E. Blair (corvus)
Changed in openstack-ci:
status: New → Confirmed
importance: Undecided → Low
milestone: none → havana
tags: added: gate-failure
Revision history for this message
Joe Gordon (jogo) wrote :

logstash query: message:"Unable to lock the administration directory (/var/lib/dpkg/), is another process using it" AND filename:"console.html"

6 hits in last 7 days

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/68282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/68282
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=59d052082db683b5e859e10262254180382d4ed2
Submitter: Jenkins
Branch: master

commit 59d052082db683b5e859e10262254180382d4ed2
Author: Joe Gordon <email address hidden>
Date: Tue Jan 21 14:51:59 2014 -0800

    Add fingerprint for bug 1097592

    Change-Id: Ia8f68be524d46b65fae7f728eada586b54a9dabd
    Related-Bug: #1097592

Revision history for this message
Jeremy Stanley (fungi) wrote :

We should check whether unattended upgrades is running on devstack slaves for some reason.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/69259

Revision history for this message
Masayuki Igawa (igawa) wrote :

I found a similar failure in http://logs.openstack.org/89/67389/2/gate/gate-grenade-dsvm/e7a14e6/console.html
Not so many times though.

 2014-01-26 06:25:02.034 | 2014-01-26 06:25:01 dpkg: error: dpkg status database is locked by another process
 2014-01-26 06:25:02.034 | 2014-01-26 06:25:02 E: Sub-process /usr/bin/dpkg returned an error code (2)

I've posted a patch for e-r query queries/1097592.yaml.
 https://review.openstack.org/69259

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/69259
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=c9dd4bd7e502594e4186a45454f8d7b7d06f0df5
Submitter: Jenkins
Branch: master

commit c9dd4bd7e502594e4186a45454f8d7b7d06f0df5
Author: Masayuki Igawa <email address hidden>
Date: Mon Jan 27 11:22:17 2014 +0900

    Add one more fingerprint for bug 1097592

    This is also a dpkg lock issue fingerprint.

    Change-Id: Ia066c01ad1be1438533687bca2a93e2b1c49ce87
    Related-Bug: #1097592

Revision history for this message
Thierry Carrez (ttx) wrote :

So... the fun part here is that this is actually happening every day at the same time... between 06:25 and 06:43 UTC.
I suspect either unattended-upgrades or some apt DB refresh.

Revision history for this message
Thierry Carrez (ttx) wrote :

Yep, cron.daily jobs run at 6:25 and shall include apt/dpkg jobs

Revision history for this message
Thierry Carrez (ttx) wrote :

Caused 15 failures over the last 7 days, you should consider prioritizing up.

Revision history for this message
Jeremy Stanley (fungi) wrote :

We hope this will be solved by https://review.openstack.org/69442 (prioritizing that review now).

Changed in openstack-ci:
status: Confirmed → In Progress
importance: Low → High
assignee: nobody → Clark Boylan (cboylan)
Revision history for this message
Sergey Lukjanov (slukjanov) wrote :
Revision history for this message
Clark Boylan (cboylan) wrote :

Currently seeing zero hits for the last 7 days using the above query slightly edited to search all console files. I am marking this fix released.

Changed in openstack-ci:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.