[Fuel CI] E: Error locking chroot session: skipping <pkg>

Bug #1559300 reported by Aleksandra Fedorova
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Sergey Kulanov

Bug Description

Sometimes on Fuel CI deb package builds fail on sbuild chroot step

 https://ci.fuel-infra.org/job/master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy/687/console

It is currently 100% reproducible on ci-slave07-scc

jenkins@ci-slave07-scc:~$ ps aux | grep sbuild

jenkins 17482 0.0 0.0 108092 9276 ? Sl 20:33 0:00 docker run --dns 8.8.8.8 --privileged --rm -v /var/cache/docker-builder/sbuild:/srv/images sbuild:latest bash -c sbuild-update -udcar trusty
root 17488 0.3 0.0 73192 20944 ? Ss 20:33 0:00 /usr/bin/perl -w /usr/bin/sbuild-update -udcar trusty
root 17638 0.0 0.0 103376 5068 ? S 20:33 0:00 schroot -d / -c trusty-amd64-sbuild-51d0b26a-586c-42a2-b012-4d1d948ffa06 --run-session -q -u root -p -- perl -e use strict; use warnings; use POSIX; use FileHandle; my $lockfile="/var/lock/sbuild"; my $try = 0; repeat: if (!sysopen( F, $lockfile, O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0644 )){ .if ($! == EEXIST) { . # lock file exists, wait . goto repeat if !open( F, "<$lockfile" ); . my $line = <F>; . my ($job, $pid, $user); . close( F ); . if ($line !~ /^(\S+)\s+(\S+)\s+(\S+)/) { ..print STDERR "Bad lock file contents ($lockfile) -- still trying\n"; . } else { ..($job, $pid, $user) = ($1, $2, $3); ..if (kill( 0, $pid ) == 0 && $! == ESRCH) { .. # process no longer exists, remove stale lock .. print STDERR "Removing stale lock file $lockfile ". ..."(job $job, pid $pid, user $user)\n"; .. if (!unlink($lockfile)) { ...if ($! != ENOENT) { ... print STDERR "Cannot remove chroot lock file $lockfile: $!\n"; ... exit 1; ...} .. } ..} . } . ++$try; . if ($try > 120) { ..print STDERR "Lockfile $lockfile still present after " . .. 120 * 5 . " seconds -- giving up\n"; ..exit 1; . } . print STDERR "Another sbuild process (job $job, pid $pid by user $user) is currently using the build chroot; waiting...\n" ..if $try == 1; . sleep 5; . goto repeat; .} else { . print STDERR "Failed to create lock file $lockfile: $!\n"; . exit 1; .} } F->print("SBUILD_UPDATE 1 root\n"); F->close(); exit 0;
root 17639 0.0 0.0 25964 4336 ? S 20:33 0:00 perl -e use strict; use warnings; use POSIX; use FileHandle; my $lockfile="/var/lock/sbuild"; my $try = 0; repeat: if (!sysopen( F, $lockfile, O_WRONLY|O_CREAT|O_TRUNC|O_EXCL, 0644 )){ .if ($! == EEXIST) { . # lock file exists, wait . goto repeat if !open( F, "<$lockfile" ); . my $line = <F>; . my ($job, $pid, $user); . close( F ); . if ($line !~ /^(\S+)\s+(\S+)\s+(\S+)/) { ..print STDERR "Bad lock file contents ($lockfile) -- still trying\n"; . } else { ..($job, $pid, $user) = ($1, $2, $3); ..if (kill( 0, $pid ) == 0 && $! == ESRCH) { .. # process no longer exists, remove stale lock .. print STDERR "Removing stale lock file $lockfile ". ..."(job $job, pid $pid, user $user)\n"; .. if (!unlink($lockfile)) { ...if ($! != ENOENT) { ... print STDERR "Cannot remove chroot lock file $lockfile: $!\n"; ... exit 1; ...} .. } ..} . } . ++$try; . if ($try > 120) { ..print STDERR "Lockfile $lockfile still present after " . .. 120 * 5 . " seconds -- giving up\n"; ..exit 1; . } . print STDERR "Another sbuild process (job $job, pid $pid by user $user) is currently using the build chroot; waiting...\n" ..if $try == 1; . sleep 5; . goto repeat; .} else { . print STDERR "Failed to create lock file $lockfile: $!\n"; . exit 1; .} } F->print("SBUILD_UPDATE 1 root\n"); F->close(); exit 0;

Tags: area-build
Changed in fuel:
status: New → Confirmed
Revision history for this message
Aleksandra Fedorova (bookwar) wrote :
Changed in fuel:
assignee: Fuel build team (fuel-build) → Sergey Kulanov (skulanov)
status: Confirmed → In Progress
Revision history for this message
Sergey Kulanov (skulanov) wrote :

All logic with sbuild chroot lock is here [1], and stale lock file should be removed without issues [2], but since we have some non-standard sbuild usage (in docker) it has PID 1 and as the result cannot be killed since PID 1 process is running:

/var/cache/docker-builder/sbuild/trusty-amd64/run/lock$ cat sbuild

SBUILD_UPDATE 1 root

so the only way is to remove lock file manually

rm /var/cache/docker-builder/sbuild/trusty-amd64/run/lock/sbuild

but you need to have root access

[1]. https://github.com/geofft/sbuild/blob/master/lib/Sbuild/Chroot.pm#L322
[2]. https://github.com/geofft/sbuild/blob/master/lib/Sbuild/Chroot.pm#L355

ps. the issue is fixed:
https://ci.fuel-infra.org/job/master.fuel-web.pkgs.ubuntu.review_fuel_web_deploy/702/console
and
mirrors was updated to
deb http://mirror.seed-us1.fuel-infra.org/pkgs/ubuntu/ trusty main universe multiverse restricted
deb http://mirror.seed-us1.fuel-infra.org/pkgs/ubuntu/ trusty-updates main universe multiverse restricted

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.