1.9.0 Error grabbing cluster configuration lock causes importing to fail.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Gavin Panella |
Bug Description
Started off as a fully up to date 1.8 MAAS server.
I upgraded it to 1.9.0 from the experimental ppa and did not reboot, it was done by doing a dist-upgrade after adding the experimental PPA.
Today, I switched sources from the releases stream to the daily stream so I could get wily images. After setting it to the daily stream, I clicked on the images tab and saw that it was updated and now showed 15.10 as an available image.
I selected 15.10 and applied the changes. The region then successfully downloaded the image and boot stuff, and it is visible here:
bladernr@
total 1858372
drwxr-xr-x 2 maas maas 4096 Sep 10 13:14 ./
drwxr-xr-x 3 maas maas 4096 Sep 10 13:14 ../
-rw-r--r-- 10 maas maas 29305609 Sep 10 12:08 boot-initrd
-rw-r--r-- 10 maas maas 6748784 Sep 10 12:07 boot-kernel
-rw-r--r-- 10 maas maas 23560748 Sep 10 12:08 di-initrd
-rw-r--r-- 10 maas maas 6750712 Sep 10 12:07 di-kernel
-rw-r--r-- 10 maas maas 1468006400 Sep 10 12:07 root-image
-rw-r--r-- 10 maas root 368578444 Sep 10 12:08 root-tgz
bladernr@
lrwxrwxrwx 1 maas maas 53 Sep 10 13:14 current -> /var/lib/
Current also points to the correct snapshot directory.
However, while Images shows wily as available:
15.10 amd64 421.1 MB 0 Thu, 10 Sep. 2015 13:14:41
15.04 amd64 408.2 MB 0 Thu, 10 Sep. 2015 13:14:42
14.04 LTS amd64 491.8 MB 3 Thu, 10 Sep. 2015 13:14:41
the cluster is out of sync and 15.10 is NOT a valid install image for deploying a system.
Looking at the maas log, I did notice these error messages (these are pulled from maas.log):
## This has been occurring since I updated to 1.9.0
Sep 10 08:14:34 critical-maas maas.bootresources: [INFO] Importing images from source: http://
.com/images/
Sep 10 08:14:35 critical-maas maas.bootresources: [INFO] Finished importing of boot images from 1 source(
s).
Sep 10 08:14:35 critical-maas maas.import-images: [INFO] Started importing boot images.
Sep 10 08:14:35 critical-maas maas.import-images: [WARNING] I/O error while syncing boot images. If this
problem persists, verify network connectivity and disk usage.
Sep 10 08:14:35 critical-maas maas.import-images: [WARNING] Finished importing boot images, the region do
es not have any boot images available.
Sep 10 08:19:33 critical-maas maas.import-images: [INFO] Started importing boot images.
Sep 10 08:19:33 critical-maas maas.import-images: [INFO] Finished importing boot images, the region does not have any new images.
Sep 10 08:19:38 critical-maas maas.boot_
#This is after I switched to the Daily stream this morning.
Sep 10 11:56:51 critical-maas maas.bootsources: [INFO] Updated boot sources cache.
Sep 10 11:56:51 critical-maas maas.bootresources: [INFO] Started importing of boot images from 1 source(s).
Sep 10 11:56:53 critical-maas maas.bootresources: [INFO] Importing images from source: http://
Sep 10 11:59:33 critical-maas maas.import-images: [INFO] Started importing boot images.
Sep 10 11:59:34 critical-maas maas.import-images: [INFO] Finished importing boot images, the region does not have any new images.
Sep 10 11:59:39 critical-maas maas.boot_
Sep 10 12:05:10 critical-maas maas.bootresources: [INFO] Finished importing of boot images from 1 source(s).
Sep 10 12:05:10 critical-maas maas.import-images: [INFO] Started importing boot images.
Sep 10 12:08:25 critical-maas maas.import-images: [INFO] Writing boot image metadata and iSCSI targets.
Sep 10 12:08:25 critical-maas maas.import-images: [INFO] Installing boot images snapshot /var/lib/
Sep 10 12:08:28 critical-maas maas.import-images: [INFO] Updating boot image iSCSI targets.
Sep 10 12:08:28 critical-maas maas.import-images: [INFO] Cleaning up old snapshots and cache.
Sep 10 12:08:28 critical-maas maas.import-images: [INFO] Finished importing boot images.
Sep 10 12:14:36 critical-maas maas.bootsources: [INFO] Updated boot sources cache.
Related branches
- Andres Rodriguez (community): Approve
-
Diff: 93 lines (+37/-15)2 files modifiedsrc/provisioningserver/config.py (+35/-14)
src/provisioningserver/rpc/boot_images.py (+2/-1)
Changed in maas: | |
assignee: | nobody → Gavin Panella (allenap) |
Changed in maas: | |
status: | Fix Committed → Fix Released |
After writing this up, I rebooted the maas server and noticed a couple things I didnt notice before... first, in clusterd.log there is a traceback when the cluster restarted:
2015-09-10 14:21:47-0400 [-] Unhandled error in Deferred: python2. 7/threading. py", line 783, in __bootstrap
self. __bootstrap_ inner() python2. 7/threading. py", line 810, in __bootstrap_inner
self. run() python2. 7/threading. py", line 763, in run
self. __target( *self._ _args, **self.__kwargs) python2. 7/dist- packages/ twisted/ python/ threadpool. py", line 191, in _worker python2. 7/dist- packages/ twisted/ python/ context. py", line 118, in callWithContext text(). callWithContext (ctx, func, *args, **kw) python2. 7/dist- packages/ twisted/ python/ context. py", line 81, in callWithContext python2. 7/dist- packages/ provisioningser ver/utils/ twisted. py", line 158, in wrapper python2. 7/dist- packages/ provisioningser ver/rpc/ boot_images. py", line 92, in _run_import
reload_ boot_images( ) python2. 7/dist- packages/ provisioningser ver/rpc/ boot_images. py", line 57, in reload_boot_images
CACHED_ BOOT_IMAGES = tftppath. list_boot_ images( config. tftp_root) python2. 7/dist- packages/ provisioningser ver/boot/ tftppath. py", line 266, in list_boot_images
extract_ image_params( path, metadata) for path in paths)) python2. 7/dist- packages/ provisioningser ver/boot/ tftppath. py", line 266, in <genexpr>
extract_ image_params( path, metadata) for path in paths)) python2. 7/dist- packages/ provisioningser ver/boot/ tftppath. py", line 180, in extract_ image_params python2. 7/dist- packages/ provisioningser ver/drivers/ osystem/ custom. py", line 58, in get_xinstall_ parameters ation.open( ) as config: python2. 7/contextlib. py", line 17, in __enter__ python2. 7/dist- packages/ provisioningser ver/config. py", line 674, in open open(filepath) as store: python2. 7/contextlib. py", line 17, in __enter__ python2. 7/dist- packages/ provisioningser ver/config. py", line 549, in open path).wait( timeout= 5.0): python2. 7/contextlib. py", line 17, in __enter__ python2. 7/dist- packages/ provisioningser ver/utils/ fs.py", line 440, in wait le(self. fslock. name)
2015-09-10 14:21:47-0400 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
--- <exception caught here> ---
File "/usr/lib/
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/
return self.currentCon
File "/usr/lib/
return func(*args,**kw)
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
arch, subarch, release, label)
File "/usr/lib/
with ClusterConfigur
File "/usr/lib/
return self.gen.next()
File "/usr/lib/
with cls.backend.
File "/usr/lib/
return self.gen.next()
File "/usr/lib/
with RunLock(
File "/usr/lib/
return self.gen.next()
File "/usr/lib/
raise self.NotAvailab
provisio...