Image status stays out-of-sync after adding custom image

Bug #1581130 reported by Mark Casey on 2016-05-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Blake Rouse

Bug Description

Xenial64, 2.0.0 (beta4+bzr4985)

The commands I ran are below, followed by the full console log. The vars and handful of if statements are present because this is me walking through a script by hand - except the part where I remove the images added in previous attempts. Unsure where to go from here.

Command history:
=======================

if [[ -z "${MAAS_ADMIN_USER:-}" ]]; then MAAS_ADMIN_USER="admin"; fi
if [[ -z "${MAAS_ADMIN_EMAIL:-}" ]]; then <email address hidden>"; fi
if [[ -z "${MAAS_ADMIN_PASS:-}" ]]; then MAAS_ADMIN_PASS="admin"; fi
MAASVM_MGMTNET_IP=10.10.1.17
MAASVM_API_URL="http://${MAASVM_MGMTNET_IP}:5240/MAAS/api/2.0"
MAAS_ADMIN_APIKEY="$(maas-region apikey --username ${MAAS_ADMIN_USER})"
maas login "${MAAS_ADMIN_USER}" "${MAASVM_API_URL}" "${MAAS_ADMIN_APIKEY}"
maas "${MAAS_ADMIN_USER}" boot-resource delete 18
maas "${MAAS_ADMIN_USER}" boot-resource delete 19
source <(wget -O- http://stable.release.core-os.net/amd64-usr/current/version.txt)
wget -nv http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
bunzip2 coreos_production_image.bin.bz2
tar -czvf coreos_production_image.bin.tgz coreos_production_image.bin
file coreos_production_image.bin
set -x
maas "${MAAS_ADMIN_USER}" boot-resources create name=custom/coreos_stable_"${COREOS_BUILD}"_"${COREOS_BRANCH}"_"${COREOS_PATCH}" architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz
set +x
grep -r coreos /var/log/maas/

Console log:
=======================

root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_USER:-}" ]]; then MAAS_ADMIN_USER="admin"; fi
root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_EMAIL:-}" ]]; then <email address hidden>"; fi
root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_PASS:-}" ]]; then MAAS_ADMIN_PASS="admin"; fi
root@maas:/home/vagrant# MAASVM_MGMTNET_IP=10.10.1.17
root@maas:/home/vagrant# MAASVM_API_URL="http://${MAASVM_MGMTNET_IP}:5240/MAAS/api/2.0"
root@maas:/home/vagrant# MAAS_ADMIN_APIKEY="$(maas-region apikey --username ${MAAS_ADMIN_USER})"
root@maas:/home/vagrant# maas login "${MAAS_ADMIN_USER}" "${MAASVM_API_URL}" "${MAAS_ADMIN_APIKEY}"

You are now logged in to the MAAS server at
http://10.10.1.17:5240/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help

root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resource delete 18
BootResource matching query does not exist.
root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resource delete 19
Success.
Machine-readable output follows:

root@maas:/home/vagrant# source <(wget -O- http://stable.release.core-os.net/amd64-usr/current/version.txt)
--2016-05-12 16:10:29-- http://stable.release.core-os.net/amd64-usr/current/version.txt
Resolving stable.release.core-os.net (stable.release.core-os.net)... 104.16.21.26, 104.16.20.26, 2400:cb00:2048:1::6810:141a, ...
Connecting to stable.release.core-os.net (stable.release.core-os.net)|104.16.21.26|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 161 [text/plain]
Saving to: 'STDOUT'

- 100%[========================================================================================================================================>] 161 --.-KB/s in 0s

2016-05-12 16:10:29 (28.5 MB/s) - written to stdout [161/161]

root@maas:/home/vagrant# wget -nv http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
2016-05-12 16:11:01 URL:http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2 [232529993/232529993] -> "coreos_production_image.bin.bz2" [1]
root@maas:/home/vagrant# bunzip2 coreos_production_image.bin.bz2
root@maas:/home/vagrant# tar -czvf coreos_production_image.bin.tgz coreos_production_image.bin
coreos_production_image.bin
root@maas:/home/vagrant# file coreos_production_image.bin
coreos_production_image.bin: DOS/MBR boot sector
root@maas:/home/vagrant# set -x
root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resources create name=custom/coreos_stable_"${COREOS_BUILD}"_"${COREOS_BRANCH}"_"${COREOS_PATCH}" architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz+ maas admin boot-resources create name=custom/coreos_stable_899_17_0 architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz
root@maas:/home/vagrant# set +x
+ set +x
root@maas:/home/vagrant# grep -r coreos /var/log/maas/
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160512/root-dd. mirrors=[]
/var/log/maas/regiond.log:2016-05-12 14:29:39 [-] 127.0.0.1 - - [12/May/2016:14:29:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 14:29:45 [-] 127.0.0.1 - - [12/May/2016:14:29:45 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 14:49:39 [-] 127.0.0.1 - - [12/May/2016:14:49:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:09:39 [-] 127.0.0.1 - - [12/May/2016:15:09:39 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:29:38 [-] 127.0.0.1 - - [12/May/2016:15:29:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:29:59 [-] 127.0.0.1 - - [12/May/2016:15:29:58 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:49:39 [-] 127.0.0.1 - - [12/May/2016:15:49:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 16:09:39 [-] 127.0.0.1 - - [12/May/2016:16:09:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 16:19:04 [-] 127.0.0.1 - - [12/May/2016:16:19:03 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160512/root-dd HTTP/1.1" 404 1354 "-" "python-requests/2.9.1"
/var/log/maas/maas.log:May 12 14:29:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 14:49:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:09:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:29:38 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:49:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 16:09:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
root@maas:/home/vagrant# date
Thu May 12 16:19:23 UTC 2016
root@maas:/home/vagrant#

Related branches

Mark Casey (mark-casey) wrote :
Download full text (12.0 KiB)

I turned in DEBUG and ran regiond in foreground and didn't get much more info. This output shows me (from another console) deleting a previous attempt and then retrying the boot-resources create. (the change in the image name from the initial report was my doing to see if the underscores or name length could be an issue)

I would really appreciate some next steps that I could undertake to help diagnose this, even if there is no immediate solution.

2016-05-16 16:52:01 [-] 10.10.1.17 - - [16/May/2016:16:52:01 +0000] "DELETE /MAAS/api/2.0/boot-resources/33/ HTTP/1.1" 204 - "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:08 [-] 127.0.0.1 - - [16/May/2016:16:52:08 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 102 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-05-16 16:52:32 [-] 10.10.1.17 - - [16/May/2016:16:52:32 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:32 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/...

Mark Casey (mark-casey) wrote :
Download full text (9.4 KiB)

Same result with an ubuntu image copied out of image cache (uploaded as tgz instead of ddtgz... I think that was right...)

COMMANDS:
================
diff -u /var/lib/maas/boot-resources/cache/root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 /var/lib/maas/boot-resources/snapshot-20160513-184938/ubuntu/amd64/hwe-x/xenial/release/root-tgz
cp -a /var/lib/maas/boot-resources/cache/root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 .
mv root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 /tmp/ubuntu1604-tgz
maas admin boot-resources create -d name=custom/foo architecture=amd64/generic filetype=tgz content@=/tmp/ubuntu1604-tgz

CONSOLE:
=================
2016-05-16 19:21:08 [-] 127.0.0.1 - - [16/May/2016:19:21:08 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 103 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:18 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:20 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:20 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP...

Read more...

Mark Casey (mark-casey) wrote :

Regarding the 404's from not being able to D/L the image after it is created. If I copy the failed '/MAAS/images-stream/custom/...' URL out of the logs and try to wget it I also get a 404.

If I use 'maas-region dbshell --installed' to remove "custom/" from the bootresource's name:

UPDATE "maasserver_bootresource" SET "name" = 'coreos' WHERE ("maasserver_bootresource"."name" = 'custom/coreos' AND "maasserver_bootresource"."architecture" = 'amd64/generic');

Then I can use the same URL that had just 404'ed to successfully manually D/L the image. Of course that breaks the web interface (and from what I can see on the half broken interface it hasn't actually gotten rid of the 'out-of-sync' message) so it doesn't really get me into a usable state.

So AFAICT seems to be a disconnect between image storage and image download regarding how the names of custom images are special.

Changed in maas:
status: New → Triaged
Changed in maas:
status: Triaged → In Progress
importance: Undecided → Critical
assignee: nobody → Blake Rouse (blake-rouse)
milestone: none → 2.0.0
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers