MAAS

Image status stays out-of-sync after adding custom image

Bug #1581130 reported by Mark Casey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
MAAS 2.0.0

Bug Description

Xenial64, 2.0.0 (beta4+bzr4985)

The commands I ran are below, followed by the full console log. The vars and handful of if statements are present because this is me walking through a script by hand - except the part where I remove the images added in previous attempts. Unsure where to go from here.

Command history:
=======================

if [[ -z "${MAAS_ADMIN_USER:-}" ]]; then MAAS_ADMIN_USER="admin"; fi
if [[ -z "${MAAS_ADMIN_EMAIL:-}" ]]; then <email address hidden>"; fi
if [[ -z "${MAAS_ADMIN_PASS:-}" ]]; then MAAS_ADMIN_PASS="admin"; fi
MAASVM_MGMTNET_IP=10.10.1.17
MAASVM_API_URL="http://${MAASVM_MGMTNET_IP}:5240/MAAS/api/2.0"
MAAS_ADMIN_APIKEY="$(maas-region apikey --username ${MAAS_ADMIN_USER})"
maas login "${MAAS_ADMIN_USER}" "${MAASVM_API_URL}" "${MAAS_ADMIN_APIKEY}"
maas "${MAAS_ADMIN_USER}" boot-resource delete 18
maas "${MAAS_ADMIN_USER}" boot-resource delete 19
source <(wget -O- http://stable.release.core-os.net/amd64-usr/current/version.txt)
wget -nv http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
bunzip2 coreos_production_image.bin.bz2
tar -czvf coreos_production_image.bin.tgz coreos_production_image.bin
file coreos_production_image.bin
set -x
maas "${MAAS_ADMIN_USER}" boot-resources create name=custom/coreos_stable_"${COREOS_BUILD}"_"${COREOS_BRANCH}"_"${COREOS_PATCH}" architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz
set +x
grep -r coreos /var/log/maas/

Console log:
=======================

root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_USER:-}" ]]; then MAAS_ADMIN_USER="admin"; fi
root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_EMAIL:-}" ]]; then <email address hidden>"; fi
root@maas:/home/vagrant# if [[ -z "${MAAS_ADMIN_PASS:-}" ]]; then MAAS_ADMIN_PASS="admin"; fi
root@maas:/home/vagrant# MAASVM_MGMTNET_IP=10.10.1.17
root@maas:/home/vagrant# MAASVM_API_URL="http://${MAASVM_MGMTNET_IP}:5240/MAAS/api/2.0"
root@maas:/home/vagrant# MAAS_ADMIN_APIKEY="$(maas-region apikey --username ${MAAS_ADMIN_USER})"
root@maas:/home/vagrant# maas login "${MAAS_ADMIN_USER}" "${MAASVM_API_URL}" "${MAAS_ADMIN_APIKEY}"

You are now logged in to the MAAS server at
http://10.10.1.17:5240/MAAS/api/2.0/ with the profile name 'admin'.

For help with the available commands, try:

  maas admin --help

root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resource delete 18
BootResource matching query does not exist.
root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resource delete 19
Success.
Machine-readable output follows:

root@maas:/home/vagrant# source <(wget -O- http://stable.release.core-os.net/amd64-usr/current/version.txt)
--2016-05-12 16:10:29-- http://stable.release.core-os.net/amd64-usr/current/version.txt
Resolving stable.release.core-os.net (stable.release.core-os.net)... 104.16.21.26, 104.16.20.26, 2400:cb00:2048:1::6810:141a, ...
Connecting to stable.release.core-os.net (stable.release.core-os.net)|104.16.21.26|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 161 [text/plain]
Saving to: 'STDOUT'

- 100%[========================================================================================================================================>] 161 --.-KB/s in 0s

2016-05-12 16:10:29 (28.5 MB/s) - written to stdout [161/161]

root@maas:/home/vagrant# wget -nv http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
2016-05-12 16:11:01 URL:http://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2 [232529993/232529993] -> "coreos_production_image.bin.bz2" [1]
root@maas:/home/vagrant# bunzip2 coreos_production_image.bin.bz2
root@maas:/home/vagrant# tar -czvf coreos_production_image.bin.tgz coreos_production_image.bin
coreos_production_image.bin
root@maas:/home/vagrant# file coreos_production_image.bin
coreos_production_image.bin: DOS/MBR boot sector
root@maas:/home/vagrant# set -x
root@maas:/home/vagrant# maas "${MAAS_ADMIN_USER}" boot-resources create name=custom/coreos_stable_"${COREOS_BUILD}"_"${COREOS_BRANCH}"_"${COREOS_PATCH}" architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz+ maas admin boot-resources create name=custom/coreos_stable_899_17_0 architecture=amd64/generic content@=coreos_production_image.bin.tgz filetype=ddtgz
root@maas:/home/vagrant# set +x
+ set +x
root@maas:/home/vagrant# grep -r coreos /var/log/maas/
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/rackd.log: builtins.OSError: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160512/root-dd. mirrors=[]
/var/log/maas/regiond.log:2016-05-12 14:29:39 [-] 127.0.0.1 - - [12/May/2016:14:29:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 14:29:45 [-] 127.0.0.1 - - [12/May/2016:14:29:45 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 14:49:39 [-] 127.0.0.1 - - [12/May/2016:14:49:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:09:39 [-] 127.0.0.1 - - [12/May/2016:15:09:39 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:29:38 [-] 127.0.0.1 - - [12/May/2016:15:29:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:29:59 [-] 127.0.0.1 - - [12/May/2016:15:29:58 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 15:49:39 [-] 127.0.0.1 - - [12/May/2016:15:49:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 16:09:39 [-] 127.0.0.1 - - [12/May/2016:16:09:38 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd HTTP/1.1" 404 1353 "-" "python-requests/2.9.1"
/var/log/maas/regiond.log:2016-05-12 16:19:04 [-] 127.0.0.1 - - [12/May/2016:16:19:03 +0000] "GET /MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160512/root-dd HTTP/1.1" 404 1354 "-" "python-requests/2.9.1"
/var/log/maas/maas.log:May 12 14:29:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 14:49:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:09:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:29:38 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 15:49:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
/var/log/maas/maas.log:May 12 16:09:39 maas maas.boot_image_download_service: [ERROR] Failed to download images: Unable to open http://localhost:5240/MAAS/images-stream/custom/amd64/generic/coreos_stable_899_17_0/20160511/root-dd. mirrors=[]
root@maas:/home/vagrant# date
Thu May 12 16:19:23 UTC 2016
root@maas:/home/vagrant#

Related branches

lp:~blake-rouse/maas/fix-1581130
Gavin Panella (community): Approve
Diff: 183 lines (+74/-9)
8 files modified
src/maasserver/forms.py (+14/-0)
src/maasserver/migrations/builtin/maasserver/0065_larger_osystem_and_distro_series.py (+27/-0)
src/maasserver/models/node.py (+2/-2)
src/maasserver/node_action.py (+2/-2)
src/maasserver/tests/test_forms_bootresource.py (+1/-1)
src/maasserver/tests/test_node_action.py (+1/-1)
src/maasserver/utils/osystems.py (+7/-1)
src/maasserver/utils/tests/test_osystems.py (+20/-2)
Revision history for this message
Mark Casey (mark-casey) wrote :
Download full text (12.0 KiB)

I turned in DEBUG and ran regiond in foreground and didn't get much more info. This output shows me (from another console) deleting a previous attempt and then retrying the boot-resources create. (the change in the image name from the initial report was my doing to see if the underscores or name length could be an issue)

I would really appreciate some next steps that I could undertake to help diagnose this, even if there is no immediate solution.

2016-05-16 16:52:01 [-] 10.10.1.17 - - [16/May/2016:16:52:01 +0000] "DELETE /MAAS/api/2.0/boot-resources/33/ HTTP/1.1" 204 - "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:08 [-] 127.0.0.1 - - [16/May/2016:16:52:08 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 102 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-05-16 16:52:32 [-] 10.10.1.17 - - [16/May/2016:16:52:32 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:32 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:33 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:33 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/2.0/boot-resources/34/upload/62/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 16:52:34 [-] 10.10.1.17 - - [16/May/2016:16:52:34 +0000] "PUT /MAAS/api/...

Revision history for this message
Mark Casey (mark-casey) wrote :
Download full text (9.4 KiB)

Same result with an ubuntu image copied out of image cache (uploaded as tgz instead of ddtgz... I think that was right...)

COMMANDS:
================
diff -u /var/lib/maas/boot-resources/cache/root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 /var/lib/maas/boot-resources/snapshot-20160513-184938/ubuntu/amd64/hwe-x/xenial/release/root-tgz
cp -a /var/lib/maas/boot-resources/cache/root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 .
mv root-tgz-78c4caf368edf45272fedf7c74b3c29d8a8c74b8e526710f09a1b59488784188 /tmp/ubuntu1604-tgz
maas admin boot-resources create -d name=custom/foo architecture=amd64/generic filetype=tgz content@=/tmp/ubuntu1604-tgz

CONSOLE:
=================
2016-05-16 19:21:08 [-] 127.0.0.1 - - [16/May/2016:19:21:08 +0000] "GET /MAAS/rpc/ HTTP/1.0" 200 103 "-" "provisioningserver.rpc.clusterservice.ClusterClientService"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:18 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:19 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:19 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:20 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP/1.1" 200 2 "-" "Python-httplib2/0.9.1 (gzip)"
2016-05-16 19:21:20 [-] 10.101.0.16 - - [16/May/2016:19:21:20 +0000] "PUT /MAAS/api/2.0/boot-resources/35/upload/63/ HTTP...

Read more...

Revision history for this message
Mark Casey (mark-casey) wrote :

Regarding the 404's from not being able to D/L the image after it is created. If I copy the failed '/MAAS/images-stream/custom/...' URL out of the logs and try to wget it I also get a 404.

If I use 'maas-region dbshell --installed' to remove "custom/" from the bootresource's name:

UPDATE "maasserver_bootresource" SET "name" = 'coreos' WHERE ("maasserver_bootresource"."name" = 'custom/coreos' AND "maasserver_bootresource"."architecture" = 'amd64/generic');

Then I can use the same URL that had just 404'ed to successfully manually D/L the image. Of course that breaks the web interface (and from what I can see on the half broken interface it hasn't actually gotten rid of the 'out-of-sync' message) so it doesn't really get me into a usable state.

So AFAICT seems to be a disconnect between image storage and image download regarding how the names of custom images are special.

Andres Rodriguez (andreserl)
Changed in maas:
status: New → Triaged
Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
importance: Undecided → Critical
assignee: nobody → Blake Rouse (blake-rouse)
milestone: none → 2.0.0
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.