[2.1.1] Controller does not look at agent_stream in model-config - it uses agent_stream that it booted with

[Larry Michel]

I have a controller that was booted with agent_stream=devel. I've since modified the model-defaults to set it to agent_stream=released. All future model should then have agent-stream set to released and they do. However, the controller continues to look in devel when deploying arm maas servers during a bundle deployment and therefore it's not able to find the arm64 agents to boot phanpy.

These are what the agent settings are set to below. To recreate, boostrap a controller with agent-stream set to devel. Then, modify the model-defaults so that agent-stream is released. Then add a new model and finally deploy a bundle with an arm server as a machine. It will not be possible to deploy the arm server.

$ juju model-defaults
Attribute Default Controller
agent-metadata-url "" -
agent-stream released -
apt-ftp-proxy "" -
apt-http-proxy "" -
apt-https-proxy "" -
apt-mirror "" -
automatically-retry-hooks true -
default-series xenial -
development false -
disable-network-management false -
enable-os-refresh-update true -
enable-os-upgrade true -
firewall-mode instance -
ftp-proxy "" -
http-proxy "" -
https-proxy "" -
ignore-machine-addresses false -
image-metadata-url "" -
image-stream released -
logforward-enabled false -
logging-config "" -
net-bond-reconfigure-delay 17 -
no-proxy "" -
provisioner-harvest-mode destroyed -
proxy-ssh false -
resource-tags "" -
ssl-hostname-verification true -
test-mode false -
transmit-vendor-metrics true -

$ juju model-config
Attribute From Value
agent-metadata-url default ""
agent-stream default released
agent-version model 2.1.1.1
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
automatically-retry-hooks default true
default-series default xenial
development default false
disable-network-management default false
enable-os-refresh-update default true
enable-os-upgrade default true
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream default released
logforward-enabled default false
logging-config model <root>=DEBUG;unit=DEBUG
net-bond-reconfigure-delay default 17
no-proxy default ""
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
ssl-hostname-verification default true
storage-default-block-source model maas
test-mode default false
transmit-vendor-metrics default true

9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.state state.go:378 creating cloud image metadata storage
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.state state.go:384 started state for model-9b7d8641-5abc-4a99-8e58-f5b40732d7c7 successfully
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.state.workers dumb.go:164 stopped state workers without error
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.state open.go:603 closed state without error
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.provider.maas environprovider.go:51 opening model "debug-phanpy".
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.environs.tools tools.go:101 finding agent binaries in stream "devel"
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.environs.tools tools.go:105 reading agent binaries with major version -1
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.environs.tools tools.go:111 filtering agent binaries by version: 2.1.1.1
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.environs.tools tools.go:114 filtering agent binaries by series: xenial
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 INFO juju.environs.tools tools.go:117 filtering agent binaries by architecture: arm64
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.environs.tools urls.go:109 trying datasource "keystone catalog"
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.apiserver request_notifier.go:115 <- [96] machine-0 {"request-id":474,"type":"Singular","version":1,"request":"Claim","params":"'params redacted'"}
9b7d8641-5abc-4a99-8e58-f5b40732d7c7: machine-0 2017-03-14 20:44:14 DEBUG juju.worker.lease manager.go:217 waking to check leases at 2017-03-14 20:45:14.693850325 +0000 UTC

$ juju status
Model Controller Cloud/Region Version
debug-phanpy maas maas 2.1.1.1

App Version Status Scale Charm Store Rev OS Notes
ubuntu waiting 0/1 ubuntu jujucharms 10 ubuntu

Unit Workload Agent Machine Public address Ports Message
ubuntu/0 waiting allocating 0 waiting for machine

Machine State DNS Inst id Series AZ
0 down pending xenial

jenkins@juju-a6a70b-integration-juju2-1:~$

These are the versions:

jenkins@juju-a6a70b-integration-juju2-1:~$ juju --version
2.1.1-xenial-amd64
jenkins@juju-a6a70b-integration-juju2-1:~$ dpkg -l|grep juju
ii juju 1:2.1.1-0ubuntu1~16.04.1~juju1 all next generation service orchestration system
ii juju-2.0 1:2.1.1-0ubuntu1~16.04.1~juju1 amd64 Juju is devops d

[Aaron Bentley (abentley)]

The version number 2.1.1.1 suggests that the devel streams were not used for the original bootstrap. x.x.x.1 numbers are created when juju uploads the agent itself.

To avoid conflating issues, could you try to reproduce using the actual devel streams?

[Larry Michel (lmic)]

@abentley I have previously confirmed through test and speaking with Curtis and Nicholas that the devel streams don't have the arm agents.

[Aaron Bentley (abentley)]

The devel streams don't contain any agents for 2.1.1.

They do contain arm agents (among others) for devel releases:
$ sstream-query http://streams.canonical.com/juju/tools/streams/v1/index2.json content_id='com.ubuntu.juju:devel:tools' -o"%(item_name)s"|grep ^2.1.*arm |sort
2.1-beta1-trusty-arm64
2.1-beta1-xenial-arm64
2.1-beta1-yakkety-arm64
2.1-beta1-zesty-arm64
2.1-beta2-trusty-arm64
2.1-beta2-xenial-arm64
2.1-beta2-yakkety-arm64
2.1-beta2-zesty-arm64
2.1-beta3-trusty-arm64
2.1-beta3-xenial-arm64
2.1-beta3-yakkety-arm64
2.1-beta3-zesty-arm64
2.1-beta4-trusty-arm64
2.1-beta4-xenial-arm64
2.1-beta4-yakkety-arm64
2.1-beta4-zesty-arm64
2.1-beta5-precise-arm64
2.1-beta5-trusty-arm64
2.1-beta5-xenial-arm64
2.1-beta5-yakkety-arm64
2.1-beta5-zesty-arm64
2.1-rc1-precise-arm64
2.1-rc1-trusty-arm64
2.1-rc1-xenial-arm64
2.1-rc1-yakkety-arm64
2.1-rc1-zesty-arm64
2.1-rc2-precise-arm64
2.1-rc2-trusty-arm64
2.1-rc2-xenial-arm64
2.1-rc2-yakkety-arm64
2.1-rc2-zesty-arm64

[Larry Michel (lmic)]

@Aaron yes, 2.1.1 is released. The location of the agents is not what this bug covers. The bug is that the controller is not seeing or honoring the model setting for agent-stream. It appears to be permanently set to what it was when the controller was bootstrapped. I had to destroy the controller and re-bootstrap to get this to work.

[Anastasia (anastasia-macmood)]

This is interesting because it will put Juju controller in an unknown state - I do not believe we have encountered situations where models would be created with different agent streams.

You will end up with some models using devel agents, released agents and all other combinations in between.

I think we intentionally use agent stream from controller to ensure homogeneity. But I will bring it up with the team.

[Nicholas Skaggs (nskaggs)]

Just commenting that bug 1674755 is very similar. I wonder what other settings may be "fixed" at bootstrap.

[Ian Booth (wallyworld)]

Referring to the original bug description, when adding a model and deploying machines to that model, the running Juju agent version influences the stream used. If the running Juju version is a development version, (which it is in this case going by the version number) then I think the stream used for subsequent tools selection will be "develop", regardless of what any model agent stream is set to. Not saying this is correct but it explains the observed behaviour.

[Ryan Beisner (1chb1n)]

I've just noticed that I'm getting 'released' images for my model instances, despite setting image-stream to daily. The controller instances, however, do use a daily image as expected. This is Juju 2.1.2 + OpenStack Provider against an Ocata undercloud.

#### On a user's model, the instance is based on the *release* image, when the *daily* image was expected.

jenkins@juju-10f68a-osci-13:~$ openstack server show f487418d-0fe8-46ec-a8e1-92ff859c9051 | grep image
| image | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20170330-disk1.img (8b4dc7a6-11ad-44a0-86ca-9844d9763704) |

jenkins@juju-10f68a-osci-13:~$ openstack server show 5d799d64-cb31-475a-ac78-59fff602e0f3 | grep image
| image | ubuntu-released/ubuntu-xenial-16.04-amd64-server-20170330-disk1.img (8b4dc7a6-11ad-44a0-86ca-9844d9763704)

jenkins@juju-10f68a-osci-13:~$ juju switch auto-osci-sv08
auto-osci-sv08:admin/controller (no change)

jenkins@juju-10f68a-osci-13:~$ juju model-config
Attribute From Value
agent-metadata-url default ""
agent-stream controller proposed
agent-version model 2.1.2
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
automatically-retry-hooks controller false
default-series controller xenial
development default false
disable-network-management default false
enable-os-refresh-update controller false
enable-os-upgrade controller false
external-network default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream controller daily
logforward-enabled default false
logging-config model <root>=WARNING;unit=DEBUG
net-bond-reconfigure-delay default 17
network model 86125222-dd54-4df2-9421-235456fab193
no-proxy default ""
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
ssl-hostname-verification default true
storage-default-block-source model cinder
test-mode controller true
transmit-vendor-metrics controller false
use-default-secgroup controller true
use-floating-ip default false

.

#### On the controller model, the instance is based on the *daily* image as expected.

jenkins@juju-10f68a-osci-13:~$ juju switch controller
auto-osci-sv08:admin/auto-osci-sv08 -> auto-osci-sv08:admin/controller

jenkins@juju-10f68a-osci-13:~$ juju machines
Machine State DNS Inst id Series AZ
0 started 172.17.108.3 e872d229-1f50-4d2a-9263-b8ade...

[Ryan Beisner (1chb1n)]

I tried again with agent-stream: released and image-stream: daily (with Juju 2.1.2 from the stable ppa) on a fresh controller and observed the same:

Attribute From Value
agent-metadata-url default ""
agent-stream controller released
agent-version model 2.1.2
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror default ""
automatically-retry-hooks controller false
default-series controller xenial
development default false
disable-network-management default false
enable-os-refresh-update controller false
enable-os-upgrade controller false
external-network default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url default ""
image-stream controller daily
logforward-enabled default false
logging-config model <root>=WARNING;unit=DEBUG
net-bond-reconfigure-delay default 17
network model 86125222-dd54-4df2-9421-235456fab193
no-proxy default ""
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
ssl-hostname-verification default true
storage-default-block-source model cinder
test-mode controller true
transmit-vendor-metrics controller false
use-default-secgroup controller true
use-floating-ip default false

Model Controller Cloud/Region Version
auto-osci-sv08 auto-osci-sv08 serverstack/serverstack 2.1.2

App Version Status Scale Charm Store Rev OS Notes
ubuntu waiting 0/4 ubuntu jujucharms 10 ubuntu

Unit Workload Agent Machine Public address Ports Message
ubuntu/0 waiting allocating 0 172.17.108.10 waiting for machine
ubuntu/1 waiting allocating 1 172.17.108.12 waiting for machine
ubuntu/2 waiting allocating 2 172.17.108.14 waiting for machine
ubuntu/3 waiting allocating 3 172.17.108.3 waiting for machine

Machine State DNS Inst id Series AZ
0 pending 172.17.108.10 a2e6d361-e72f-4b85-aa2d-6383ce3ab8a0 xenial nova
1 pending 172.17.108.12 9614d394-58f7-49a7-9578-1267cc7aa902 xenial nova
2 pending 172.17.108.14 cf6c3952-1f71-4e6b-991e-295ea5703931 xenial nova
3 pending 172.17.108.3 4072c647-2c7b-4eba-b085-3d507662cd24 xenial nova

(clients) jenkins@juju-10f68a-osci-13:~/tools/foo/charm-test-infra$ openstack server list | grep auto-osci-sv08
| 4072c647-2c7b-4eba-b085-3d507662cd24 | juju-902164-auto-osci-sv08-3 | ACTIVE | auto-osci_...

[Larry Michel (lmic)]

@Ryan, in our case, the workaround I used for changing agent_stream was to bootstrap a new controller with the right settings. That seemed to be the only way to make the switch.

[Nicholas Skaggs (nskaggs)]

I believe this is fixed now in 2.3-rc2, with the other upgrade changes.