Cannot use Ubuntu Pro with Juju on AWS

[Casey Marshall]

Tried to configure Juju to launch Ubuntu Pro VMs in AWS by using the new `juju metadata add-image` feature in Juju 2.8. I bootstrapped a controller with Juju 2.8.1 in AWS us-west-2 and tried configuring Juju to use Pro with:

 juju metadata add-image --series trusty ami-006bf064149e7c7e5
 juju metadata add-image --series xenial ami-0261d1c600dfdd37d
 juju metadata add-image --series bionic ami-09109b16d2d9d5779
 juju metadata add-image --series focal ami-037fd1e90f947c0aa

These AMI ids were obtained by subscribing to the corresponding Ubuntu Pro listing in AWS Marketplace and copying from the AMI shown on the "Configure" page.

Then did `juju deploy ubuntu` and observed status. Juju seems to be unable to find the Pro AMI. I'm using cloud credentials for the same account that I used to subscribe to the Pro marketplace offers so I should have access.

Every 2.0s: juju status slate: Wed Jul 29 12:36:53 2020

Model Controller Cloud/Region Version SLA Timestamp
test aws-us-west-2 aws/us-west-2 2.8.1 unsupported 12:36:54-05:00

App Version Status Scale Charm Store Rev OS Notes
ubuntu waiting 0/1 ubuntu jujucharms 15 ubuntu

Unit Workload Agent Machine Public address Ports Message
ubuntu/0 waiting allocating 0 waiting for machine

Machine State DNS Inst id Series AZ Message
0 pending pending bionic failed to start machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)), retrying in 10s (9 more attempts)

I also tried copying Pro to my "Service Catalog" in the AWS Console and using the AMI from the generated template, same result.

See also https://discourse.juju.is/t/juju-launching-ubuntu-pro-instances-in-aws-azure/3389/2

[Casey Marshall (cmars)]

At timclick's request in that discourse thread above, I gave this another attempt. This time, it worked. I bootstrapped in a different region, us-east-1, where I usually test pro -- and it got a little further!

The instance definitely looks like Ubuntu Pro, there's a marketplace product code!

ubuntu@ip-172-31-46-56:~$ curl http://169.254.169.254/latest/dynamic/instance-identity/document
{
  "accountId" : "937910461948",
  "architecture" : "x86_64",
  "availabilityZone" : "us-east-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : [ "ex9l8qw4x21426rsfg7am8gym" ],
  "imageId" : "ami-09109b16d2d9d5779",
  "instanceId" : "i-08261909fac3393de",
  "instanceType" : "t3a.micro",
  "kernelId" : null,
  "pendingTime" : "2020-07-29T21:43:25Z",
  "privateIp" : "172.31.46.56",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2017-09-30"
}

However, ua client doesn't work on this instance and auto-attach fails (which is bad, because I'm paying for Pro...). The error in /var/lib/ubuntu-advantage.log shows:

2020-07-29 21:47:53,331 - util.py:(373) [DEBUG]: Reading file: /sys/hypervisor/uuid
2020-07-29 21:47:53,331 - util.py:(373) [DEBUG]: Reading file: /sys/class/dmi/id/product_uuid
2020-07-29 21:47:53,332 - util.py:(373) [DEBUG]: Reading file: /sys/class/dmi/id/product_serial
2020-07-29 21:47:53,332 - util.py:(373) [DEBUG]: Reading file: /etc/os-release
2020-07-29 21:47:54,916 - cli.py:(637) [ERROR]: Unhandled exception, please file a bug
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 614, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 660, in main
    return args.action(args, cfg)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 52, in new_f
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 425, in action_auto_attach
    token = _get_contract_token_from_cloud_identity(cfg)
  File "/usr/lib/python3/dist-packages/uaclient/cli.py", line 396, in _get_contract_token_from_cloud_identity
    current_iid = identity.get_instance_id()
  File "/usr/lib/python3/dist-packages/uaclient/clouds/identity.py", line 31, in get_instance_id
    out, _err = util.subp(["cloud-init", "query", "instance_id"])
  File "/usr/lib/python3/dist-packages/uaclient/util.py", line 518, in subp
    out, err = _subp(args, rcs, capture, timeout)
  File "/usr/lib/python3/dist-packages/uaclient/util.py", line 478, in _subp
    stderr=err.decode("utf-8"),
uaclient.util.ProcessExecutionError: Failed running command 'cloud-init query instance_id' [exit(1)]. Message: Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 11, in <module>
    load_entry_point('cloud-init==20.2', 'console_scripts', 'cloud-init')()
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main
    get_uptime=True, func=functor, args=(name, args))
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2620, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/query.py", line 123, in handle_args
    instance_data['user...

[Casey Marshall (cmars)]

Output from `juju debug-log` in us-west-2 again (where I got the AMI not found error the first time):

c@slate:~$ juju debug-log
controller-0: 17:14:08 INFO juju.worker.apicaller [a805df] "machine-0" successfully connected to "localhost:17070"
controller-0: 17:14:13 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 17:14:13 INFO juju.worker.logger logger worker started
controller-0: 17:14:13 INFO juju.worker.machineundertaker setting up machine undertaker
controller-0: 17:14:13 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for default (a805df8f-11c7-4d45-85ea-69a2024b640d)
controller-0: 17:14:13 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for default (a805df8f-11c7-4d45-85ea-69a2024b640d)
controller-0: 17:14:13 INFO juju.worker.provisioner provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
controller-0: 17:14:52 INFO juju.worker.provisioner found machine pending provisioning id:0, details:0
controller-0: 17:14:52 INFO juju.worker.provisioner provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
controller-0: 17:14:54 INFO juju.worker.provisioner trying machine 0 StartInstance in availability zone us-west-2a
controller-0: 17:14:58 WARNING juju.worker.provisioner machine 0 failed to start in availability zone us-west-2a: cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
controller-0: 17:14:58 WARNING juju.worker.provisioner failed to start machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)), retrying in 10s (10 more attempts)
controller-0: 17:15:08 INFO juju.worker.provisioner trying machine 0 StartInstance in availability zone us-west-2b
controller-0: 17:15:09 WARNING juju.worker.provisioner machine 0 failed to start in availability zone us-west-2b: cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
controller-0: 17:15:09 WARNING juju.worker.provisioner failed to start machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)), retrying in 10s (9 more attempts)
controller-0: 17:15:19 INFO juju.worker.provisioner trying machine 0 StartInstance in availability zone us-west-2c
controller-0: 17:15:20 WARNING juju.worker.provisioner machine 0 failed to start in availability zone us-west-2c: cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
controller-0: 17:15:20 WARNING juju.worker.provisioner failed to start machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)), retrying in 10s (8 more attempts)
controller-0: 17:15:30 INFO juju.worker.provisioner trying machine 0 StartInstance in availability zone us-west-2d
controller-0: 17:15:30 WARNING juju.worker.provisioner machine 0 failed to start in availability zone us-west-2d: cannot run instances: The image id '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
controller-0...

[Tim Penhey (thumper)]

Are you sure it is the same AMI in the other region? It is possible that
AWS have different AMIs for ubuntu pro in the different regions.

It seems strange that you can find ubuntu pro in one region but not the
other. The only difference here really is the remote side.

On Thu, Jul 30, 2020 at 10:30 AM Casey Marshall <email address hidden>
wrote:

> Output from `juju debug-log` in us-west-2 again (where I got the AMI not
> found error the first time):
>
>
> c@slate:~$ juju debug-log
> controller-0: 17:14:08 INFO juju.worker.apicaller [a805df] "machine-0"
> successfully connected to "localhost:17070"
> controller-0: 17:14:13 INFO juju.worker.logforwarder config change - log
> forwarding not enabled
> controller-0: 17:14:13 INFO juju.worker.logger logger worker started
> controller-0: 17:14:13 INFO juju.worker.machineundertaker setting up
> machine undertaker
> controller-0: 17:14:13 INFO juju.worker.pruner.action status history
> config: max age: 336h0m0s, max collection size 5120M for default
> (a805df8f-11c7-4d45-85ea-69a2024b640d)
> controller-0: 17:14:13 INFO juju.worker.pruner.statushistory status
> history config: max age: 336h0m0s, max collection size 5120M for default
> (a805df8f-11c7-4d45-85ea-69a2024b640d)
> controller-0: 17:14:13 INFO juju.worker.provisioner
> provisioner-harvest-mode is set to destroyed; unknown instances not stopped
> []
> controller-0: 17:14:52 INFO juju.worker.provisioner found machine pending
> provisioning id:0, details:0
> controller-0: 17:14:52 INFO juju.worker.provisioner
> provisioner-harvest-mode is set to destroyed; unknown instances not stopped
> []
> controller-0: 17:14:54 INFO juju.worker.provisioner trying machine 0
> StartInstance in availability zone us-west-2a
> controller-0: 17:14:58 WARNING juju.worker.provisioner machine 0 failed to
> start in availability zone us-west-2a: cannot run instances: The image id
> '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
> controller-0: 17:14:58 WARNING juju.worker.provisioner failed to start
> machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]'
> does not exist (InvalidAMIID.NotFound)), retrying in 10s (10 more attempts)
> controller-0: 17:15:08 INFO juju.worker.provisioner trying machine 0
> StartInstance in availability zone us-west-2b
> controller-0: 17:15:09 WARNING juju.worker.provisioner machine 0 failed to
> start in availability zone us-west-2b: cannot run instances: The image id
> '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
> controller-0: 17:15:09 WARNING juju.worker.provisioner failed to start
> machine 0 (cannot run instances: The image id '[ami-09109b16d2d9d5779]'
> does not exist (InvalidAMIID.NotFound)), retrying in 10s (9 more attempts)
> controller-0: 17:15:19 INFO juju.worker.provisioner trying machine 0
> StartInstance in availability zone us-west-2c
> controller-0: 17:15:20 WARNING juju.worker.provisioner machine 0 failed to
> start in availability zone us-west-2c: cannot run instances: The image id
> '[ami-09109b16d2d9d5779]' does not exist (InvalidAMIID.NotFound)
> controller-0: 17:15:20 WARNING juju.worker.provisioner failed to start
> machine 0 (cannot run instances: The ...

[Casey Marshall (cmars)]

Thanks for pointing that out, AMIs are region specific, my mistake.

Once we got the region/AMI thing sorted out, we found a cloud-init bug -- but no real issue with Juju. Nice work on the metadata add-image feature!