openstack provider: juju does not give any error if trusty instance forever in pending because of missing image metadata

[Saverio Proto]

I am using Juju 2.0.3-xenial-amd64 against an Openstack cloud.

The controller is Xenial, so to start the controller I just did:

juju metadata generate-image -d ~/engines -i bc49e459-453d-48b3-a6e4-e572dc778b91 -s xenial -r ZH -u https://keystone.cloud.switch.ch:5000/v3
juju bootstrap --metadata-source /home/ubuntu/engines SWITCHengines saveriotest

Now if I need to deploy something that needs a Trusty machine:
juju deploy cs:bundle/mediawiki-single

everything is pending forever, and I dont see any log anywhere. I guess Juju should report somewhere there there is an error and it cannot go forward.

I started everything again from scratch and this time I also added metadata for a trusty image:

juju metadata generate-image -d ~/engines -i 66a5ff70-0971-453b-baea-019096f29887 -s trusty -r ZH -u https://keystone.cloud.switch.ch:5000/v3

Now I see the trusty VMs starting on openstack when I deploy the mediawiki charm.

Am I missing something ? There should not be some error message somewhere to troubleshoot ?

[Anastasia (anastasia-macmood)]

@Saverio Proto.
Please clarify what version of Juju you are using :D

[Saverio Proto (zioproto)]

Juju version 2.0.3-xenial-amd64

[Anastasia (anastasia-macmood)]

We track Juju 2.x bugs in "juju" project. I'll re-target :)

I guess that when you say "..if I need to deploy something that needs a Trusty machine...everything is pending forever..." what you are actually observing is that command "juju deploy" returns fine but the provisioning of machines is "pending".

Provisioning of machines is happening asynchronously by a worker. So there would not be anything useful returned from "deploy" command itself.

We have improved "juju status" values and changes in Juju 2.1 and your experience should be better with newer version. I will double check whether we specifically surface image related error in status.

Meanwhile the best practice to diagnose why a machine is in "pending" is by examining its log. Controller and model logs would have information about whether the image was found on or not.

[Anastasia (anastasia-macmood)]

As a follow-up, after double-checking :)

We do surface provisioning errors in juju status when status is requested in yaml format: tabular format has limited space because of the more rigid structure and alignments.

If you were to run 'juju status --format=yaml', you would see something along the lines of:

...
$ juju status --format=yaml
model:
  name: default
  controller: amcinternal
  cloud: oscloud1
  region: RegionOne
  version: 2.1.0.1
machines:
  "0":
    juju-status:
      current: down
      message: agent is not communicating with the server
      since: 20 Feb 2017 07:49:12Z
    instance-id: pending
    machine-status:
      current: provisioning error
      message: no "trusty" images in RegionOne with arches [amd64]
      since: 20 Feb 2017 07:49:12Z
    series: trusty
  "1":
    juju-status:
      current: down
      message: agent is not communicating with the server
      since: 20 Feb 2017 07:49:51Z
    instance-id: pending
    machine-status:
      current: provisioning error
      message: no "trusty" images in RegionOne with arches [amd64]
      since: 20 Feb 2017 07:49:51Z
    series: trusty
...

[Saverio Proto (zioproto)]

Hello Anastasia,
I never see the message: no "trusty" images ...

I did wait several minutes now, after how long I should get out of the pending status ?

thank you

ubuntu@xenial:~$ date
Mon Feb 20 10:11:54 CET 2017
ubuntu@xenial:~$ juju status --format=yaml
model:
  name: default
  controller: saveriotest
  cloud: SWITCHengines
  region: ZH
  version: 2.0.3.1
machines:
  "0":
    juju-status:
      current: down
      message: agent is not communicating with the server
      since: 20 Feb 2017 10:03:09+01:00
    instance-id: pending
    machine-status:
      current: pending
      since: 20 Feb 2017 10:02:28+01:00
    series: trusty
  "1":
    juju-status:
      current: down
      message: agent is not communicating with the server
      since: 20 Feb 2017 10:03:44+01:00
    instance-id: pending
    machine-status:
      current: pending
      since: 20 Feb 2017 10:02:30+01:00
    series: trusty
applications:
  mediawiki:
    charm: cs:trusty/mediawiki-3
    series: trusty
    os: ubuntu
    charm-origin: jujucharms
    charm-name: mediawiki
    charm-rev: 3
    exposed: false
    application-status:
      current: waiting
      message: waiting for machine
      since: 20 Feb 2017 10:02:27+01:00
    relations:
      db:
      - mysql
    units:
      mediawiki/0:
        workload-status:
          current: waiting
          message: waiting for machine
          since: 20 Feb 2017 10:02:27+01:00
        juju-status:
          current: allocating
          since: 20 Feb 2017 10:02:27+01:00
        machine: "0"
  mysql:
    charm: cs:trusty/mysql-29
    series: trusty
    os: ubuntu
    charm-origin: jujucharms
    charm-name: mysql
    charm-rev: 29
    exposed: false
    application-status:
      current: waiting
      message: waiting for machine
      since: 20 Feb 2017 10:02:29+01:00
    relations:
      cluster:
      - mysql
      db:
      - mediawiki
    units:
      mysql/0:
        workload-status:
          current: waiting
          message: waiting for machine
          since: 20 Feb 2017 10:02:29+01:00
        juju-status:
          current: allocating
          since: 20 Feb 2017 10:02:29+01:00
        machine: "1"
ubuntu@xenial:~$

[Anastasia (anastasia-macmood)]

@Saverio Proto (zioproto),

Responsiveness depends on your network and setup. Mine took couple of minutes over company's VPN.

My version is also more recent then yours - I have tried with Juju 2.1 tip which will be 2.1.0 GA at some stage. Can you try with 2.1-rc2?

[Saverio Proto (zioproto)]

Hello,

I installed from ppa:juju/stable as suggested in the GettingStarted guide:
https://jujucharms.com/docs/stable/getting-started

I am just testing in a dev environment, so no stress about production bugs at the moment.

I guess if you see the bug fixed in 2.1 you are right. I am new to juju I have no idea what are the patches that you guys are backporting to the current stable release.

When is 2.1 going to be released ? You would suggest me to use a different ppa for my environment? Is 2.0 something already very old ?

If this is already fixed in 2.1 you can go ahead and close the bug for me. If core developers want to backport this fix to 2.0 stable release I am available for testing.

Thank you !

[Anastasia (anastasia-macmood)]

ppa:juju/stable does indeed contain the production quality Juju 2.0 series which is currently at 2.0.3. We are not planning to release any further 2.0.x releases as we are full steam ahead to deliver 2.1.0 (GA) imminently soon.

You can get Juju 2.1 (2.1-rc2 at the moment) from the juju devel ppa:

   sudo add-apt-repository ppa:juju/devel

   sudo apt-get update

   sudo apt-get install juju