Confusing error: Creating container: can't get info for image 'ubuntu-trusty'

[Jason Hobbs]

We hit a confusing error and it took over a week and a couple of juju devs getting involved to determine it was probably a temporary network outage.

There should be a better error message, or better yet, juju should keep trying for some period of time.

On a deploy of OpenStack in OIL, LXD containers on one machine failed to be created due to this error:

"Creating container: can't get info for image 'ubuntu-trusty'"

Trusty container setup on other machines in the same deployment worked fine.

This is with 2.0 beta 11 and 2.0 RC2.

Here is the juju status:

http://paste.ubuntu.com/20733843/

I will attach the logs from the machine and the logs from the controller node.

[Jason Hobbs (jason-hobbs)]

These are the logs from the machine where the error occurred.

[Anastasia (anastasia-macmood)]

@Jason

LXD requires seccomp which is not on trusty. You will have problems with LXD on trusty. see lp#1600311

[Jason Hobbs (jason-hobbs)]

bug 1600311 is powerpc specific. I spoke with Stéphane and he says on amd64 (which this system is) LXD should work fine on trusty, even with the trusty kernel.

[Anastasia (anastasia-macmood)]

The package still needs to be in trusty. It's in backports but not in updates.

We are not able to support LXD on trusty until this happens.

[Anastasia (anastasia-macmood)]

I'll create another bug for Juju to report this earlier.

[Anastasia (anastasia-macmood)]

Opened bug to warn users earlier: https://bugs.launchpad.net/juju-core/+bug/1607109

[Jason Hobbs (jason-hobbs)]

Anastasia,

The maas images we're using have LXD from backports in them. Is the lack
of support a technical thing or a policy thing?

Jason

On Thu, Jul 28, 2016 at 12:30 AM, Anastasia <<email address hidden>
> wrote:

> Opened bug to warn users earlier: https://bugs.launchpad.net/juju-
> core/+bug/1607109
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> Invalid
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

[Jason Hobbs (jason-hobbs)]

To be clear, we're using the daily images from
http://images.maas.io/ephemeral-v2/daily/ - which is the default for MAAS
2.0

On Thu, Jul 28, 2016 at 8:23 AM, Jason Hobbs <email address hidden>
wrote:

> Anastasia,
>
> The maas images we're using have LXD from backports in them. Is the lack
> of support a technical thing or a policy thing?
>
> Jason
>
> On Thu, Jul 28, 2016 at 12:30 AM, Anastasia <
> <email address hidden>> wrote:
>
>> Opened bug to warn users earlier: https://bugs.launchpad.net/juju-
>> core/+bug/1607109 <https://bugs.launchpad.net/juju-core/+bug/1607109>
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1605986
>>
>> Title:
>> Creating container: can't get info for image 'ubuntu-trusty'
>>
>> Status in juju-core:
>> Invalid
>>
>> Bug description:
>> On a deploy of OpenStack in OIL, LXD containers on one machine failed
>> to be created due to this error:
>>
>> "Creating container: can't get info for image 'ubuntu-trusty'"
>>
>> Trusty container setup on other machines in the same deployment worked
>> fine.
>>
>>
>> This is with 2.0 beta 11 and 2.0 RC2.
>>
>> Here is the juju status:
>>
>> http://paste.ubuntu.com/20733843/
>>
>> I will attach the logs from the machine and the logs from the
>> controller node.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>>
>
>

[Anastasia (anastasia-macmood)]

Jason,

Both LXD and juju2 are in backports in trusty, hence we do support it.

I have re-classified the bug accordingly. Thank you for your patience \o/

[Tim Penhey (thumper)]

The machines that juju starts do not have backports enabled. So when Juju starts a trusty machine, it cannot get LXD since it is only in backports.

If we do want to support this, Juju would have to gain the ability to add backports to the sources for the machines it starts.

[Jason Hobbs (jason-hobbs)]

Tim/Anastasia,

Backports are enabled in the trusty MAAS images juju is starting here.

Are you seeing otherwise?

On Fri, Jul 29, 2016 at 1:42 AM, Anastasia <email address hidden>
wrote:

> ** Changed in: juju-core
> Status: Triaged => Incomplete
>
> ** Changed in: juju-core
> Importance: Critical => Undecided
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> Incomplete
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

[Jason Hobbs (jason-hobbs)]

According to bug 1568895, an both daily and released MAAS images have backports enabled.

[Jason Hobbs (jason-hobbs)]

I verified this is the case by deploying a trusty instance in maas.
backports is enabled:

ubuntu@skookum:~$ apt-cache policy lxd
lxd:
  Installed: (none)
  Candidate: 2.0.3-0ubuntu2~ubuntu14.04.1
  Version table:
     2.0.3-0ubuntu2~ubuntu14.04.1 0
        100 http://archive.ubuntu.com//ubuntu/ trusty-backports/universe
amd64 Packages

On Fri, Jul 29, 2016 at 8:27 AM, Jason Hobbs <email address hidden>
wrote:

> According to bug 1568895, an both daily and released MAAS images have
> backports enabled.
>
> ** Changed in: juju-core
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> New
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

[Andrew McDermott (frobware)]

When this happens can you log into the host and capture the output from:

 $ ip route

I'm wondering whether the host machine has a route out. If not, I can understand that MAAS/host communication could work but fetching the image from the internet would understandably fail.

[Jason Hobbs (jason-hobbs)]

Andrew,

The cloud-init logs attached in first comment show that there was a route
out setup:

http://paste.ubuntu.com/21391121/

On Fri, Jul 29, 2016 at 12:17 PM, Andrew McDermott <
<email address hidden>> wrote:

> When this happens can you log into the host and capture the output from:
>
> $ ip route
>
> I'm wondering whether the host machine has a route out. If not, I can
> understand that MAAS/host communication could work but fetching the
> image from the internet would understandably fail.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> New
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

[Andrew McDermott (frobware)]

I just ran into this too, and I definitely don't have a route out:

ubuntu@maas20-node2:~$ sudo ip route
10.0.0.0/24 dev lxdbr0 proto kernel scope link src 10.0.0.1
10.12.20.0/24 dev br-ens5 proto kernel scope link src 10.12.20.7
10.12.20.0/24 dev br-ens6 proto kernel scope link src 10.12.20.9
10.12.20.0/24 dev br-bond0 proto kernel scope link src 10.12.20.6
10.100.0.0/20 dev br-ens5.100 proto kernel scope link src 10.100.0.3

ubuntu@maas20-node2:~$ sudo lxc launch ubuntu:trusty c1
Creating c1
error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443

ubuntu@maas20-node2:~$ wget kernel.org
--2016-07-29 13:24:50-- http://kernel.org/
Resolving kernel.org (kernel.org)... 2620:3:c000:a:0:1991:8:25, 2001:4f8:1:10:0:1991:8:25, 198.145.20.140, ...
Connecting to kernel.org (kernel.org)|2620:3:c000:a:0:1991:8:25|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|2001:4f8:1:10:0:1991:8:25|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|198.145.20.140|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|149.20.4.69|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|199.204.44.194|:80... failed: Network is unreachable.

Yet I do have a gateway specified in ENI:

ubuntu@maas20-node2:~$ grep -B 2 -A 5 gateway /etc/network/interfaces
auto br-bond0
iface br-bond0 inet static
    gateway 10.12.20.1
    address 10.12.20.6/24
    hwaddress 52:54:00:44:4f:42
    bridge_ports bond0
    dns-nameservers 10.12.20.2

And the device is UP:

ubuntu@maas20-node2:~$ ip a show br-bond0
9: br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 52:54:00:44:4f:42 brd ff:ff:ff:ff:ff:ff
    inet 10.12.20.6/24 brd 10.12.20.255 scope global br-bond0
       valid_lft forever preferred_lft forever
    inet 10.12.20.106/24 brd 10.12.20.255 scope global secondary br-bond0:1
       valid_lft forever preferred_lft forever
    inet 10.12.20.107/24 brd 10.12.20.255 scope global secondary br-bond0:2
       valid_lft forever preferred_lft forever
    inet 10.12.20.108/24 brd 10.12.20.255 scope global secondary br-bond0:3
       valid_lft forever preferred_lft forever
    inet 10.12.20.109/24 brd 10.12.20.255 scope global secondary br-bond0:4
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe44:4f42/64 scope link
       valid_lft forever preferred_lft forever

[Andrew McDermott (frobware)]

https://github.com/juju/juju/pull/5791 appears to be the culprit, at least for trusty.

[Dimiter Naydenov (dimitern)]

https://github.com/juju/juju/pull/5791 can't be the culprit as it landed in beta12 (to fix bug 1584616), whereas this bug is for beta11.

Also, looking at the logs provided it's unclear yet what's the cause. Will keep digging for the cause.

[Dimiter Naydenov (dimitern)]

I had a chat with jhobbs about this, and it looks like the issue is due to intermittent network connectivity issue on that machine 2. It happens occasionally on other machines. This is because Juju does not cache the lxd images on the controller, like with juju agents(tools), so the lxd cloud image is downloaded and cached for each provisioned host machine (before starting the first container on it). I'll find and post a link to the bug about that.

As agreed on IRC, marking this as Incomplete, to be reopened if it happens again (with beta13 the issue seems to be a lot less frequent).

[Richard Harding (rharding)]

Agree at the least we should improve the error message output until we land the lxd image caching work.

[Anastasia (anastasia-macmood)]

This bug has become about usability, early error reporting and clearer message.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.