Confusing error: Creating container: can't get info for image 'ubuntu-trusty'

Bug #1605986 reported by Jason Hobbs on 2016-07-24
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
juju
Medium
Unassigned

Bug Description

We hit a confusing error and it took over a week and a couple of juju devs getting involved to determine it was probably a temporary network outage.

There should be a better error message, or better yet, juju should keep trying for some period of time.

On a deploy of OpenStack in OIL, LXD containers on one machine failed to be created due to this error:

"Creating container: can't get info for image 'ubuntu-trusty'"

Trusty container setup on other machines in the same deployment worked fine.

This is with 2.0 beta 11 and 2.0 RC2.

Here is the juju status:

http://paste.ubuntu.com/20733843/

I will attach the logs from the machine and the logs from the controller node.

Jason Hobbs (jason-hobbs) wrote :
Jason Hobbs (jason-hobbs) wrote :

These are the logs from the machine where the error occurred.

description: updated
Anastasia (anastasia-macmood) wrote :

@Jason

LXD requires seccomp which is not on trusty. You will have problems with LXD on trusty. see lp#1600311

Jason Hobbs (jason-hobbs) wrote :

bug 1600311 is powerpc specific. I spoke with Stéphane and he says on amd64 (which this system is) LXD should work fine on trusty, even with the trusty kernel.

Changed in juju-core:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 2.0-beta14
Anastasia (anastasia-macmood) wrote :

The package still needs to be in trusty. It's in backports but not in updates.

We are not able to support LXD on trusty until this happens.

Changed in juju-core:
status: Triaged → Invalid
Anastasia (anastasia-macmood) wrote :

I'll create another bug for Juju to report this earlier.

Anastasia (anastasia-macmood) wrote :

Opened bug to warn users earlier: https://bugs.launchpad.net/juju-core/+bug/1607109

Anastasia,

The maas images we're using have LXD from backports in them. Is the lack
of support a technical thing or a policy thing?

Jason

On Thu, Jul 28, 2016 at 12:30 AM, Anastasia <<email address hidden>
> wrote:

> Opened bug to warn users earlier: https://bugs.launchpad.net/juju-
> core/+bug/1607109
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> Invalid
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

Jason Hobbs (jason-hobbs) wrote :

To be clear, we're using the daily images from
http://images.maas.io/ephemeral-v2/daily/ - which is the default for MAAS
2.0

On Thu, Jul 28, 2016 at 8:23 AM, Jason Hobbs <email address hidden>
wrote:

> Anastasia,
>
> The maas images we're using have LXD from backports in them. Is the lack
> of support a technical thing or a policy thing?
>
> Jason
>
> On Thu, Jul 28, 2016 at 12:30 AM, Anastasia <
> <email address hidden>> wrote:
>
>> Opened bug to warn users earlier: https://bugs.launchpad.net/juju-
>> core/+bug/1607109 <https://bugs.launchpad.net/juju-core/+bug/1607109>
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1605986
>>
>> Title:
>> Creating container: can't get info for image 'ubuntu-trusty'
>>
>> Status in juju-core:
>> Invalid
>>
>> Bug description:
>> On a deploy of OpenStack in OIL, LXD containers on one machine failed
>> to be created due to this error:
>>
>> "Creating container: can't get info for image 'ubuntu-trusty'"
>>
>> Trusty container setup on other machines in the same deployment worked
>> fine.
>>
>>
>> This is with 2.0 beta 11 and 2.0 RC2.
>>
>> Here is the juju status:
>>
>> http://paste.ubuntu.com/20733843/
>>
>> I will attach the logs from the machine and the logs from the
>> controller node.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>>
>
>

Jason,

Both LXD and juju2 are in backports in trusty, hence we do support it.

I have re-classified the bug accordingly. Thank you for your patience \o/

Changed in juju-core:
status: Invalid → Triaged
Tim Penhey (thumper) wrote :

The machines that juju starts do not have backports enabled. So when Juju starts a trusty machine, it cannot get LXD since it is only in backports.

If we do want to support this, Juju would have to gain the ability to add backports to the sources for the machines it starts.

Changed in juju-core:
status: Triaged → Incomplete
importance: Critical → Undecided

Tim/Anastasia,

Backports are enabled in the trusty MAAS images juju is starting here.

Are you seeing otherwise?

On Fri, Jul 29, 2016 at 1:42 AM, Anastasia <email address hidden>
wrote:

> ** Changed in: juju-core
> Status: Triaged => Incomplete
>
> ** Changed in: juju-core
> Importance: Critical => Undecided
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> Incomplete
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

According to bug 1568895, an both daily and released MAAS images have backports enabled.

Changed in juju-core:
status: Incomplete → New

I verified this is the case by deploying a trusty instance in maas.
backports is enabled:

ubuntu@skookum:~$ apt-cache policy lxd
lxd:
  Installed: (none)
  Candidate: 2.0.3-0ubuntu2~ubuntu14.04.1
  Version table:
     2.0.3-0ubuntu2~ubuntu14.04.1 0
        100 http://archive.ubuntu.com//ubuntu/ trusty-backports/universe
amd64 Packages

On Fri, Jul 29, 2016 at 8:27 AM, Jason Hobbs <email address hidden>
wrote:

> According to bug 1568895, an both daily and released MAAS images have
> backports enabled.
>
> ** Changed in: juju-core
> Status: Incomplete => New
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> New
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

When this happens can you log into the host and capture the output from:

 $ ip route

I'm wondering whether the host machine has a route out. If not, I can understand that MAAS/host communication could work but fetching the image from the internet would understandably fail.

Andrew,

The cloud-init logs attached in first comment show that there was a route
out setup:

http://paste.ubuntu.com/21391121/

On Fri, Jul 29, 2016 at 12:17 PM, Andrew McDermott <
<email address hidden>> wrote:

> When this happens can you log into the host and capture the output from:
>
> $ ip route
>
> I'm wondering whether the host machine has a route out. If not, I can
> understand that MAAS/host communication could work but fetching the
> image from the internet would understandably fail.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1605986
>
> Title:
> Creating container: can't get info for image 'ubuntu-trusty'
>
> Status in juju-core:
> New
>
> Bug description:
> On a deploy of OpenStack in OIL, LXD containers on one machine failed
> to be created due to this error:
>
> "Creating container: can't get info for image 'ubuntu-trusty'"
>
> Trusty container setup on other machines in the same deployment worked
> fine.
>
>
> This is with 2.0 beta 11 and 2.0 RC2.
>
> Here is the juju status:
>
> http://paste.ubuntu.com/20733843/
>
> I will attach the logs from the machine and the logs from the
> controller node.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1605986/+subscriptions
>

I just ran into this too, and I definitely don't have a route out:

ubuntu@maas20-node2:~$ sudo ip route
10.0.0.0/24 dev lxdbr0 proto kernel scope link src 10.0.0.1
10.12.20.0/24 dev br-ens5 proto kernel scope link src 10.12.20.7
10.12.20.0/24 dev br-ens6 proto kernel scope link src 10.12.20.9
10.12.20.0/24 dev br-bond0 proto kernel scope link src 10.12.20.6
10.100.0.0/20 dev br-ens5.100 proto kernel scope link src 10.100.0.3

ubuntu@maas20-node2:~$ sudo lxc launch ubuntu:trusty c1
Creating c1
error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443

ubuntu@maas20-node2:~$ wget kernel.org
--2016-07-29 13:24:50-- http://kernel.org/
Resolving kernel.org (kernel.org)... 2620:3:c000:a:0:1991:8:25, 2001:4f8:1:10:0:1991:8:25, 198.145.20.140, ...
Connecting to kernel.org (kernel.org)|2620:3:c000:a:0:1991:8:25|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|2001:4f8:1:10:0:1991:8:25|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|198.145.20.140|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|149.20.4.69|:80... failed: Network is unreachable.
Connecting to kernel.org (kernel.org)|199.204.44.194|:80... failed: Network is unreachable.

Yet I do have a gateway specified in ENI:

ubuntu@maas20-node2:~$ grep -B 2 -A 5 gateway /etc/network/interfaces
auto br-bond0
iface br-bond0 inet static
    gateway 10.12.20.1
    address 10.12.20.6/24
    hwaddress 52:54:00:44:4f:42
    bridge_ports bond0
    dns-nameservers 10.12.20.2

And the device is UP:

ubuntu@maas20-node2:~$ ip a show br-bond0
9: br-bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 52:54:00:44:4f:42 brd ff:ff:ff:ff:ff:ff
    inet 10.12.20.6/24 brd 10.12.20.255 scope global br-bond0
       valid_lft forever preferred_lft forever
    inet 10.12.20.106/24 brd 10.12.20.255 scope global secondary br-bond0:1
       valid_lft forever preferred_lft forever
    inet 10.12.20.107/24 brd 10.12.20.255 scope global secondary br-bond0:2
       valid_lft forever preferred_lft forever
    inet 10.12.20.108/24 brd 10.12.20.255 scope global secondary br-bond0:3
       valid_lft forever preferred_lft forever
    inet 10.12.20.109/24 brd 10.12.20.255 scope global secondary br-bond0:4
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe44:4f42/64 scope link
       valid_lft forever preferred_lft forever

Changed in juju-core:
status: New → Confirmed
Andrew McDermott (frobware) wrote :

https://github.com/juju/juju/pull/5791 appears to be the culprit, at least for trusty.

Changed in juju-core:
status: Confirmed → Triaged
importance: Undecided → Critical
Changed in juju-core:
assignee: nobody → Dimiter Naydenov (dimitern)
Changed in juju-core:
status: Triaged → In Progress
Dimiter Naydenov (dimitern) wrote :

https://github.com/juju/juju/pull/5791 can't be the culprit as it landed in beta12 (to fix bug 1584616), whereas this bug is for beta11.

Also, looking at the logs provided it's unclear yet what's the cause. Will keep digging for the cause.

Dimiter Naydenov (dimitern) wrote :

I had a chat with jhobbs about this, and it looks like the issue is due to intermittent network connectivity issue on that machine 2. It happens occasionally on other machines. This is because Juju does not cache the lxd images on the controller, like with juju agents(tools), so the lxd cloud image is downloaded and cached for each provisioned host machine (before starting the first container on it). I'll find and post a link to the bug about that.

As agreed on IRC, marking this as Incomplete, to be reopened if it happens again (with beta13 the issue seems to be a lot less frequent).

Changed in juju-core:
status: In Progress → Incomplete
assignee: Dimiter Naydenov (dimitern) → nobody
Changed in juju-core:
importance: Critical → Undecided
milestone: 2.0-beta14 → none
Changed in juju-core:
assignee: nobody → Katherine Cox-Buday (cox-katherine-e)
assignee: Katherine Cox-Buday (cox-katherine-e) → nobody
Changed in juju-core:
status: Incomplete → Invalid
Changed in juju-core:
status: Invalid → New
summary: - Creating container: can't get info for image 'ubuntu-trusty'
+ Confusing error: Creating container: can't get info for image 'ubuntu-
+ trusty'
description: updated
tags: added: usability
Changed in juju-core:
status: New → Triaged
importance: Undecided → Medium
milestone: none → 2.0.0
Changed in juju-core:
importance: Medium → High
Richard Harding (rharding) wrote :

Agree at the least we should improve the error message output until we land the lxd image caching work.

affects: juju-core → juju
Changed in juju:
milestone: 2.0.0 → none
milestone: none → 2.0.0
Changed in juju:
assignee: nobody → Richard Harding (rharding)
Changed in juju:
milestone: 2.0.0 → 2.1.0
Changed in juju:
assignee: Richard Harding (rharding) → nobody
Anastasia (anastasia-macmood) wrote :

This bug has become about usability, early error reporting and clearer message.

Changed in juju:
importance: High → Medium
Curtis Hovey (sinzui) on 2017-02-17
Changed in juju:
milestone: 2.1-rc2 → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers