Juju does not deploy CentOS images in LXD

[Matt Bruzek]

I am trying to deploy CentOS image with LXC and believe it should be possible because I can run CentOS in LXC using standard LXC commands.

$ sudo lxc-create -t centos -n centos7 -- --release 7
$ sudo lxc-start -n centos7
$ sudo lxc-attach -n centos7
bash-4.2# cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)
bash-4.2# uname -a
Linux centos7 3.19.0-28-generic #30-Ubuntu SMP Mon Aug 31 15:52:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I have created a "centos" local Juju environment with default-series: centos7

Also created a centos charm that runs a simple yum command in a centos7 release directory.

I can bootstrap the centos environment (debug log attached) but when I try to deploy the centos charm
$ juju deploy --repository=workspace/charms local:centos7/centos
Added charm "local:centos7/centos-0" to the environment.

Juju can not find the tools for the centos image and the deployment is stuck in pending "no matching tools available".

[Matt Bruzek (mbruzek)]

I do not see centos listed in the index at streams.canonical.com:

https://streams.canonical.com/juju/devel/tools/streams/v1/index2.json

But I do see the tools for the version of Juju that I am using 1.25-alpha1

https://streams.canonical.com/juju/tools/devel/juju-1.25-alpha1-centos7-amd64.tgz

[Bogdan Teleaga (bteleaga)]

CentOS is only supported in cloud providers atm and has only been tested on a subset of those. Since there's no image metadata for CentOS on streams.canonical.com you'd probably have to use openstack or maas(which both support custom image metadata). You could theoretically get any provider that has cloud images for CentOS with cloudinit running, but it does require some small modifications to the source code.

[Bogdan Teleaga (bteleaga)]

Oh, and it should probably look at https://streams.canonical.com/juju/tools/streams/v1/index2.json

[Matt Bruzek (mbruzek)]

Is there any technical reason for CentOS support only on the cloud providers? Ubuntu is able to run CentOS LXC containers as was shown in the bug description. I see centos is in the tools index. Deploying a CentOS LXC image in Juju should be technically possible (and highly desired by customers).

[Curtis Hovey (sinzui)]

I think this issue is really about the fact the cloud-images.ubuntu.com only contains ubuntu images.
    http://cloud-images.ubuntu.com/releases/streams/v1/index.json
and in the more general case of cloud images suitable for lxc
    http://cloud-images.ubuntu.com/

Regardless of the existence of agents for centos7, Juju needs to find an image to deploy first. We are committed to providing images streams for ubuntu, windows, and centos7 in the near future. We also hope Juju will allow us to manually provision existing centos7 instances to make it easier for every one to try this feature as it is being developed.

Also, The real location for agent streams is.
    https://streams.canonical.com/juju/tools/streams/v1/index2.json
Centos7 is listed there. We make centos agents for 1.24+. We use "agent-stream" to select the products.

[Ian Booth (wallyworld)]

I'm not sure why image metadata is being discussed. The issue doesn't appear to be related to starting an image. It's the local provider - there is no image id that is relevant.

The error is that Juju cannot find suitable tools to run inside the container.

[Robert C Jennings (rcj)]

This would require the changing use of query data with the `ubuntu-cloudimg-query` command for LXC image download to use simplestreams instead (https://github.com/juju/juju/blob/master/container/image.go#L74)

Query data is deprecated.
Query data will only list Ubuntu cloud images.

[Cheryl Jennings (cherylj)]

See also bug #1506121

[Dean Maniatis (dean-maniatis)]

To be able to move from local dev. environment (using LXD containers) to private cloud (e.g. for staging) to finally be able to deploy to a public cloud ( e.g. production) is one of the key benefits of using Juju for operations. What sets Juju apart from existing solutions is that you can do this in a very streamlined operation with blazing fast response and almost no extra effort from user.

Given that the majority of enterprise world uses RHEL based OS for their workloads, the current limited support of Juju with RHEL-based OSes (no LXD support, support for private cloud, support with AWS with custom simplestreams repo) creates a discontinuity in the above use-case. I find this feature/bug currently one of the most important to gain traction in enterprise without having to spend a lot of effort to port legacy applications from RHEL to Ubuntu just so that you can have a PoC with Juju.

Nevertheless, I'm pen to any suggestions/hacks from community on how to use Juju with CentOS LXD locally so I can finally demo my cloud workload (7+ different VM types, 20+ microservices) on Juju.

[Andrew Wilkins (axwalk)]

With a small patch to Juju, it is possible to manually provision CentOS in a LXD container: https://github.com/juju/juju/pull/6954

[Andrew Wilkins (axwalk)]

I had a thought on what we can do as a short-term measure. It requires code changes to Juju, but not a lot. It just needs to be changed to use an already-cached image if it can't also find it in a remote.

I've experimented a bit, and managed to apply a few changes to the centos/7 image from linuxcontainers.org so that it runs cloud-init. I'll look at providing a program/script that does this repeatably.

[Andrew Wilkins (axwalk)]

I'm not sure that this will make it into 2.1, but I have a PR here: https://github.com/juju/juju/pull/6970.

This can be used in conjunction with https://github.com/axw/juju-lxd-centos-image-builder, which modified and repacks the linuxcontainers.org centos/7 image with enough cloud-init bits to get Juju going.

[Dean Maniatis (dean-maniatis)]

Hello Andrew, thanks for the code, I will have a look and give you feedback. In general the feature to be able to specify custom local LXD images, whether during bootstrapping or during deployment, i think is really useful for use-cases beyond the above.

[Andrew Wilkins (axwalk)]

I'm going to mark this as Fix Committed, as the necessary Juju changes are committed. We're still lacking a pre-canned image for CentOS-with-cloud-init, but that's independent of Juju.

[Dan Watkins (oddbloke)]

The cloud-images project is specifically for issues related to the Ubuntu cloud images, so I've marked this as Invalid.

[Matt Bruzek (mbruzek)]

The Juju documentation indicates we can deploy CentOS charms. https://jujucharms.com/docs/2.0/juju-centos

juju deploy lp:~me/centos7/charm

The user doesn't know (or care) if the image is a cloud image.

It has been in the Juju documentation since 1.24 and it hasn't worked that I know of.

[Dean Maniatis (dean-maniatis)]

There should be workflow parity between deploying Ubuntu or CentOS based charms. The fix provided remedies the problem but introduces some extra steps from the end-user making the whole experience a little bit more elaborate. What about pushing Andrew's work on preparing the centos7 image for Juju part of the CI of linuxcontainers.org?

[Erik Lönroth (erik-lonroth)]

I'm returning to this issue as I have serious problems with the state I'm left in.

At the moment, its very difficult for me to distinguish this issue between:

1. This being a real bug.
2. This being a method problem, me not knowing how to make this work. (which I still can't)

Think about it. A NEW juju user will believe this is an error - concluding juju as broken and go on with their lives. That cant be good.

[feng xia (fengxia41103)]

I have been struggling to bootstrap a CentOS slave using LXD or MAAS cloud, both failed for different reasons:

LXD; no tools found, though stream sever has the agent listed (2.0.2).
MAAS: provisioned centOS node can not ping internet, thus yum will fail with repo not found.

My objective is to create a Juju managed CentOS node, as little setup as possible. LXD would be fine.

[Andrew Wilkins (axwalk)]

Feng, this bug was fixed in Juju 2.1.0; you will have to upgrade to get it working in LXD. I've just tested things with 2.1.0, and it's working for me. Here's what I did:

1. I created a compatible LXD image, as described in my blog [0] (see the final paragraph in particular).
2. Bootstrapped a Juju 2.1.0 "localhost" (LXD) controller. See the release notes [1] for instructions on how to get the latest release of Juju.
3. Ran "juju add-machine --series=centos7".

[0] https://awilkins.id.au/post/juju-2.1-lxd-centos/
[1] https://jujucharms.com/docs/2.1/reference-release-notes