Consistently Failing - Deploy OpenStack all in one node using Ansible

[Manoj]

I have tried all the alternatives provided to Deploy OpenStack all in one node using Ansible

I build the images "with and without" setting up docker registry.

I want to just deploy Binary docker images and deploy OpenStack using Ansible.
So I don't have to build images locally.

I am using following command to deploy

sudo tools/kolla-ansible -i ansible/inventory/all-in-one -p ansible/site.yml deploy

It Always failes with Following ERROR

TASK: [common | Starting log_data container] **********************************
failed: [localhost] => {"changes": ["{\"stream\":\"Trying to pull repository docker.io/kollaglue/centos-binary-data ...\"}\r\n", "{\"status\":\" not found\"}\r\n", "{\"errorDetail\":{\"message\":\"Error: image kollaglue/centos-binary-data:latest not found\"},\"error\":\"Error: image kollaglue/centos-binary-data:latest not found\"}\r\n"], "failed": true, "status": ""}
msg: Unrecognized status from pull.

Please document all the steps and configuration at one place.

And give me pointers to resolve this issue.

I can see all the images in docker repository using docker images, so why ansible is not able to find those images.

Deploying Playbook : ansible-playbook -i ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml ansible/site.yml

PLAY [ceph-mon;ceph-osd] ******************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [ceph | Ensuring templates config directory exists] *********************
skipping: [localhost]

TASK: [ceph | Ensuring config directory exists] *******************************
skipping: [localhost]

TASK: [ceph | Copying over config(s)] *****************************************
skipping: [localhost] => (item=['roles/ceph/templates/ceph.conf.j2', u'/usr/share/kolla/templates/ceph-mon/ceph.conf_minimal'])
skipping: [localhost] => (item=['/etc/kolla/config/ceph.conf', u'/usr/share/kolla/templates/ceph-mon/ceph.conf_augment'])

TASK: [ceph | Merging the config files and saving to the final destination] ***
skipping: [localhost]

TASK: [ceph | Ensuring templates config directory exists] *********************
skipping: [localhost]

TASK: [ceph | Ensuring config directory exists] *******************************
skipping: [localhost]

TASK: [ceph | Copying over config(s)] *****************************************
skipping: [localhost] => (item=['roles/ceph/templates/ceph.conf.j2', u'/usr/share/kolla/templates/ceph-osd/ceph.conf_minimal'])
skipping: [localhost] => (item=['/etc/kolla/config/ceph.conf', u'/usr/share/kolla/templates/ceph-osd/ceph.conf_augment'])

TASK: [ceph | Merging the config files and saving to the final destination] ***
skipping: [localhost]

TASK: [ceph | Cleaning up temp file on localhost] *****************************

ASK: [ceph | Generating Initial Ceph keyrings and monmap] ********************
skipping: [localhost]

TASK: [ceph | Waiting for a few settings for cluster to generate keys] ********
skipping: [localhost]

TASK: [ceph | Setting host for cluster files] *********************************
skipping: [localhost]

TASK: [ceph | Fetching Ceph keyrings] *****************************************
skipping: [localhost]

TASK: [ceph | Reading json from variable] *************************************
skipping: [localhost]

TASK: [ceph | Pushing Ceph keyring for OSDs] **********************************
skipping: [localhost] => (item={# ceph_files['ceph.client.admin.keyring'] #})

TASK: [ceph | Pushing Ceph keyrings for Mons] *********************************
skipping: [localhost] => (item={# ceph_files['ceph.client.admin.keyring'] #})
skipping: [localhost] => (item={# ceph_files['ceph.client.mon.keyring'] #})
skipping: [localhost] => (item={# ceph_files['ceph.monmap'] #})

TASK: [ceph | Starting ceph-mon container] ************************************
skipping: [localhost]

TASK: [ceph | Looking up disks to bootstrap for Ceph] *************************
skipping: [localhost]

TASK: [ceph | Bootstrapping Ceph OSDs] ****************************************
skipping: [localhost]

TASK: [ceph | Waiting for bootstrap containers to exit] ***********************
                                                                                                                                   87,1 56%

TASK: [ceph | Cleaning up bootstrap containers] *******************************
skipping: [localhost]

TASK: [ceph | Looking up OSDs for Ceph] ***************************************
skipping: [localhost]

TASK: [ceph | Mounting Ceph OSD volumes] **************************************
skipping: [localhost] => (item=osds.disks)

TASK: [ceph | Gathering OSD IDs] **********************************************
skipping: [localhost] => (item=osds.disks)

TASK: [ceph | Starting ceph-osds container] ***********************************
skipping: [localhost] => (item=[{u'skipped': True, u'changed': False}, 'osds.disks'])

PLAY [haproxy;mariadb;rabbitmq;cinder-api;glance-api;keystone;nova-api;neutron-server;swift-proxy-server] ***

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [common | Ensuring config directory exists] *****************************
ok: [localhost]

TASK: [common | Copying over config(s)] ***************************************
ok: [localhost]

TASK: [common | Starting log_data container] **********************************
failed: [localhost] => {"changes": ["{\"stream\":\"Trying to pull repository docker.io/kollaglue/centos-binary-data ...\"}\r\n", "{\"status\":\" not found\"}\r\n", "{\"errorDetail\":{\"message\":\"Error: image kollaglue/centos-binary-data:latest not found\"},\"error\":\"Error: image kollaglue/centos-binary-data:latest not found\"}\r\n"], "failed": true, "status": ""}
msg: Unrecognized status from pull.

FATAL: all hosts have already failed -- aborting

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/site.retry

localhost : ok=11 changed=0 unreachable=0 failed=1

Command failed ansible-playbook -i ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml ansible/site.yml

[Sam Yaple (s8m)]

With the output you have in this ticket it shows you are trying to fetch images from the Docker Hub rather than use locally build images. If you want to use locally built images for an AIO you have to set the value 'docker_pull_policy: "missing"' in the globals.yml

If you are pushing to a local registry you must configure your registry connection in the globals.yml There is an example in the file that you can reference.

[Manoj (manoj-kumar-jainman)]

Sam,

My environment is Centos 7.

 I am using default globals.xml (No modifification except network interface)

My docker_pull_policy: configuration is missing
and kolla_install_type: "binary"

Still this issue is coming.

I want to fetch images from Docker hub not from local registry.

I am not sure if have to configure specific Docker Hub to fetch kolla images.

If so, what is Docker hub URL to fetch kolla images and how to configure it.

Is there some way, I can test it working?

I have also use tools\cleanup-images , tools\cleanuphost , tools\cleanup-containers and rerun command to deploy

sudo tools/kolla-ansible -i ansible/inventory/all-in-one -p ansible/site.yml deploy

But I got same error always.

PLease point me how I can solve this issue.

Thanks in Advance
Manoj

[Sam Yaple (s8m)]

Manoj,

If you set 'docker_pull_policy: "missing"' and still have this issue it is because you do not have any images on your host.

You need to build images first and then run the ansible playbooks the with correct settings in the globals.yml

If you still have issues, please post the output of `docker images` and the contents of your globals.yml

[Vu Nguyen Duy (CBR09) (nguyenduyvu099)]

Hi Manoj,
You can use images(ubuntu-source) on DockerHub that Sam uploaded. You must change options in globals.yml.

kolla_base_distro: "ubuntu" # default is centos
kolla_install_type: "source" # default is binary
docker_pull_policy: "always" # default is always.
openstack_release: "liberty-rc1" # default is latest

If not, like Sam said, you need to build images first and run the ansible playbooks with correct settings in the globals.yml

[Manoj (manoj-kumar-jainman)]

Hi Sam,

I have started working on this project around a month before part time. As
it took me more time than expected to (Partial successfully) install
OpenStack.
Here is the summary of my first experience with Kolla (Installing
Openstack). As a first time user It is expected to face some challanges -

1. There are lots of options (All-in-one / Multinode ) - Ansible / Heat
/Vagrant / Binary / Source.

2. The document is very confusing for first time user as the steps are
spread over several document.

3. Doc is not very clear about what mode is needed to build. You have to
build even for binary mode. As I was experimenting, My guess was you don't
have to build.

4. You have to mandatory run docker registry (I used port 5010, as 5000
was conflicting with keystone public port and Deployment fails)

sudo docker run -d -p 5010:5010 --restart=always --name registry registry:2

5. I have to experiment several time with option source / binary.
 globals.yml is only used for Deploying Kolla. So changing option for
build type in global.yml. I have to use following command

sudo tools/build.py -n localhost:5010/kollaglue --base centos --type
source --push --debug 2>&1 | tee KollaBuild.txt
  [The options from globals.yml is not used here]

this corresponds to following options in gloabls.yml used during during
deployment. Options for build and deployment should match otherwise image
will not be found.

kolla_install_type: "source"
docker_registry: "localhost:5010"
docker_namespace: "kollaglue"
docker_insecure_registry: "True"
docker_pull_policy: "missing"

On sucessful build you should able to find images (THERE WILL BE SEVERAL
IMAGES)

[docker@hyp group_vars]$ docker images | grep nova
localhost:5010/kollaglue/centos-source-nova-compute latest
             161b4a8d78c7 17 hours ago 778.3 MB
localhost:5010/kollaglue/centos-source-nova-novncproxy latest
             6ead5ef52336 17 hours ago 719.4 MB

6. Most of the times (Builing images will fail first time). Please rerun.
  Even after that there are several images failed to build.

INFO:__main__:Images that failed to build
INFO:__main__:===========================
ERROR:__main__:heat-api-cfn^M Failed with status: error
ERROR:__main__:cinder-base^M Failed with status: error
ERROR:__main__:rsyslog^M Failed with status: error
ERROR:__main__:designate-api^M Failed with status: error
ERROR:__main__:openstack-base^M Failed with status: error
ERROR:__main__:gnocchi-base^M Failed with status: error
ERROR:__main__:openvswitch-vswitchd^M Failed with
status: error
ERROR:__main__:nova-network^M Failed with status: error
ERROR:__main__:designate-central^M Failed with
status: error
ERROR:__main__:murano-engine^M Failed with status: error
ERROR:__main__:neutron-server^M Failed with status: error
ERROR:__main__:magnum-base^M Failed with status: error
ERROR:__main__:magnum-conductor^M Failed w...

[Manoj (manoj-kumar-jainman)]

More Steps ---

7. After image are build, You need to deploy Openstack. Make sure your
gloabls.yml has following entries correctly

kolla_install_type: "source"
docker_registry: "localhost:5010"
docker_namespace: "kollaglue"
docker_insecure_registry: "True"
docker_pull_policy: "missing"

kolla_internal_address: "10.1.2.10" //Find the free address on you
primary network interface. My primary network 10.1.2 / 24

#network_interface: "eth0"
network_interface: "enp5s0f0" //Find the primary interface
using ifconfig. My primary interface was enp5s0f0
#neutron_external_interface: "eth1"
neutron_external_interface: "enp5s0f1"

8. Modify ansible/group-vars/all.yml to override default properties. I did
modify following property. Otherwise deployment was failing.

kolla_install_type: "source"

9. I always got problem to start haproxy container. All deployment stops
after this container failed to start. To overcome this issue
 Modify ansible/group-vars/all.yml , disable haproxy
enable_haproxy: "no"

10. deploy Openstack

sudo ansible-playbook -i inventory/all-in-one -e @/etc/kolla/globals.yml -e
@/etc/kolla/passwords.yml site.yml 2>&1 | tee KollaDeploy.txt

If something is failed kill all the ansible running instance. I found
ansible-playbook keep running after failure. So I found the process and
kill it manually and rerun.

With EVERYTHING OK. You will get messge "localhost :
ok=289 changed=167 unreachable=0 failed=0"

Also I found, it does skips deploying lots of container. Not Sure Why?

TASK: [ironic | Waiting for Ironic bootstrap container to exit]
***************
skipping: [localhost]

TASK: [ironic | Cleaning up boostrap container]
*******************************
skipping: [localhost]

TASK: [ironic | Starting Ironic-api container]
********************************
skipping: [localhost]

TASK: [ironic | Starting Ironic-conductor container]
**************************
skipping: [localhost]

TASK: [ironic | Starting Ironic-discoverd container]
**************************
skipping: [localhost]

TASK: [ironic | Starting Ironic-pxe container]
********************************
skipping: [localhost]

PLAY RECAP
********************************************************************
localhost : ok=289 changed=167 unreachable=0 failed=0

11. If the deployment is successful, Lauch OpenStack with url http://<your
configure IP>. You will see openstack admin page.

SAM, It is good experience to work with Kolla. I thing it will be useful
for the community who is first time doing it.

If you are Ok, I can capture the structured information on Wiki.

Thanks
Manoj

On Sun, Oct 4, 2015 at 6:42 PM, Manoj Jain <email address hidden>
wrote:

> Hi Sam,
>
>
> I have started working on this project around a month before part time. As
> it took me more time than expected to (Partial successfully) install
> OpenStack.
> Here is the summary of my first experience with Kolla (Installing
> Openstack). As a first time user It is expected to face some challanges -
>
>
> 1. There are lots of options (All-in-one / Multinode ) - Ansible / Heat
> /Vagrant / Binary / Source.
>
> ...

[Steven Dake (sdake)]

Manoj,

I'd recommend making documentation changes as you think would make the documenataion more clear. We have had several different people improve on the documentation over time, perhaps its your turn to make em better :)

Use the standard git workflow. We don't require bug ids, and i'd prefer documentation be stored in the repository rather then a wiki page.

Regards,
-steve

[Dodi (dodi-4)]

changing the openstack_release in /etc/kolla/globals.yml did the trick when pulling the docker images as the tag "latest" was not an available tag.

changed to -> openstack_release: "liberty-rc1"

[Steven Dake (sdake)]

Folks,

I am closing this as invalid. We do not currently provide consist images in the docker hub. If you want to use Kolla you must push images to your local registry or alternatively use the local docker cache as outlined in:
http://docs.openstack.org/developer/kolla