After deploy of a murano-env with "Kubernetes Cluster (package)" with setting "Custom Docker registry URL" env became unavailable.

[Alexander Koryagin]

Hello,
Please take a look at the following issue:
    After deploy of a murano-env with "Kubernetes Cluster (package)" with setting "Custom Docker registry URL" env became unavailable.

Configuration:
  MOS 9.0 official ISO: MirantisOpenStack-9.0.iso

  id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
  ---+--------+---------------------+---------+------------+-------------------+------------+---------------+--------+---------
   4 | ready | slave-01_controller | 1 | 10.109.2.3 | 64:3a:68:0b:7f:af | controller | | 1 | 1
   1 | ready | slave-02_controller | 1 | 10.109.2.4 | 64:34:8b:c8:9e:57 | controller | | 1 | 1
   3 | ready | slave-03_controller | 1 | 10.109.2.5 | 64:fb:3a:e3:b0:39 | controller | | 1 | 1
   2 | ready | slave-04_compute | 1 | 10.109.2.6 | 64:b3:56:cb:60:3b | compute | | 1 | 1
   7 | ready | slave-05_compute | 1 | 10.109.2.7 | 64:b5:04:27:69:f6 | compute | | 1 | 1
   5 | ready | slave-06_compute | 1 | 10.109.2.8 | 64:43:29:cf:9c:81 | compute | | 1 | 1
   6 | ready | slave-07_cinder | 1 | 10.109.2.9 | 64:fd:f9:b3:fe:e0 | cinder | | 1 | 1

Actions:

1) OK - Deploy Env with Murano enabled.

2) OK - Install several murano packeges:
    Horizon -> Applications -> Manage -> Packages.
    Install:
    - Kubernetes Cluster | FQN = com.mirantis.docker.kubernetes.KubernetesCluster
    - Kubernetes Pod | FQN = com.mirantis.docker.kubernetes.KubernetesPod
    - Docker HTTPd | FQN = com.example.docker.DockerHTTPd

3) OK - In security rules open port 5000.

4) OK - Create ubuntu VM and assign floating IP to it.

5) OK - Inside created VM with ubuntu install docker and set up local Docker registry.

    # sudo apt-get update
    # curl -sSL https://get.docker.com/gpg | sudo apt-key add -
    # curl -sSL https://get.docker.com/ | sudo sh
    # sudo usermod -aG docker $(whoami)

    # docker version
    # docker run hello-world
    # docker run -d -p 5000:5000 --restart=always --name registry registry:2

    # docker pull ubuntu
    # docker tag ubuntu localhost:5000/local_ubuntu
    # docker push localhost:5000/local_ubuntu
    # docker pull localhost:5000/local_ubuntu

6) OK - Get address on your local Docker registry.
    It will be <floating IP of VM>:5000.
    In my case it is = 10.109.4.132:5000

7) OK - Create murano env.
    Horizon -> Applications -> Catalog -> Environments.

8) OK - Add to env following packages.
    - Kubernetes Cluster
    - Kubernetes Pod
    - Docker HTTPd

    Inside "Kubernetes Cluster" options set:
    - Custom Docker registry URL = 10.109.4.132:5000

9) OK - Run deploy ot the Murano env.

10) OK - Check that all instances are running.
    Here are my instances:
     - Name: murano-ynlxais7dy6w41-minion-1-onoco6xjhsso; Image: ubuntu14.04-x64-kubernetes; IP: 10.0.93.5, Floating IPs: 10.109.4.135;
     - Name: murano-ynlxais7dy6w41-gateway-1-6ecdu5qovso2; Image: ubuntu14.04-x64-kubernetes; IP: 10.0.93.6, Floating IPs: 10.109.4.136;
     - Name: murano-ynlxais7dy6w41-master-1-ouvafkv4k62f; Image: ubuntu14.04-x64-kubernetes; IP: 10.0.93.4, Floating IPs: 10.109.4.134;
     - Name: ubuntu_docker_registry; Image: image_ubuntu; IP: 10.109.5.10, Floating IPs: 10.109.4.132;

11) NOT OK - Check that deployment of the murano-env finished successfully.

    Deployment of the murano-env finished with unknonw status.
    It is not possible to check it because not possible to open "Horizon -> Applications -> Catalog -> Environments".

    Following error appears in web-browser:
      Something went wrong!
      An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.

    Deployment, launched from murano API finished with error:
      HTTPBadRequest: 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400)

    Here is extraction from "Horizon -> Applications -> Catalog -> Environments -> Env_Name -> Logs" before it became unavailable:
      . . .
      2016-08-23 11:25:42 — Action deploy is scheduled
      2016-08-23 11:25:52 — Creating VMs for Kubernetes cluster
      2016-08-23 11:25:53 — Installing HTTPd
      2016-08-23 11:25:53 — Deploying Replication Controller for Pod testpod
      2016-08-23 11:25:55 — Setting up Kubernetes cluster
      2016-08-23 11:25:55 — Creating Kubernetes Master
      2016-08-23 11:25:55 — Creating Kubernetes Node minion-1
      2016-08-23 11:25:55 — Creating Gateway node for Kubernetes services
      2016-08-23 11:35:36 — Configuring etcd master node
      2016-08-23 11:35:43 — Configuring etcd node minion-1
      2016-08-23 11:36:19 — Configuring etcd node gateway-1
      2016-08-23 11:36:37 — Setup Flannel network on master node
      2016-08-23 11:36:37 — Setup Kubernetes API host
      2016-08-23 11:36:44 — Kubernetes API is now available at http://10.109.4.134:8080
      2016-08-23 11:36:44 — Setup Flannel network on minion-1
      2016-08-23 11:36:45 — Setup Flannel network on gateway-1
      2016-08-23 11:36:49 — Setup Kubernetes Node on minion-1
      2016-08-23 11:36:49 — Setup Load Balancer on gateway-1
      2016-08-23 11:36:51 — Gateway gateway-1 is now available at 10.109.4.136
      2016-08-23 11:36:51 — cAdvisor monitoring for Node minion-1 is now available at http://10.109.4.135:4194
      2016-08-23 11:37:25 — Kubernetes cluster is up and running
      2016-08-23 11:37:29 — Pod testpod is ready
      2016-08-23 11:37:29 — Creating services for Pod testpod
      . . .

    # . openrc
    # murano environment-list
      Could not initialise glance client. Image creation will be unavailable.
      +----------------------------------+----------------------+--------+---------------------+---------------------+
      | ID | Name | Status | Created | Updated |
      +----------------------------------+----------------------+--------+---------------------+---------------------+
      | 6e1bca0883fb4a5ba55d86fafe08e38c | MuranoEnv_1145281055 | ready | 2016-08-23T11:20:10 | 2016-08-23T11:38:00 |
      +----------------------------------+----------------------+--------+---------------------+---------------------+

    # murano environment-show MuranoEnv_1145281055
      Could not initialise glance client. Image creation will be unavailable.
      400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect. (HTTP 400)

Performing the same actions, but WITHOUT setting "Custom Docker registry URL" in "Kubernetes Cluster" package will lead to successful murano-env deploy.

[Alexander Koryagin (akoryagin)]

> mysql -u murano -p
> use murano;
> select * from environment;

http://paste.openstack.org/show/562475/

[Alexander Koryagin (akoryagin)]

Update:
Buf not related to Docker and setting "Custom Docker registry" in "Kubernetes Cluster (package)".
It can be reproduced without creating VM with docker and without changing any values in "Kubernetes Cluster (package)" configuration.

Skipping steps 3-7 will lead to murano env successful deploy.
Performing 'KubernetesCluster' deletion (step 3) and then adding back (step 5) - will lead to issue after deploy.

Content of variables:
  pprint(all_services_dict) -> http://paste.openstack.org/show/562922/
  pprint(kub_cluster_info) -> http://paste.openstack.org/show/562924/
  pprint(all_services_after_del_dict) -> http://paste.openstack.org/show/562925/
  pprint(all_services_after_post_dict) -> http://paste.openstack.org/show/562926/

Actions:

# 1) Get information about all serveces that will be added to murano env.
all_services = murano.murano.services.list(environment.id, session.id)
all_services_dict = [i.to_dict() for i in all_services]

# 2) Get latest information about 'KubernetesCluster'.
kub_cluster_info = [
    i.to_dict() for i in all_services
    if 'KubernetesCluster' in i.to_dict()['?']['type']][0]

# 3) Delete 'KubernetesCluster' service from murano env.
murano.murano.services.delete(
    environment.id, session_id=session.id,
    path='/{0}'.format(kub_cluster_info['?']['id']))

# 4) Check that 'KubernetesCluster' was deleted from murano env.
all_services_after_del = murano.murano.services.list(environment.id, session.id)
all_services_after_del_dict = [i.to_dict() for i in all_services_after_del]
assert not all_services_dict == all_services_after_del_dict

# 5) Post back 'KubernetesCluster' to murano env.
murano.murano.services.post(
    environment.id, session_id=session.id, path='/',
    data=kub_cluster_info)

# 6) Check that 'KubernetesCluster' was added to murano env.
all_services_after_post = murano.murano.services.list(environment.id, session.id)
all_services_after_post_dict = [i.to_dict() for i in all_services_after_post]

# 7) Check that murano env now has the same information (after 'KubernetesCluster' was added) as before any modifications.
all_services_dict.sort()
all_services_after_post_dict.sort()
assert all_services_dict == all_services_after_post_dict

# 8) Start deploy of a murano env.
murano.murano.sessions.deploy(environment.id, session.id)

# 9) Wait till it will be finishe. Here will be an issue.

[Alexander Koryagin (akoryagin)]

PS. All operations above ^^^ are performed with not deployed murano-env.

[Timur Nurlygayanov (tnurlygayanov)]

<131>Aug 25 19:30:02 node-2 murano-api: 2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics [req-e798198a-8ade-417d-a0a2-0f0234059905 26652d18a72443b5873aa6368d3a2366 f9c0fc982b0a4f23a2deb1cf42515f72 - - -] API Environments method Show raised an exception
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics Traceback (most recent call last):
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/api/v1/request_statistics.py", line 57, in wrap
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics result = func(*args, **kwargs)
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/utils.py", line 73, in __inner
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics return func(self, request, environment_id, *args, **kwargs)
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/api/v1/environments.py", line 118, in show
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics env['services'] = get_data(environment_id, '/services', session_id)
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics File "/usr/lib/python2.7/dist-packages/murano/db/services/core_services.py", line 65, in get_data
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics srv['?']['status'] = get_status(environment_id, srv['?']['id'])
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics TypeError: string indices must be integers
2016-08-25 19:30:02.778 18552 ERROR murano.api.v1.request_statistics
<135>Aug 25 19:30:02 node-2 murano-api: 2016-08-25 19:30:02.792 18552 DEBUG murano.api.v1.request_statistics [req-e798198a-8ade-417d-a0a2-0f0234059905 26652d18a72443b5873aa6368d3a2366 f9c0fc982b0a4f23a2deb1cf42515f72 - - -] Updating count stats for Environments, Show on object <murano.api.v1.request_statistics.RequestStatisticsCollection object at 0x7fa203eec590> update_error_count /usr/lib/python2.7/dist-packages/murano/api/v1/request_statistics.py:84
<131>Aug 25 19:30:02 node-2 murano-api: 2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi [req-e798198a-8ade-417d-a0a2-0f0234059905 26652d18a72443b5873aa6368d3a2366 f9c0fc982b0a4f23a2deb1cf42515f72 - - -] Exception handling resource: string indices must be integers
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi Traceback (most recent call last):
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi File "/usr/lib/python2.7/dist-packages/murano/common/wsgi.py", line 429, in __call__
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi action_result = self.execute_action(action, request, **action_args)
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi File "/usr/lib/python2.7/dist-packages/murano/common/wsgi.py", line 444, in execute_action
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi return self.dispatch(self.controller, action, request, **action_args)
2016-08-25 19:30:02.792 18552 ERROR murano.common.wsgi File "/usr/lib/python2.7/dist-packages/murano/common/wsgi.py", l...

[Timur Nurlygayanov (tnurlygayanov)]

logs of Murano API http://paste.openstack.org/show/563447/

[Timur Nurlygayanov (tnurlygayanov)]

Murano team, could you please provide a fix or at least workaround for the issue? It is blocker for the feature testing.

[Dmytro Dovbii (ddovbii)]

I've already started investigating, but was unable to reproduce bug. akoryagin shared code of tests with me and I'm going to continue investigation using them

[Alexander Koryagin (akoryagin)]

Following WA has helped:

all_services = murano.murano.services.list(environment.id, session.id)
all_services_dict = [i.to_dict() for i in all_services]

# Delete from murano env info about KubernetesCluster to update it
for service in all_services_dict:
    murano.delete_service(environment, session, service)

# Set "Custom Docker registry URL" and "Docker registry mirror URL" in
# "Kubernetes Cluster" config.
all_services_dict[0]['dockerRegistry'] = '{0}:{1}'.format(
    vm_floating_ip, self.docker_registry_port)

# Add back updated info about KubernetesCluster to murano env
for service in all_services_dict:
    murano.create_service(environment, session, service)

# Deploy environment
deployed_environment = murano.deploy_environment(environment, session)

Previously I've deleted one service, change it and then add back --> Env deploy error.
In the code above, I've deleted all services, then add changes and add back all services --> Env deploy was successful.