unable to launch containers, Unexpected Env format for 'ContainerConfig.Env'

[Steve Noyes]

I upgraded kolla, built all new containers, and upgraded kolla-mesos. When I run, all container launches in mesos fail:

E0210 15:39:21.319217 25394 slave.cpp:3703] Container '78ecedb1-b07f-4595-9f51-e0eac93d966c' for executor 'kolla_infra_memcached_memcached.5cd9d8c8-d036-11e5-956f-7a22ec38d4c7' of framework f850ae0f-13e7-4f0d-a50e-d2ce4935164a-0001 failed to start: Unable to create image: Unexpected Env format for 'ContainerConfig.Env'

W0210 15:39:21.321494 25389 docker.cpp:1050] Ignoring updating unknown container: 78ecedb1-b07f-4595-9f51-e0eac93d966c

E0210 15:39:22.365705 25390 slave.cpp:3703] Container '48fd906d-5af0-436c-952f-4d100942f447' for executor 'kolla_openstack_nova_nova-compute.5d7f197b-d036-11e5-956f-7a22ec38d4c7' of framework f850ae0f-13e7-4f0d-a50e-d2ce4935164a-0001 failed to start: Unable to create image: Unexpected Env format for 'ContainerConfig.Env'

E0210 15:39:22.366060 25390 slave.cpp:3703] Container '5c65646e-6812-4d64-9850-50b5b0b6f54d' for executor 'kolla_openstack_nova_nova-consoleauth.5d7d6bca-d036-11e5-956f-7a22ec38d4c7' of framework f850ae0f-13e7-4f0d-a50e-d2ce4935164a-0001 failed to start: Unable to create image: Unexpected Env format for 'ContainerConfig.Env'

W0210 15:39:22.379226 25394 docker.cpp:1050] Ignoring updating unknown container: 48fd906d-5af0-436c-952f-4d100942f447

W0210 15:39:22.383344 25394 docker.cpp:1050] Ignoring updating unknown container: 5c65646e-6812-4d64-9850-50b5b0b6f54d

This could be a problem specific to my environment, but I'm not sure how t9o figure out what is going wrong here. Any help is appreciated.

Running docker 1.9.1, docker-py 1.6 mesos 0.27.0
Linux ol7-dev-mesos2 4.1.12-32.2.1.el7uek.x86_64 #2 SMP Sun Jan 31 23:50:34 PST 2016 x86_64 x86_64 x86_64 GNU/Linux

[Steve Noyes (steve-noyes)]

When I downgraded mesos to 0.25.0, it started working again.

[Steve Noyes (steve-noyes)]

Looks like it's related to this mesos checkin:

https://<email address hidden>/msg05586.html

[Angus Salkeld (asalkeld)]

It looks like we need to add:

ENV KOLLA_...

To the base image in kolla.

See: https://docs.docker.com/engine/reference/builder/

It seems like mesos is using docker inspect to make sure that the env variable exists first, not sure why.

[Yong Feng (fengyong-gm)]

It seems Mesos has fixed the issue in https://issues.apache.org/jira/browse/MESOS-4587.

Please build a new Mesos and try again.