mongodb charm in allocating state - causes deployment to timeout

[Larry Michel]

This is for automated OpenStack Deployment. The deployment times out after > 1 hour and mongodb charm is still allocating.

From console.txt:
---------------------------------------------------------------------------
+ . ./pipeline_parameters
++ export OPENSTACK_RELEASE=icehouse
++ OPENSTACK_RELEASE=icehouse
++ export COMPUTE=nova-kvm
++ COMPUTE=nova-kvm
++ export BLOCK_STORAGE=cinder-iscsi
++ BLOCK_STORAGE=cinder-iscsi
++ export IMAGE_STORAGE=glance-ceph
++ IMAGE_STORAGE=glance-ceph
++ export PIPELINE_ID=08899cb6-e642-4acc-8eb0-e5c36f78cb2b
++ PIPELINE_ID=08899cb6-e642-4acc-8eb0-e5c36f78cb2b
++ export NETWORKING=nova-network-flatdhcp
++ NETWORKING=nova-network-flatdhcp
++ export UBUNTU_RELEASE=precise
++ UBUNTU_RELEASE=precise
---------------------------------------------------------------------------

---------------------------------------------------------------------------
    mongodb:
      branch: lp:charms/mongodb
      to:
      - lxc:nova-cloud-controller=0
---------------------------------------------------------------------------

---------------------------------------------------------------------------
2015-05-18 07:18:52,621 [INFO] deployer.import: Deploying service heat using local:precise/heat
2015-05-18 07:19:05,428 [INFO] deployer.import: Deploying service keystone using local:precise/keystone
2015-05-18 07:19:18,210 [INFO] deployer.import: Deploying service mongodb using local:precise/mongodb
2015-05-18 07:19:30,208 [INFO] deployer.import: Deploying service mysql using local:precise/mysql
2015-05-18 07:19:41,208 [INFO] deployer.import: Deploying service nova-compute using local:precise/nova-compute
2015-05-18 07:19:51,072 [INFO] deployer.import: Deploying service openstack-dashboard using local:precise/openstack-dashboard
2015-05-18 07:20:02,549 [INFO] deployer.import: Deploying service rabbitmq-server using local:precise/rabbitmq-server
2015-05-18 07:20:19,350 [DEBUG] deployer.import: Adding units...
2015-05-18 07:20:20,998 [WARNING] deployer.import: Config specifies num units for subordinate: ceilometer-agent
2015-05-18 07:20:20,999 [DEBUG] deployer.import: Service 'ceph' does not need any more units added.
2015-05-18 07:20:20,999 [DEBUG] deployer.import: Service 'cinder' does not need any more units added.
2015-05-18 07:20:21,002 [DEBUG] deployer.import: Service 'nova-cloud-controller' does not need any more units added.
2015-05-18 07:20:21,002 [DEBUG] deployer.import: Service 'ceilometer' does not need any more units added.
2015-05-18 07:20:21,003 [DEBUG] deployer.import: Service 'glance' does not need any more units added.
2015-05-18 07:20:21,004 [DEBUG] deployer.import: Service 'heat' does not need any more units added.
2015-05-18 07:20:21,004 [DEBUG] deployer.import: Service 'keystone' does not need any more units added.
2015-05-18 07:20:21,006 [DEBUG] deployer.import: Service 'mongodb' does not need any more units added.
...
2015-05-18 07:34:46,381 [DEBUG] deployer.env: Delta machine: 5/lxc/0 change:pending
2015-05-18 07:34:51,300 [DEBUG] deployer.env: Delta machine: 5/lxc/0 change:started
2015-05-18 07:34:51,382 [DEBUG] deployer.env: Delta unit: mongodb/0 change:allocating
2015-05-18 07:35:01,302 [DEBUG] deployer.env: Delta machine: 3/lxc/1 change:pending
2015-05-18 07:35:06,304 [DEBUG] deployer.env: Delta machine: 3/lxc/1 change:started
2015-05-18 07:35:06,386 [DEBUG] deployer.env: Delta machine: 1/lxc/0 change:pending
2015-05-18 07:35:06,387 [DEBUG] deployer.env: Delta unit: rabbitmq-server/0 change:installing
2015-05-18 07:35:36,314 [DEBUG] deployer.env: Delta machine: 2/lxc/0 change:pending
2015-05-18 07:35:56,318 [DEBUG] deployer.env: Delta machine: 1/lxc/1 change:pending
2015-05-18 07:36:16,320 [DEBUG] deployer.env: Delta unit: keystone/0 change:started
...
2015-05-18 07:52:51,500 [DEBUG] deployer.env: Delta unit: ceph/1 change:started
2015-05-18 07:53:06,509 [DEBUG] deployer.env: Delta unit: nova-compute/1 change:started
2015-05-18 08:49:32,673 [DEBUG] deployer.env: Connecting to environment...
2015-05-18 08:49:33,835 [DEBUG] deployer.env: Connected to environment
2015-05-18 08:49:33,837 [ERROR] deployer.import: Reached deployment timeout.. exiting
2015-05-18 08:49:33,838 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 214, in run
    self.check_timeout()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 165, in check_timeout
    raise ErrorExit()
ErrorExit
2015-05-18 08:49:33,853 [INFO] oil_ci.deploy.oil_deployer: Saving deployment artifacts to: ./artifacts
+ rc=1
+ echo 'Deployment returned: 1'
Deployment returned: 1
---------------------------------------------------------------------------

From juju_status.yaml.
---------------------------------------------------------------------------
  mongodb:
    charm: local:precise/mongodb-30
    exposed: false
    relations:
      replica-set:
      - mongodb
    units:
      mongodb/0:
        agent-state: allocating
        machine: 5/lxc/0
        public-address: 10.245.45.86
---------------------------------------------------------------------------

I have attached bundle and all machine logs.

[Wouter van Bommel (woutervb)]

The speed at which machine's are allocated and related timeouts is something that is out of control of the charm code.