[contrail] Deployment has failed with 50 hw nodes

[Sergey Galkin]

Steps to reproduce
1. Install Fuel 6.1
2. Install contrail plugin
3. Try to deploy cluster with 3 controller in HA, 43 computes and 3 OS nodes

Deployment has failed with error

Deployment has failed. Method granular_deploy. Failed to execute hook 'contrail-2.0.0' Failed to run command cd /etc/fuel/plugins/contrail-2.0/ && if hiera user_node_name | egrep '^contrail-[0-9]'; then puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp; fi

 File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
DataError: (DataError) value too long for type character varying(255)
 'UPDATE nodes SET status=%(status)s, error_type=%(error_type)s, error_msg=%(error_msg)s WHERE nodes.id = %(nodes_id)s' {'status': u'error', 'nodes_id': 88, 'error_type': u'deploy', 'error_msg': u"Failed to run command cd /etc/fuel/plugins/contrail-2.0/ && if hiera user_node_name | egrep '^contrail-[0-9]'; then puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp; fi\n\nTask: ab64b86e-9cbe-491f-8e2d-d6c8763073f9: shell timeout error: execution expired\nTask timeout: 720, Retries: 3"}

snapshot available in http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-14_12-15-53.tar.xz

[Vladimir Sharshov (vsharshov)]

Main problem on node-104. Because we run network reconfiguration in same time then run shell, we can get such error. I suggest to include shell condition in to puppet manifest and run it as puppet task. It is tolerant with temporary network problem.

More details:

D, [2015-09-15T11:25:29.778436 #7925] DEBUG -- : base.rb:168:in `create_reply' Encoded a message for request c44b655cadb0507998307e39bb7dc058
D, [2015-09-15T11:25:29.778648 #7925] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Rabbitmq
D, [2015-09-15T11:25:29.778828 #7925] DEBUG -- : rabbitmq.rb:258:in `publish' Sending a broadcast message to RabbitMQ target '/reply-queue/amq.gen-7FCnFPWA2vOnhPO89BvPVg' with headers '{"expiration"=>"4294967000"}'
E, [2015-09-15T11:25:29.779128 #7925] ERROR -- : agents.rb:138:in `rescue in block in dispatch' Execution of execute_shell_command failed: Stomp::Error::NoCurrentConnection
E, [2015-09-15T11:25:29.779247 #7925] ERROR -- : agents.rb:139:in `rescue in block in dispatch' /usr/lib/ruby/vendor_ruby/stomp/connection.rb:323:in `publish'
                /usr/share/mcollective/plugins/mcollective/connector/rabbitmq.rb:260:in `publish'
                /usr/lib/ruby/vendor_ruby/mcollective/message.rb:223:in `publish'
                /usr/lib/ruby/vendor_ruby/mcollective/runner.rb:131:in `reply'
                /usr/lib/ruby/vendor_ruby/mcollective/runner.rb:85:in `block in agentmsg'
                /usr/lib/ruby/vendor_ruby/mcollective/agents.rb:132:in `block (2 levels) in dispatch'
                /usr/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
                /usr/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'