[contrail] Deployment has failed with 50 hw nodes

Bug #1495538 reported by Sergey Galkin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel Contrail plugin
Fix Released
Critical
Fuel Plugin Contrail

Bug Description

Steps to reproduce
1. Install Fuel 6.1
2. Install contrail plugin
3. Try to deploy cluster with 3 controller in HA, 43 computes and 3 OS nodes

Deployment has failed with error

Deployment has failed. Method granular_deploy. Failed to execute hook 'contrail-2.0.0' Failed to run command cd /etc/fuel/plugins/contrail-2.0/ && if hiera user_node_name | egrep '^contrail-[0-9]'; then puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp; fi

 File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
DataError: (DataError) value too long for type character varying(255)
 'UPDATE nodes SET status=%(status)s, error_type=%(error_type)s, error_msg=%(error_msg)s WHERE nodes.id = %(nodes_id)s' {'status': u'error', 'nodes_id': 88, 'error_type': u'deploy', 'error_msg': u"Failed to run command cd /etc/fuel/plugins/contrail-2.0/ && if hiera user_node_name | egrep '^contrail-[0-9]'; then puppet apply /etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp; fi\n\nTask: ab64b86e-9cbe-491f-8e2d-d6c8763073f9: shell timeout error: execution expired\nTask timeout: 720, Retries: 3"}

snapshot available in http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-14_12-15-53.tar.xz

Tags: scale
Sergey Galkin (sgalkin)
tags: added: scale
Changed in fuel-plugins:
assignee: nobody → Vladimir Sharshov (vsharshov)
Changed in fuel-plugins:
importance: Undecided → Critical
Andrian Noga (anoga)
affects: fuel-plugins → mos
Changed in mos:
milestone: none → 6.1-mu-2
Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Main problem on node-104. Because we run network reconfiguration in same time then run shell, we can get such error. I suggest to include shell condition in to puppet manifest and run it as puppet task. It is tolerant with temporary network problem.

More details:

D, [2015-09-15T11:25:29.778436 #7925] DEBUG -- : base.rb:168:in `create_reply' Encoded a message for request c44b655cadb0507998307e39bb7dc058
D, [2015-09-15T11:25:29.778648 #7925] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Rabbitmq
D, [2015-09-15T11:25:29.778828 #7925] DEBUG -- : rabbitmq.rb:258:in `publish' Sending a broadcast message to RabbitMQ target '/reply-queue/amq.gen-7FCnFPWA2vOnhPO89BvPVg' with headers '{"expiration"=>"4294967000"}'
E, [2015-09-15T11:25:29.779128 #7925] ERROR -- : agents.rb:138:in `rescue in block in dispatch' Execution of execute_shell_command failed: Stomp::Error::NoCurrentConnection
E, [2015-09-15T11:25:29.779247 #7925] ERROR -- : agents.rb:139:in `rescue in block in dispatch' /usr/lib/ruby/vendor_ruby/stomp/connection.rb:323:in `publish'
                /usr/share/mcollective/plugins/mcollective/connector/rabbitmq.rb:260:in `publish'
                /usr/lib/ruby/vendor_ruby/mcollective/message.rb:223:in `publish'
                /usr/lib/ruby/vendor_ruby/mcollective/runner.rb:131:in `reply'
                /usr/lib/ruby/vendor_ruby/mcollective/runner.rb:85:in `block in agentmsg'
                /usr/lib/ruby/vendor_ruby/mcollective/agents.rb:132:in `block (2 levels) in dispatch'
                /usr/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
                /usr/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'

Changed in mos:
status: New → Confirmed
assignee: Vladimir Sharshov (vsharshov) → Fuel Plugin Contrail (fuel-plugin-contrail)
Changed in mos:
status: Confirmed → Fix Committed
milestone: 6.1-mu-2 → none
affects: mos → fuel-plugins
Yevgeniy (yshapovalov)
affects: fuel-plugins → fuel-plugin-contrail
Yevgeniy (yshapovalov)
Changed in fuel-plugin-contrail:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.