tripleo

mistral.actions.action_factory.GetOvercloudConfig : string longer than 2147483647 bytes

Bug #1796436 reported by Harald Jensås on 2018-10-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Harald Jensås	tripleo stein-2

Bug Description

When deploying TLS everywhere overcloud mistral action fails:
The action raised an exception [action_ex_id=8cfcdeae-e509-4094-bf88-3ea1d763d83f, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{u'container_config': u'overcloud-config', u'container': u'overcloud'}']
string longer than 2147483647 bytes

executor.log:
-------------
2018-10-06 02:52:40.396 6 WARNING mistral.executors.default_executor [req-22e26344-82e7-425d-b8b9-2ef2afc1a848 228d0f3783c54e9faef847f71622453f b7ef212959e2439f89bdef23168e95b3 - default default] The action raised an exception [action_ex_id=8cfcdeae-e509-4094-bf88-3ea1d763d83f, action_cls='<class 'mistral.actions.action_factory.GetOvercloudConfig'>', attributes='{}', params='{u'container_config': u'overcloud-config', u'container': u'overcloud'}']
string longer than 2147483647 bytes: OverflowError: string longer than 2147483647 bytes
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor Traceback (most recent call last):
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/mistral/executors/default_executor.py", line 114, in run_action
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor result = action.run(action_ctx)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/tripleo_common/actions/config.py", line 93, in run
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor t.read())
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1878, in put_object
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor response_dict=response_dict)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1722, in _retry
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor service_token=self.service_token, **kwargs)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1356, in put_object
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor conn.request('PUT', path, contents, headers)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 444, in request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor files=files, **self.requests_args)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 427, in _request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor return self.request_session.request(*arg, **kwarg)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 512, in request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor resp = self.send(prep, **send_kwargs)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 622, in send
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor r = adapter.send(request, **kwargs)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 445, in send
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor timeout=timeout
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor chunked=chunked)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 356, in _make_request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor conn.request(method, url, **httplib_request_kw)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/httplib.py", line 1041, in request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor self._send_request(method, url, body, headers)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor self.endheaders(body)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor self._send_output(message_body)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor self.send(msg)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/httplib.py", line 857, in send
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor self.sock.sendall(data)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 171, in sendall
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor v = self.send(data_to_send)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 146, in send
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor super(GreenSSLSocket, self).send, data, flags)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor return func(*a, **kw)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor File "/usr/lib64/python2.7/ssl.py", line 710, in send
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor v = self._sslobj.write(data)
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor OverflowError: string longer than 2147483647 bytes
2018-10-06 02:52:40.396 6 ERROR mistral.executors.default_executor

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2018-10-07:

Lowering to Medium. After switching my libvirt undercloud vm back to a snapshot created prior to a number of stack create/stack delete cycles I no longer see this issue.

Changed in tripleo:
importance:	High → Medium

Revision history for this message

Brent Eagles (beagles) wrote on 2018-10-16:

I hit this issue and one of the following seemed to resolve it (I did both without checking if it resolved it)

1. restarted the mistral containers
2. deleted the overcloud-config and overcloud swift containers

I suspect it was the latter that really did the trick.

Revision history for this message

Juan Antonio Osorio Robles (juan-osorio-robles) wrote on 2018-10-19:

Should we keep this issue open then?

Revision history for this message

Brent Eagles (beagles) wrote on 2018-10-24:

I think so. It has happened to me twice the month.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-24: Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/613155

Changed in tripleo:
assignee:	nobody → Harald Jensås (harald-jensas)
status:	Triaged → In Progress

Revision history for this message

Harald Jensås (harald-jensas) wrote on 2018-10-24:

Not sure about that fix, I wrote it when I first hit this issue but then did'nt test it since I did'nt see the issue again.

Juan Antonio Osorio Robles (juan-osorio-robles) on 2018-10-30

Changed in tripleo:
milestone:	stein-1 → stein-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-07: Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/613155
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=6cc63952569c2b3b8573bc90586c090a2373040b
Submitter: Zuul
Branch: master

commit 6cc63952569c2b3b8573bc90586c090a2373040b
Author: Harald Jensås <email address hidden>
Date: Sat Oct 6 03:52:34 2018 +0200

Fix - config-download tarball upload OverflowError

    Pass the file object to swift client instead of reading it.
    When the object passed have a 'read' method the client will
    write data in chunks. Chunk size defaults to 65536.

Closes-Bug: #1796436
Change-Id: Ibdfa1019ebf291f30180fc2257d225e96c3306aa

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-18: Fix included in openstack/tripleo-common 10.2.0

This issue was fixed in the openstack/tripleo-common 10.2.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.