Periodic task reports Manifest exceeds maximum allowed size (524288 bytes)

Bug #1563248 reported by Eli Qiao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Magnum
Confirmed
Medium
Eli Qiao

Bug Description

Mesos monitor reports
2016-03-29 17:54:27.472 TRACE magnum.service.periodic Traceback (most recent call last):
2016-03-29 17:54:27.472 TRACE magnum.service.periodic File "/opt/stack/magnum/magnum/service/periodic.py", line 160, in _send_bay_metrics
2016-03-29 17:54:27.472 TRACE magnum.service.periodic monitor.pull_data()
2016-03-29 17:54:27.472 TRACE magnum.service.periodic File "/opt/stack/magnum/magnum/conductor/mesos_monitor.py", line 46, in pull_data
2016-03-29 17:54:27.472 TRACE magnum.service.periodic master = jsonutils.loads(urlfetch.get(mesos_master_url))
2016-03-29 17:54:27.472 TRACE magnum.service.periodic File "/opt/stack/magnum/magnum/common/urlfetch.py", line 80, in get
2016-03-29 17:54:27.472 TRACE magnum.service.periodic " bytes)" % cfg.CONF.max_manifest_size)
2016-03-29 17:54:27.472 TRACE magnum.service.periodic URLFetchError: Manifest exceeds maximum allowed size (524288 bytes)

Manually get data from http://$Master:5050/state, it's 592840 bytes

Eli Qiao (taget-9)
Changed in magnum:
assignee: nobody → Eli Qiao (taget-9)
Revision history for this message
Eli Qiao (taget-9) wrote :

I this can be reproduced after we created Mesos bay sometime(with apps running on it).

I wonder if we should increase cfg.CONF.max_manifest_size(seems this won't be root cause)

or

modify the status check url, currently the url is $master_ip:5050/state

Changed in magnum:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Eli Qiao (taget-9) wrote :
Download full text (662.6 KiB)

{u'build_user': u'root', u'pid': u'master@192.168.0.6:5050', u'build_time': 1444683448.0, u'cluster': u'mesosbay', u'unregistered_frameworks': [], u'id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6', u'git_sha': u'2dd7f7ee115fe00b8e098b0a10762a4fa8f4600f', u'build_date': u'2015-10-12 20:57:28', u'hostname': u'192.168.0.6', u'version': u'0.25.0', u'log_dir': u'/var/log/mesos', u'orphan_tasks': [], u'deactivated_slaves': 0.0, u'start_time': 1459304739.06778, u'git_tag': u'0.25.0', u'completed_frameworks': [], u'elected_time': 1459304755.46932, u'leader': u'master@192.168.0.6:5050', u'activated_slaves': 1.0, u'frameworks': [{u'tasks': [{u'executor_id': u'', u'name': u'1', u'framework_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-0000', u'state': u'TASK_RUNNING', u'statuses': [{u'timestamp': 1459389557.24362, u'state': u'TASK_RUNNING', u'labels': [{u'value': u'', u'key': u'Docker.NetworkSettings.IPAddress'}], u'container_status': {u'network_infos': [{u'ip_address': u''}]}}], u'slave_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-S0', u'id': u'1.2c42cd87-f6e4-11e5-9b3a-02428c424452', u'resources': {u'mem': 16.0, u'disk': 0, u'cpus': 0.1, u'ports': u'[31717-31717]'}}, {u'executor_id': u'', u'name': u'1', u'framework_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-0000', u'state': u'TASK_RUNNING', u'statuses': [{u'timestamp': 1459389504.99605, u'state': u'TASK_RUNNING', u'labels': [{u'value': u'', u'key': u'Docker.NetworkSettings.IPAddress'}], u'container_status': {u'network_infos': [{u'ip_address': u''}]}}], u'slave_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-S0', u'id': u'1.0d4c51d6-f6e4-11e5-9b3a-02428c424452', u'resources': {u'mem': 16.0, u'disk': 0, u'cpus': 0.1, u'ports': u'[31121-31121]'}}, {u'executor_id': u'', u'name': u'2', u'framework_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-0000', u'state': u'TASK_RUNNING', u'statuses': [{u'timestamp': 1459389018.5079, u'state': u'TASK_RUNNING', u'labels': [{u'value': u'', u'key': u'Docker.NetworkSettings.IPAddress'}], u'container_status': {u'network_infos': [{u'ip_address': u''}]}}], u'slave_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-S0', u'id': u'2.eb35038c-f6e2-11e5-9b3a-02428c424452', u'resources': {u'mem': 16.0, u'disk': 0, u'cpus': 0.1, u'ports': u'[31940-31940]'}}], u'name': u'marathon', u'executors': [], u'used_resources': {u'mem': 48.0, u'disk': 0, u'cpus': 0.3, u'ports': u'[31121-31121, 31717-31717, 31940-31940]'}, u'webui_url': u'http://192.168.0.6:8080', u'hostname': u'192.168.0.6', u'pid': u'scheduler-6b9d912a-d2fb-4d56-9141-1b4c3fe8d73c@192.168.0.6:45316', u'capabilities': [], u'checkpoint': True, u'offers': [], u'failover_timeout': 604800.0, u'completed_tasks': [{u'executor_id': u'', u'name': u'2', u'framework_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-0000', u'state': u'TASK_FINISHED', u'statuses': [{u'timestamp': 1459339891.05181, u'state': u'TASK_RUNNING', u'labels': [{u'value': u'', u'key': u'Docker.NetworkSettings.IPAddress'}], u'container_status': {u'network_infos': [{u'ip_address': u''}]}}, {u'timestamp': 1459340890.985, u'state': u'TASK_FINISHED', u'container_status': {u'network_infos': [{u'ip_address': u'192.168.0.7'}]}}], u'slave_id': u'482e4894-6b6a-4d79-be92-2fa20ab961e6-S0', u'id': u'2...

Revision history for this message
Eli Qiao (taget-9) wrote :

These mess of information is from

`frameworks` field

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.