Monitor logs full of cloud meta-data connection errors

Bug #1216689 reported by Andreas Hasenack
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Client
Fix Released
Low
Chad Smith

Bug Description

Every 15min:
2013-08-25 18:33:07,321 ERROR [MainThread] Got error while fetching meta-data: <PyCurlError args=(7, 'couldn't connect to host')>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 207, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/landscape/lib/fetch.py", line 98, in fetch
    raise PyCurlError(e.args[0], e.args[1])
PyCurlError: Error 7: couldn't connect to host
2013-08-25 18:48:07,391 INFO [MainThread] Got notification of impending exchange. Notifying all plugins.
2013-08-25 18:48:07,717 INFO [MainThread] Queueing a message with updated data watcher info for landscape.monitor.activeprocessinfo.ActiveProcessInfo.
2013-08-25 18:48:07,720 INFO [MainThread] Queueing url fetch http://169.254.169.254/latest/meta-data/instance-id.
2013-08-25 18:48:10,719 ERROR [MainThread] Got error while fetching meta-data: <PyCurlError args=(7, 'couldn't connect to host')>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 207, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/landscape/lib/fetch.py", line 98, in fetch
    raise PyCurlError(e.args[0], e.args[1])
PyCurlError: Error 7: couldn't connect to host
2013-08-25 19:03:10,602 INFO [MainThread] Got notification of impending exchange. Notifying all plugins.
2013-08-25 19:03:11,063 INFO [MainThread] Queueing a message with updated data watcher info for landscape.monitor.activeprocessinfo.ActiveProcessInfo.
2013-08-25 19:03:11,065 INFO [MainThread] Queueing url fetch http://169.254.169.254/latest/meta-data/instance-id.
2013-08-25 19:03:14,063 ERROR [MainThread] Got error while fetching meta-data: <PyCurlError args=(7, 'couldn't connect to host')>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 207, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/landscape/lib/fetch.py", line 98, in fetch
    raise PyCurlError(e.args[0], e.args[1])
PyCurlError: Error 7: couldn't connect to host

I think the first time the error happens per lifetime of a process, we should log that it failed and not backtrace. And hide the subsequent attempts, unless it starts working, then we log that it started working.

We should definitely not backtrace.

Tags: defect

Related branches

tags: removed: kanban
Chad Smith (chad.smith)
Changed in landscape-client:
assignee: nobody → Chad Smith (chad.smith)
status: New → In Progress
Changed in landscape-client:
milestone: 13.08 → 13.09
Chad Smith (chad.smith)
Changed in landscape-client:
status: In Progress → Fix Committed
David Britton (dpb)
Changed in landscape-client:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.