[SRU] Alarms fail on Rocky

Bug #1799406 reported by Liam Young
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
Fix Released
Undecided
Corey Bryant
OpenStack AODH Charm
Invalid
Undecided
Unassigned
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Rocky
Fix Released
Undecided
Unassigned
oslo.i18n
Confirmed
Medium
Unassigned
aodh (Ubuntu)
Fix Released
Undecided
Unassigned
Cosmic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
In a rocky deployment aodh is failing to raise alarms due to what seems like a locale issue. If I purge packages:

apt-get purge aodh-api aodh-evaluator aodh-expirer aodh-notifier aodh-listener python3-aodh libapache2-mod-wsgi-py3 python-apt python-aodh python-memcache

Then disable the AodhCharmRocky class and rerun install so that it pulls in the py2 packages then things start working.

The error from /var/log/aodh/aodh-listener.log:

http://paste.ubuntu.com/p/FCft2GGnfN/

ERROR oslo_db.sqlalchemy.exc_filters [-] DB exception wrapped.: AttributeError: 'list' object has no attribute 'lower'
ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
ERROR oslo_db.sqlalchemy.exc_filters context)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
ERROR oslo_db.sqlalchemy.exc_filters query = self.mogrify(query, args)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
ERROR oslo_db.sqlalchemy.exc_filters query = query % self._escape_args(args, conn)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
ERROR oslo_db.sqlalchemy.exc_filters return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
ERROR oslo_db.sqlalchemy.exc_filters return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
ERROR oslo_db.sqlalchemy.exc_filters return self.escape(obj, self.encoders)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
ERROR oslo_db.sqlalchemy.exc_filters return "'" + self.escape_string(obj) + "'"
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
ERROR oslo_db.sqlalchemy.exc_filters return converters.escape_string(s)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
ERROR oslo_db.sqlalchemy.exc_filters return value.translate(_escape_table)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
ERROR oslo_db.sqlalchemy.exc_filters self.has_plural_form)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
ERROR oslo_db.sqlalchemy.exc_filters fallback=True)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.6/gettext.py", line 525, in translation
ERROR oslo_db.sqlalchemy.exc_filters mofiles = find(domain, localedir, languages, all=True)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.6/gettext.py", line 482, in find
ERROR oslo_db.sqlalchemy.exc_filters for nelang in _expand_lang(lang):
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
ERROR oslo_db.sqlalchemy.exc_filters loc = locale.normalize(loc)
ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python3.6/locale.py", line 397, in normalize
ERROR oslo_db.sqlalchemy.exc_filters code = localename.lower()
ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'list' object has no attribute 'lower'
ERROR oslo_db.sqlalchemy.exc_filters
ERROR aodh.evaluator [-] alarm state update failed: oslo_db.exception.DBError: 'list' object has no attribute 'lower'
ERROR aodh.evaluator Traceback (most recent call last):
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
ERROR aodh.evaluator context)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
ERROR aodh.evaluator cursor.execute(statement, parameters)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
ERROR aodh.evaluator query = self.mogrify(query, args)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
ERROR aodh.evaluator query = query % self._escape_args(args, conn)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
ERROR aodh.evaluator return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
ERROR aodh.evaluator return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
ERROR aodh.evaluator return self.escape(obj, self.encoders)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
ERROR aodh.evaluator return "'" + self.escape_string(obj) + "'"
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
ERROR aodh.evaluator return converters.escape_string(s)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
ERROR aodh.evaluator return value.translate(_escape_table)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
ERROR aodh.evaluator self.has_plural_form)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
ERROR aodh.evaluator fallback=True)
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 525, in translation
ERROR aodh.evaluator mofiles = find(domain, localedir, languages, all=True)
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 482, in find
ERROR aodh.evaluator for nelang in _expand_lang(lang):
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
ERROR aodh.evaluator loc = locale.normalize(loc)
ERROR aodh.evaluator File "/usr/lib/python3.6/locale.py", line 397, in normalize
ERROR aodh.evaluator code = localename.lower()
ERROR aodh.evaluator AttributeError: 'list' object has no attribute 'lower'
ERROR aodh.evaluator
ERROR aodh.evaluator The above exception was the direct cause of the following exception:
ERROR aodh.evaluator
ERROR aodh.evaluator Traceback (most recent call last):
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/aodh/evaluator/__init__.py", line 128, in _refresh
ERROR aodh.evaluator self._storage_conn.update_alarm(alarm)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/aodh/storage/impl_sqlalchemy.py", line 273, in update_alarm
ERROR aodh.evaluator alarm.as_dict())
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 3426, in update
ERROR aodh.evaluator update_op.exec_()
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1326, in exec_
ERROR aodh.evaluator self._do_exec()
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1493, in _do_exec
ERROR aodh.evaluator self._execute_stmt(update_stmt)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/orm/persistence.py", line 1333, in _execute_stmt
ERROR aodh.evaluator mapper=self.mapper)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1176, in execute
ERROR aodh.evaluator bind, close_with_result=True).execute(clause, params or {})
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
ERROR aodh.evaluator return meth(self, multiparams, params)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
ERROR aodh.evaluator return connection._execute_clauseelement(self, multiparams, params)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
ERROR aodh.evaluator compiled_sql, distilled_params
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
ERROR aodh.evaluator context)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
ERROR aodh.evaluator util.raise_from_cause(newraise, exc_info)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
ERROR aodh.evaluator reraise(type(exception), exception, tb=exc_tb, cause=cause)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
ERROR aodh.evaluator raise value.with_traceback(tb)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
ERROR aodh.evaluator context)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 508, in do_execute
ERROR aodh.evaluator cursor.execute(statement, parameters)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 163, in execute
ERROR aodh.evaluator query = self.mogrify(query, args)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 142, in mogrify
ERROR aodh.evaluator query = query % self._escape_args(args, conn)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in _escape_args
ERROR aodh.evaluator return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 122, in <genexpr>
ERROR aodh.evaluator return dict((key, conn.literal(val)) for (key, val) in args.items())
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 819, in literal
ERROR aodh.evaluator return self.escape(obj, self.encoders)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 806, in escape
ERROR aodh.evaluator return "'" + self.escape_string(obj) + "'"
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 825, in escape_string
ERROR aodh.evaluator return converters.escape_string(s)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/pymysql/converters.py", line 73, in _escape_unicode
ERROR aodh.evaluator return value.translate(_escape_table)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 85, in translate
ERROR aodh.evaluator self.has_plural_form)
ERROR aodh.evaluator File "/usr/lib/python3/dist-packages/oslo_i18n/_message.py", line 117, in _translate_msgid
ERROR aodh.evaluator fallback=True)
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 525, in translation
ERROR aodh.evaluator mofiles = find(domain, localedir, languages, all=True)
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 482, in find
ERROR aodh.evaluator for nelang in _expand_lang(lang):
ERROR aodh.evaluator File "/usr/lib/python3.6/gettext.py", line 206, in _expand_lang
ERROR aodh.evaluator loc = locale.normalize(loc)
ERROR aodh.evaluator File "/usr/lib/python3.6/locale.py", line 397, in normalize
ERROR aodh.evaluator code = localename.lower()
ERROR aodh.evaluator oslo_db.exception.DBError: 'list' object has no attribute 'lower'
ERROR aodh.evaluator

[Test Case]
1) Deploy openstack with aodh (we're deploying with the development versions of the openstack charms which deploy py3 payloads)
2) Once deployed/configured create an alarm with:
openstack alarm create \
  --type event \
  --name instance_off \
  --description 'Instance powered OFF' \
  --event-type "compute.instance.power_off.*" \
  --enable True \
  --query "traits.instance_id=string::INSTANCE_ID" \
  --alarm-action 'log://' \
  --ok-action 'log://' \
  --insufficient-data-action 'log://'
3) create a server instance
4) power server instance off to trigger alarm
5) ensure alarm is successfully raised with no traceback (see /var/log/aodh/aodh-listener.log)

[Regression Potential]
The fix is a minimal fix that should result in low regression potential.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The problem looks to be that pymysql/converters.py is calling oslo_i18n's translate() method with a list argument (named _escape_table). translate expects desired_locale (string) to be passed in, not a list.

1) The call to translate is in /usr/lib/python3/dist-packages/pymysql/converters.py:
     value.translate(_escape_table).

2) This calls into the translate() function in /usr/lib/python3/dist-packages/oslo_i18n/_message.py

3) Looking at the argument that gets passed to 'def translate(self, desired_locale=None)' looks like the _escape_table:

desired_locale=['\\0', '\x01', '\x02', '\x03', '\x04', '\x05', '\x06', '\x07', '\x08', '\t', '\\n', '\x0b', '\x0c', '\\r', '\x0e', '\x0f', '
\x10', '\x11', '\x12', '\x13', '\x14', '\x15', '\x16', '\x17', '\x18', '\x19', '\\Z', '\x1b', '\x1c', '\x1d', '\x1e', '\x1f', ' ', '!', '\\"', '#', '$', '%', '&', "\\'", '(', ')', '*', '+', ',', '-', '.',
 '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W'
, 'X', 'Y', 'Z', '[', '\\\\', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~', '\x7
f']

4) languages ends up as a nested list in /usr/lib/python3/dist-packages/oslo_i18n/_message.py:
    lang = gettext.translation(domain,
                               localedir=locale_dir,
                               languages=[desired_locale],
                               fallback=True)

5) languages are looped through in /usr/lib/python3.6/gettext.py:
  for lang in languages:
      for nelang in _expand_lang(lang):

6) _expand_lang calls local.normalize with a list (_escape_table) and that causes the "AttributeError: 'list' object has no attribute 'lower'":
  locale.normalize(loc)

The normalize() call should instead be getting called with a locale string, something such as: locale.normalize('EN_US.utf-8').

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Please ignore comment #2 (hidden).

Revision history for this message
Corey Bryant (corey.bryant) wrote :

# Sample for recreating
#
# py2 results: https://paste.ubuntu.com/p/JrDMDr9yMk/
# py3 results: https://paste.ubuntu.com/p/n2zt3Bw3VJ/

import locale
import pymysql

class Message():
    def translate(self, desired_locale=None):
        print("translate(desired_local={})".format(desired_locale))
        locale.normalize(desired_locale)

conn = pymysql.connect(user='aodh', password='pw', host='1.2.3.4')
cur = conn.cursor()
cur.execute('SELECT %(var)s', dict(var=99, msg=Message()))

Revision history for this message
Corey Bryant (corey.bryant) wrote :

I'm having a hard time determining where the problem is. Adding upstream aodh, oslo.db, and oslo.messaging to see if anyone else can help.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Part of the reason why this only occurs on py3 is because the code path is different in pymysql for py2 vs py3 (see lines 68-102):

https://github.com/PyMySQL/PyMySQL/blob/v0.9.2/pymysql/converters.py#L68

Revision history for this message
Corey Bryant (corey.bryant) wrote :

A bit more details traceback: https://paste.ubuntu.com/p/PWhMqKM5Ht/

Revision history for this message
Corey Bryant (corey.bryant) wrote :

/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py
-----------------------------------------------------------
do_execute, line 508: cursor.execute(statement, parameters)

statement=UPDATE alarm SET alarm_id=%(alarm_id)s, enabled=%(enabled)s, name=%(name)s, type=%(type)s, severity=%(severity)s, description=%(description)s, timestamp=%(timestamp)s, user_id=%(user_id)s, project_id=%(project_id)s, state=%(state)s, state_reason=%(state_reason)s, state_timestamp=%(state_timestamp)s, ok_actions=%(ok_actions)s, alarm_actions=%(alarm_actions)s, insufficient_data_actions=%(insufficient_data_actions)s, repeat_actions=%(repeat_actions)s, rule=%(rule)s, time_constraints=%(time_constraints)s WHERE alarm.alarm_id = %(alarm_id_1)s

parameters={'alarm_id': 'cd47c788-58df-4604-b21e-565d4055f872', 'enabled': 1, 'name': 'cpu_1', 'type': 'event', 'severity': 'low', 'description': 'Alarm when * event occurred.', 'timestamp': datetime.datetime(2018, 10, 24, 12, 52, 55, 651317), 'user_id': 'feebe0e4f0264bebb53b1c2468a9da77', 'project_id': 'bc679e36e26b4640b1392e524d176ec2', 'state': 'alarm', 'state_reason': 'Event <id=fb7bf120-bacf-4972-88b2-c4a750098d67,event_type=compute.instance.power_off.start> hits the query <query=[]>.', 'state_timestamp': datetime.datetime(2018, 10, 24, 12, 52, 55, 651317), 'ok_actions': '[]', 'alarm_actions': '[]', 'insufficient_data_actions': '[]', 'repeat_actions': 0, 'rule': '{"event_type": "*", "query": []}', 'time_constraints': '[]', 'alarm_id_1': 'cd47c788-58df-4604-b21e-565d4055f872'}

aodh/storage/models.py (parameters above maps to class Alarm)
-------------------------------------------------------------

import datetime

from aodh.i18n import _
from aodh.storage import base

class Alarm(base.Model):
    """
    An alarm to monitor.

    :param alarm_id: UUID of the alarm
    :param type: type of the alarm
    :param name: The Alarm name
    :param description: User friendly description of the alarm
    :param enabled: Is the alarm enabled
    :param state: Alarm state (ok/alarm/insufficient data)
    :param state_reason: Alarm state reason
    :param rule: A rule that defines when the alarm fires
    :param user_id: the owner/creator of the alarm
    :param project_id: the project_id of the creator
    :param evaluation_periods: the number of periods
    :param period: the time period in seconds
    :param time_constraints: the list of the alarm's time constraints, if any
    :param timestamp: the timestamp when the alarm was last updated
    :param state_timestamp: the timestamp of the last state change
    :param ok_actions: the list of webhooks to call when entering the ok state
    :param alarm_actions: the list of webhooks to call when entering the
                          alarm state
    :param insufficient_data_actions: the list of webhooks to call when
                                      entering the insufficient data state
    :param repeat_actions: Is the actions should be triggered on each
                           alarm evaluation.
    :param severity: Alarm level (low/moderate/critical)
    """

Revision history for this message
Corey Bryant (corey.bryant) wrote :

parameters= map to the Alarm class variables above

Revision history for this message
Corey Bryant (corey.bryant) wrote :

/usr/lib/python3/dist-packages/pymysql/connections.py (note translate in dir(obj) output)
-----------------------------------------------------------------------------------------
line 806, in escape

806: return "'" + self.escape_string(obj) + "'"

obj=event
dir(obj)=['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', __subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The failure is caused when _escape_unicode(value) [1] is called with the following (it expects a str for py3):

value=Event <id=fa2a8295-e609-48f2-af6e-13c3219d1a9b,event_type=compute.instance.power_on.start> hits the query <query=[]>.

type(value)=<class 'oslo_i18n._message.Message'>

[1] https://github.com/PyMySQL/PyMySQL/blob/v0.9.2/pymysql/converters.py#L68

I can't seem to figure out where that value is coming from.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Ok a little progress, value:

value=Event <id=fa2a8295-e609-48f2-af6e-13c3219d1a9b,event_type=compute.instance.power_on.start> hits the query <query=[]>.

is coming from aodh/evaluator/event.py:

    def _fire_alarm(self, alarm, event):
        """Update alarm state and fire alarm via alarm notifier."""

        state = evaluator.ALARM
        reason = (_('Event <id=%(id)s,event_type=%(event_type)s> hits the '
                    'query <query=%(alarm_query)s>.') %
                  {'id': event.id,
                   'event_type': event.get_value('event_type'),
                   'alarm_query': json.dumps(alarm.obj.rule['query'],
                                             sort_keys=True)})
        reason_data = {'type': 'event', 'event': event.obj}
        always_record = alarm.obj.repeat_actions
        self._refresh(alarm.obj, state, reason, reason_data, always_record)

Revision history for this message
Corey Bryant (corey.bryant) wrote :

where:

from aodh.i18n import _

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Confirmed that changing:

reason = (_('Event <id=%(id)s,event_type=%(event_type)s> hits the '
                    'query <query=%(alarm_query)s>.') %
                  {'id': event.id,
                   'event_type': event.get_value('event_type'),
                   'alarm_query': json.dumps(alarm.obj.rule['query'],
                                             sort_keys=True)})

to:

reason = (('Event <id=%(id)s,event_type=%(event_type)s> hits the '
                    'query <query=%(alarm_query)s>.') %
                  {'id': event.id,
                   'event_type': event.get_value('event_type'),
                   'alarm_query': json.dumps(alarm.obj.rule['query'],
                                             sort_keys=True)})

prevents the error. The call to escape_unicode(value) passes value with type(value)=<class 'str'>.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Is this a problem with oslo.i18n? The oslo_i18n._message.Message class defines translate() as:

  translate(self, desired_locale=None)

However, the Message class inherits from six.text_type which also has a translate() method but it takes a table (and optional deletechars) argument(s) rather than a locale:

  py2: str.translate(table[, deletechars])
  py3: str.translate(table)

  https://docs.python.org/2/library/stdtypes.html#str.translate
  https://docs.python.org/3/library/stdtypes.html#str.translate

Is this an unfortunate naming of the Message.translate() method or is it actually meant to override the base class method with different argument types?

From oslo_i18n/_message.py:
---------------------------

class Message(six.text_type):
    """A Message object is a unicode object that can be translated.
    ...

    def translate(self, desired_locale=None):
    """Translate this message to the desired locale.

no longer affects: oslo.messaging
no longer affects: oslo.db
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to aodh (master)

Fix proposed to branch: master
Review: https://review.openstack.org/613656

Changed in aodh:
assignee: nobody → Corey Bryant (corey.bryant)
status: New → In Progress
description: updated
summary: - Alarms fail on Rocky
+ [SRU] Alarms fail on Rocky
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Liam, or anyone else affected,

Accepted aodh into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/aodh/7.0.0-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-cosmic
Changed in aodh (Ubuntu Cosmic):
status: New → Fix Committed
Changed in aodh (Ubuntu):
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to aodh (master)

Reviewed: https://review.openstack.org/613656
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=dbf6d2770dcd97d2ef2939ca735309e44b946c4d
Submitter: Zuul
Branch: master

commit dbf6d2770dcd97d2ef2939ca735309e44b946c4d
Author: Corey Bryant <email address hidden>
Date: Fri Oct 26 13:58:56 2018 -0400

    PY3: Ensure _refresh is passed str type for reason

    Drop the use of oslo.i18n's Message class when defining the reason
    argument that is passed to _refresh(). Ensuing code that _refresh()
    uses, in particular pymysql [1], expects a unicode str type rather
    than an oslo_i18n._message.Message type.

    [1] https://github.com/PyMySQL/PyMySQL/blob/v0.9.2/pymysql/converters.py#L68

    Change-Id: Icc59edc8a91ba7ae6929e853f1bf09bd25cdc039
    Closes-Bug: #1799406

Changed in aodh:
status: In Progress → Fix Released
Changed in cloud-archive:
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to aodh (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/616263

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Verified successfully on rocky-proposed: https://paste.ubuntu.com/p/z83B3DDHHW/

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Verified successfully on cosmic-proposed: https://paste.ubuntu.com/p/HsPFYHFsYr/

tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
tags: added: verification-rocky-done
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package aodh - 7.0.0-0ubuntu5

---------------
aodh (7.0.0-0ubuntu5) disco; urgency=medium

  * d/tests/aodh-shebangs-py3: Update shebang for Python 3.7.

 -- Corey Bryant <email address hidden> Wed, 07 Nov 2018 09:22:58 -0500

Changed in aodh (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package aodh - 7.0.0-0ubuntu4

---------------
aodh (7.0.0-0ubuntu4) cosmic; urgency=medium

  * d/p/ensure-refresh-gets-str-type.patch: Cherry-picked from upstream
    (https://review.openstack.org/#/c/613656/) to enable successful
    alarms in py3 (LP: #1799406).

 -- Corey Bryant <email address hidden> Fri, 26 Oct 2018 14:16:13 -0400

Changed in aodh (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for aodh has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package aodh - 7.0.0-0ubuntu4~cloud0
---------------

 aodh (7.0.0-0ubuntu4~cloud0) bionic-rocky; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 aodh (7.0.0-0ubuntu4) cosmic; urgency=medium
 .
   * d/p/ensure-refresh-gets-str-type.patch: Cherry-picked from upstream
     (https://review.openstack.org/#/c/613656/) to enable successful
     alarms in py3 (LP: #1799406).

Changed in cloud-archive:
status: Fix Committed → Fix Released
Liam Young (gnuoy)
Changed in charm-aodh:
status: New → Invalid
Changed in oslo.i18n:
status: New → Invalid
Revision history for this message
Ben Nemec (bnemec) wrote :

I don't think this is invalid on oslo.i18n. It sounds like there's a problem with the Message class that caused this. The Aodh fix is only a workaround for that.

There have been discussions around removing lazy translation completely, in which case this would go away. If that doesn't happen, we should look into how to fix this.

Changed in oslo.i18n:
status: Invalid → Confirmed
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-aodh (master)

Change abandoned by Ryan Beisner (<email address hidden>) on branch: master
Review: https://review.openstack.org/614797

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/aodh 8.0.0.0rc1

This issue was fixed in the openstack/aodh 8.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.