Kolla Watcher Service - Unable to determine the name of the local timezone

[Karl Kloppenborg]

==> /var/log/kolla/watcher/watcher-decision-engine.log <==
2019-03-26 00:39:17.510 7 INFO watcher.cmd.decisionengine [-] Starting Watcher Decision Engine service in PID 7
2019-03-26 00:39:17.598 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal saving_energy already exists
2019-03-26 00:39:17.599 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal dummy already exists
2019-03-26 00:39:17.600 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal noisy_neighbor already exists
2019-03-26 00:39:17.600 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal server_consolidation already exists
2019-03-26 00:39:17.600 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal thermal_optimization already exists
2019-03-26 00:39:17.601 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal airflow_optimization already exists
2019-03-26 00:39:17.601 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal unclassified already exists
2019-03-26 00:39:17.601 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal hardware_maintenance already exists
2019-03-26 00:39:17.601 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal cluster_maintaining already exists
2019-03-26 00:39:17.602 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Goal workload_balancing already exists
2019-03-26 00:39:17.610 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy saving_energy unchanged
2019-03-26 00:39:17.610 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy dummy unchanged
2019-03-26 00:39:17.611 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy storage_capacity_balance unchanged
2019-03-26 00:39:17.611 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy dummy_with_scorer unchanged
2019-03-26 00:39:17.612 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy outlet_temperature unchanged
2019-03-26 00:39:17.612 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy noisy_neighbor unchanged
2019-03-26 00:39:17.613 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy zone_migration unchanged
2019-03-26 00:39:17.613 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy vm_workload_consolidation unchanged
2019-03-26 00:39:17.614 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy basic unchanged
2019-03-26 00:39:17.615 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy actuator unchanged
2019-03-26 00:39:17.615 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy host_maintenance unchanged
2019-03-26 00:39:17.616 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy workload_stabilization unchanged
2019-03-26 00:39:17.617 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy workload_balance unchanged
2019-03-26 00:39:17.617 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy uniform_airflow unchanged
2019-03-26 00:39:17.618 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Strategy dummy_with_resize unchanged
2019-03-26 00:39:17.623 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Scoring Engine dummy_avg_scorer unchanged
2019-03-26 00:39:17.623 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Scoring Engine dummy_min_scorer unchanged
2019-03-26 00:39:17.624 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Scoring Engine dummy_max_scorer unchanged
2019-03-26 00:39:17.624 7 INFO watcher.decision_engine.sync [req-9059f84d-9aa6-4f5e-83c2-f5a4eae96797 - - - - -] Scoring Engine dummy_scorer unchanged
2019-03-26 00:39:17.729 7 CRITICAL python-watcher [req-8093a378-1eb0-4476-8d4e-09bf8ea1449f - - - - -] Unhandled error: ValueError: Unable to determine the name of the local timezone -- you must explicitly specify the name of the local timezone. Please refrain from using timezones like EST to prevent problems with daylight saving time. Instead, use a locale based timezone name (such as Europe/Helsinki).
2019-03-26 00:39:17.729 7 ERROR python-watcher Traceback (most recent call last):
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/bin/watcher-decision-engine", line 10, in <module>
2019-03-26 00:39:17.729 7 ERROR python-watcher sys.exit(main())
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/watcher/cmd/decisionengine.py", line 46, in main
2019-03-26 00:39:17.729 7 ERROR python-watcher de_service = watcher_service.Service(manager.DecisionEngineManager)
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/watcher/common/service.py", line 189, in __init__
2019-03-26 00:39:17.729 7 ERROR python-watcher ep(self) for ep in self.manager.conductor_endpoints
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/watcher/decision_engine/messaging/audit_endpoint.py", line 40, in __init__
2019-03-26 00:39:17.729 7 ERROR python-watcher self._continuous_handler = c_handler.ContinuousAuditHandler().start()
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/watcher/decision_engine/audit/continuous.py", line 199, in start
2019-03-26 00:39:17.729 7 ERROR python-watcher jobstore='memory')
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py", line 413, in add_job
2019-03-26 00:39:17.729 7 ERROR python-watcher 'trigger': self._create_trigger(trigger, trigger_args),
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py", line 907, in _create_trigger
2019-03-26 00:39:17.729 7 ERROR python-watcher return self._create_plugin_instance('trigger', trigger, trigger_args)
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/apscheduler/schedulers/base.py", line 892, in _create_plugin_instance
2019-03-26 00:39:17.729 7 ERROR python-watcher return plugin_cls(**constructor_kwargs)
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/apscheduler/triggers/interval.py", line 38, in __init__
2019-03-26 00:39:17.729 7 ERROR python-watcher self.timezone = astimezone(timezone)
2019-03-26 00:39:17.729 7 ERROR python-watcher File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/apscheduler/util.py", line 86, in astimezone
2019-03-26 00:39:17.729 7 ERROR python-watcher 'Unable to determine the name of the local timezone -- you must explicitly '
2019-03-26 00:39:17.729 7 ERROR python-watcher ValueError: Unable to determine the name of the local timezone -- you must explicitly specify the name of the local timezone. Please refrain from using timezones like EST to prevent problems with daylight saving time. Instead, use a locale based timezone name (such as Europe/Helsinki).
2019-03-26 00:39:17.729 7 ERROR python-watcher

[Mark Goddard (mgoddard)]

Which version of kolla/kolla-ansible? Which OS? Source/binary?

[Mark Goddard (mgoddard)]

We map /etc/localtime into the container. What timezone are you in?

[Marek Grudzinski (ivve)]

Hey, just tested this and can confirm the problem. Except for Watcher, Freezer is also affected by this issue.

OS: Ubuntu 18.04, Ubuntu 16.04
kolla_install_type: source
kolla_base_distro: ubuntu
openstack_release: stein
kolla-ansible: 8.0.0.0rc1
timezone:
timedatectl
      Local time: Tue 2019-05-21 20:35:01 CEST
  Universal time: Tue 2019-05-21 18:35:01 UTC
        RTC time: Tue 2019-05-21 18:35:01
       Time zone: Europe/Stockholm (CEST, +0200)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no

[Marek Grudzinski (ivve)]

Don't think this is related to timezone issues but configuration of Watcher.. I'll troubleshoot this and check back!

[Marek Grudzinski (ivve)]

Hi again, seems I was wrong. After diggin in some code in apscheduler it seems it needs /etc/timezone as well. I included it and services started.

[Marek Grudzinski (ivve)]

Not sure if this is the best way to solve it.. but for those who want a fix, this can be added after these lines:

      - "/etc/timezone:/etc/timezone:ro"

https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/watcher/defaults/main.yml#L12
https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/watcher/defaults/main.yml#L34
https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/watcher/defaults/main.yml#L45

[Mark Goddard (mgoddard)]

Hi Marek, thanks for debugging this issue. In most containers (including watcher) we map /etc/localtime:/etc/localtime:ro as a docker volume. It seems this is not sufficient in this case though.

Can you propose a fix for this issue?

[Marek Grudzinski (ivve)]

https://review.opendev.org/661068 Add missing timezone mount for watcher (apscheduler).

[Karl Kloppenborg (karlkloppenborg)]

Hi Mark,

I'm really sorry - I didn't get notified about your comments.

Let me know if I can help in any way.

[Mark Goddard (mgoddard)]

Fix proposed by Dincer: https://review.opendev.org/#/c/704258

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radosław Piliszek (<email address hidden>) on branch: master
Review: https://review.opendev.org/661068
Reason: obsoleted ^

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704258
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4b5df0d8660543230a48d279f508c3205c8fc406
Submitter: Zuul
Branch: master

commit 4b5df0d8660543230a48d279f508c3205c8fc406
Author: Dincer Celik <email address hidden>
Date: Sun Jan 26 23:05:13 2020 +0300

    Introduce /etc/timezone to Debian/Ubuntu containers

    Some services look for /etc/timezone on Debian/Ubuntu, so we should
    introduce it to the containers.

    In addition, added prechecks for /etc/localtime and /etc/timezone.

    Closes-Bug: #1821592
    Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/719035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/719035
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=37ffbe164bbb5ab0848947e65369950e9dfdcba9
Submitter: Zuul
Branch: stable/train

commit 37ffbe164bbb5ab0848947e65369950e9dfdcba9
Author: Dincer Celik <email address hidden>
Date: Sun Jan 26 23:05:13 2020 +0300

    Introduce /etc/timezone to Debian/Ubuntu containers

    Some services look for /etc/timezone on Debian/Ubuntu, so we should
    introduce it to the containers.

    In addition, added prechecks for /etc/localtime and /etc/timezone.

    Closes-Bug: #1821592
    Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3
    (cherry picked from commit 4b5df0d8660543230a48d279f508c3205c8fc406)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/723031

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723031
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d576249f35e9a62ec2a21e80853cb85ece4aa54a
Submitter: Zuul
Branch: stable/stein

commit d576249f35e9a62ec2a21e80853cb85ece4aa54a
Author: Dincer Celik <email address hidden>
Date: Fri Apr 24 23:19:36 2020 +0300

    Introduce /etc/timezone to Debian/Ubuntu containers

    Some services look for /etc/timezone on Debian/Ubuntu, so we should
    introduce it to the containers.

    In addition, added prechecks for /etc/localtime and /etc/timezone.

    Closes-Bug: #1821592
    Change-Id: I9fef14643d1bcc7eee9547eb87fa1fb436d8a6b3
    (cherry picked from commit 4b5df0d8660543230a48d279f508c3205c8fc406)