| 2016-11-15 09:04:55 |
Victor Tapia |
bug |
|
|
added bug |
| 2016-11-15 11:02:47 |
Louis Bouchard |
nominated for series |
|
Ubuntu Zesty |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
bug task added |
|
sssd (Ubuntu Zesty) |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
bug task added |
|
sssd (Ubuntu Xenial) |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
nominated for series |
|
Ubuntu Trusty |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
bug task added |
|
sssd (Ubuntu Trusty) |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
nominated for series |
|
Ubuntu Yakkety |
|
| 2016-11-15 11:02:47 |
Louis Bouchard |
bug task added |
|
sssd (Ubuntu Yakkety) |
|
| 2016-11-22 16:20:55 |
Victor Tapia |
sssd (Ubuntu Trusty): assignee |
|
Victor Tapia (vtapia) |
|
| 2016-11-22 16:20:58 |
Victor Tapia |
sssd (Ubuntu Xenial): assignee |
|
Victor Tapia (vtapia) |
|
| 2016-11-22 16:21:00 |
Victor Tapia |
sssd (Ubuntu Yakkety): assignee |
|
Victor Tapia (vtapia) |
|
| 2016-11-22 16:21:02 |
Victor Tapia |
sssd (Ubuntu Zesty): assignee |
|
Victor Tapia (vtapia) |
|
| 2017-02-14 16:59:50 |
Victor Tapia |
description |
When SSSD fails to connect to a provider (LDAP, for instance) it creates a timed event with tevent_add_timer() in order to retry in ~1 min. Tevent relies on CLOCK_REALTIME, using absolute epoch time, so when the time changes (e.g. NTP sync) the scheduled event is affected.
Reproducer:
1. Modify the /etc/hosts file to force a failed resolution and restart sssd
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [resolv_gethostbyname_dns_query] (0x0100): Trying to resolve A record of 'ldap' in DNS
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [resolv_gethostbyname_done] (0x0040): querying hosts database failed [5]: Input/output error
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
...
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline]
2. Within the ~1 minute window, change the date (date --set) to a previous time (2 hour in my example). Note: if /etc/resolv.conf or a network interface is modified, SSSD providers will restart, but the scheduled retry will be kept.
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [recreate_ares_channel] (0x0100): Initializing new c-ares channel
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [recreate_ares_channel] (0x0100): Destroying the old c-ares channel
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'name not resolved'
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_set_port_status] (0x0100): Marking port 389 of server 'ldap' as 'neutral'
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
...
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'not working'
...
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline]
3. Once it gets to the programmed date (2 hours later), it retries:
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'name not resolved'
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_set_port_status] (0x0100): Marking port 389 of server 'ldap' as 'neutral'
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
...
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'not working'
...
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline] |
[Impact]
* SSSD is affected by clock shifts because it's built on libtevent, that doesn't use monotonic clocks.
* After an event has been scheduled, if the time drifts to the past SSSD won't recover and the event will have to wait the shifted time to be executed.
[Test Case]
* Modify the /etc/hosts file to force a failed resolution and restart sssd.
* Within the first ~1 minute retry window, change the date (date --set) to a previous time (e.g. 2h).
* "Tail" the log file. The event will be executed at the original schedule, past those 2h.
[Regression Potential]
* None
[Other Info]
* Version 1.15 implements a watchdog that detects time shifts and resets itselft but doesn't reschedule the scheduled events.
* There's a network monitor based on netlink that detects interface changes and restarts the providers, rescheduling the scheduled events.
* Such restart can be triggered with SIGUSR2. Sending that signal after the watchdog restarts fixes this issue.
* Upstream bug: https://fedorahosted.org/sssd/ticket/3285
[Original Description]
When SSSD fails to connect to a provider (LDAP, for instance) it creates a timed event with tevent_add_timer() in order to retry in ~1 min. Tevent relies on CLOCK_REALTIME, using absolute epoch time, so when the time changes (e.g. NTP sync) the scheduled event is affected.
Reproducer:
1. Modify the /etc/hosts file to force a failed resolution and restart sssd
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [resolv_gethostbyname_dns_query] (0x0100): Trying to resolve A record of 'ldap' in DNS
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [resolv_gethostbyname_done] (0x0040): querying hosts database failed [5]: Input/output error
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
...
(Tue Oct 25 09:37:14 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline]
2. Within the ~1 minute window, change the date (date --set) to a previous time (2 hour in my example). Note: if /etc/resolv.conf or a network interface is modified, SSSD providers will restart, but the scheduled retry will be kept.
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [recreate_ares_channel] (0x0100): Initializing new c-ares channel
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [recreate_ares_channel] (0x0100): Destroying the old c-ares channel
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'name not resolved'
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_set_port_status] (0x0100): Marking port 389 of server 'ldap' as 'neutral'
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
...
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'not working'
...
(Tue Oct 25 07:41:46 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline]
3. Once it gets to the programmed date (2 hours later), it retries:
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'name not resolved'
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_set_port_status] (0x0100): Marking port 389 of server 'ldap' as 'neutral'
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'LDAP'
...
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [fo_resolve_service_done] (0x0020): Failed to resolve server 'ldap': Could not contact DNS servers
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [set_server_common_status] (0x0100): Marking server 'ldap' as 'not working'
...
(Tue Oct 25 09:38:25 2016) [sssd[be[openstacklocal]]] [check_online_callback] (0x0100): Backend returned: (1, 0, <NULL>) [Provider is Offline] |
|
| 2017-02-15 16:28:52 |
Victor Tapia |
sssd (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2017-02-15 16:28:58 |
Victor Tapia |
sssd (Ubuntu Yakkety): importance |
Undecided |
Medium |
|
| 2017-02-15 16:29:02 |
Victor Tapia |
sssd (Ubuntu Zesty): importance |
Undecided |
Medium |
|
| 2017-02-15 16:29:03 |
Victor Tapia |
sssd (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2017-02-15 16:30:18 |
Victor Tapia |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2017-02-15 16:30:57 |
Victor Tapia |
tags |
sts |
sts sts-sponsor sts-sru |
|
| 2017-02-15 16:38:49 |
Victor Tapia |
attachment added |
|
zesty_sssd_1.15.0-3ubuntu3.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4819666/+files/zesty_sssd_1.15.0-3ubuntu3.debdiff |
|
| 2017-02-21 10:33:49 |
Louis Bouchard |
sssd (Ubuntu Zesty): status |
New |
In Progress |
|
| 2017-02-21 10:33:55 |
Louis Bouchard |
sssd (Ubuntu Yakkety): status |
New |
Triaged |
|
| 2017-02-21 10:33:59 |
Louis Bouchard |
sssd (Ubuntu Xenial): status |
New |
Triaged |
|
| 2017-02-21 10:34:02 |
Louis Bouchard |
sssd (Ubuntu Trusty): status |
New |
Triaged |
|
| 2017-02-21 10:35:26 |
Louis Bouchard |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2017-02-21 12:12:34 |
Launchpad Janitor |
sssd (Ubuntu Zesty): status |
In Progress |
Fix Released |
|
| 2017-02-22 15:01:44 |
Victor Tapia |
attachment added |
|
yakkety-sssd_1.13.4-3ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824488/+files/yakkety-sssd_1.13.4-3ubuntu1.debdiff |
|
| 2017-02-22 15:02:07 |
Victor Tapia |
attachment added |
|
xenial-sssd_1.13.4-1ubuntu1.2.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824489/+files/xenial-sssd_1.13.4-1ubuntu1.2.debdiff |
|
| 2017-02-22 15:02:27 |
Victor Tapia |
attachment added |
|
trusty-sssd_1.11.8-0ubuntu0.4.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824490/+files/trusty-sssd_1.11.8-0ubuntu0.4.debdiff |
|
| 2017-02-22 15:04:49 |
Victor Tapia |
attachment removed |
trusty-sssd_1.11.8-0ubuntu0.4.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824490/+files/trusty-sssd_1.11.8-0ubuntu0.4.debdiff |
|
|
| 2017-02-22 16:18:56 |
Victor Tapia |
attachment added |
|
trusty-sssd_1.11.8-0ubuntu0.4.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824602/+files/trusty-sssd_1.11.8-0ubuntu0.4.debdiff |
|
| 2017-02-23 10:51:17 |
Louis Bouchard |
attachment added |
|
yakkety-sssd_1.13.4-3ubuntu0.1.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4825142/+files/yakkety-sssd_1.13.4-3ubuntu0.1.debdiff |
|
| 2017-02-23 10:51:43 |
Louis Bouchard |
attachment removed |
yakkety-sssd_1.13.4-3ubuntu1.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4824488/+files/yakkety-sssd_1.13.4-3ubuntu1.debdiff |
|
|
| 2017-02-23 10:55:45 |
Louis Bouchard |
tags |
sts sts-sponsor sts-sru |
sts sts-sru |
|
| 2017-02-23 17:45:29 |
Brian Murray |
sssd (Ubuntu Yakkety): status |
Triaged |
Fix Committed |
|
| 2017-02-23 17:45:33 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-02-23 17:45:39 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2017-02-23 17:45:46 |
Brian Murray |
tags |
sts sts-sru |
sts sts-sru verification-needed |
|
| 2017-02-23 17:48:32 |
Brian Murray |
sssd (Ubuntu Xenial): status |
Triaged |
Fix Committed |
|
| 2017-02-23 17:49:50 |
Brian Murray |
sssd (Ubuntu Trusty): status |
Triaged |
Fix Committed |
|
| 2017-02-24 15:32:55 |
Victor Tapia |
attachment added |
|
librt link fix https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4826011/+files/trusty-sssd_1.11.8-0ubuntu0.5.debdiff |
|
| 2017-02-28 09:18:53 |
Victor Tapia |
attachment removed |
trusty-sssd_1.11.8-0ubuntu0.5.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4826011/+files/trusty-sssd_1.11.8-0ubuntu0.5.debdiff |
|
|
| 2017-02-28 09:19:07 |
Victor Tapia |
attachment added |
|
trusty-sssd_1.11.8-0ubuntu0.5.debdiff https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1641875/+attachment/4828066/+files/trusty-sssd_1.11.8-0ubuntu0.5.debdiff |
|
| 2017-03-02 15:01:50 |
Victor Tapia |
tags |
sts sts-sru verification-needed |
sts sts-sru verification-done-xenial verification-done-yakkety verification-needed |
|
| 2017-03-06 11:39:54 |
Victor Tapia |
tags |
sts sts-sru verification-done-xenial verification-done-yakkety verification-needed |
sts sts-sru verification-done |
|
| 2017-03-06 19:18:22 |
Launchpad Janitor |
sssd (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-03-06 19:18:35 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-03-06 19:19:38 |
Launchpad Janitor |
sssd (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-03-15 17:22:08 |
Launchpad Janitor |
sssd (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2017-03-22 15:41:37 |
Louis Bouchard |
tags |
sts sts-sru verification-done |
sts sts-sru-done verification-done |
|
