where is suppossed to be placed the debug directive? i did place in
the common file but logs the same

Walter Heck escribió:
> Could you run in DEBUG mode and add output here? Just put 'debug 1'
> line in your configs an restart them. I'd be curiouse to see what's
> up..
> 
> Walter
> 
> On Fri, Oct 23, 2009 at 18:03, ufaria <email address hidden> wrote:
>> Yes, i've waited a lot of minutes and tried to change to online
>> several times. the only workaround is restart the monitor.
>>
>>
>> Walter Heck escribió:
>>> I've seen this happen. Have you waited more then just a few seconds?
>>> It usually comes back up after 1 minute or so in all cases I have
>>> seen..
>>>
>>> Walter
>>>
>>> On Fri, Oct 23, 2009 at 17:25, ufaria <email address hidden> wrote:
>>>> Public bug reported:
>>>>
>>>> when a node dies like "reboot -nf" after restart and pass the node to online status, sometimes
>>>> the node change automatically  to "replication delay" status but the cluster is doing nothing. and
>>>> all checks are ok.
>>>>
>>>> the only "workaround" is restart mmm-monitor to make that checks again the node ok and
>>>> pass it to online status.
>>>>
>>>> i have debian 5.0.3, the perl modules are all of the distribution.
>>>>
>>>> HERE I KILL THE NODE:
>>>>
>>>> 2009/10/22 21:03:08 FATAL Can't reach agent on host 'rkdb2'
>>>> 2009/10/22 21:03:10 ERROR Check 'ping' on 'rkdb2' has failed for 11 seconds! Message: ERROR: Could not ping 192.168.71.181
>>>> 2009/10/22 21:03:11 FATAL State of host 'rkdb2' changed from ONLINE to HARD_OFFLINE (ping: not OK, mysql: OK)
>>>> 2009/10/22 21:03:11 ERROR Can't send offline status notification to 'rkdb2' - killing it!
>>>> 2009/10/22 21:03:11 FATAL Could not kill host 'rkdb2' - there may be some duplicate ips now! (There's no binary configured for killing hosts.)
>>>> 2009/10/22 21:03:15 ERROR Check 'mysql' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:15 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:16 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:35  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:03:43  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:03:56 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:04:03  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:04:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:04:44 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:04:52  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:05:11  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:05:17  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>>>> 2009/10/22 21:05:23 FATAL Agent on host 'rkdb2' is reachable again
>>>> 2009/10/22 21:05:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>>>>
>>>> HERE THE NODE IS ONLINE:
>>>>
>>>> 2009/10/22 21:05:53 FATAL State of host 'rkdb2' changed from
>>>> HARD_OFFLINE to AWAITING_RECOVERY
>>>>
>>>> HERE I CHANGE TO ONLINE :
>>>>
>>>> 2009/10/22 21:06:01 FATAL Admin changed state of 'rkdb2' from
>>>> AWAITING_RECOVERY to ONLINE
>>>>
>>>> AND THEN, IT PASS TO REPLICATION DELAY:
>>>>
>>>> 2009/10/22 21:06:02 FATAL State of host 'rkdb2' changed from ONLINE to
>>>> REPLICATION_DELAY
>>>>
>>>> NOW I RESTART THE MONITOR:
>>>>
>>>> 2009/10/22 21:06:31  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>>>> 2009/10/22 21:06:31  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>>>> 2009/10/22 21:06:31  WARN rkdb1: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:31  WARN rkdb2: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:41  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>>>> 2009/10/22 21:06:41  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>>>> 2009/10/22 21:06:41  WARN rkdb1: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:41  WARN rkdb2: auto_increment_increment (1) should be >= 2
>>>>
>>>> AND PASS BYSELF TO ONLINE:
>>>>
>>>> 2009/10/22 21:06:45 FATAL State of host 'rkdb2' changed from
>>>> REPLICATION_DELAY to ONLINE
>>>>
>>>> ** Affects: mysql-mmm
>>>>     Importance: Undecided
>>>>         Status: New
>>>>
>>>> --
>>>> mmm-monitor don't pass to online state
>>>> https://bugs.launchpad.net/bugs/458907
>>>> You received this bug notification because you are subscribed to mysql-
>>>> mmm.
>>>>
>>>> Status in MMM for MySQL: New
>>>>
>>>> Bug description:
>>>> when a node dies like "reboot -nf" after restart and pass the node to online status, sometimes
>>>> the node change automatically  to "replication delay" status but the cluster is doing nothing. and
>>>> all checks are ok.
>>>>
>>>> the only "workaround" is restart mmm-monitor to make that checks again the node ok and
>>>> pass it to online status.
>>>>
>>>> i have debian 5.0.3, the perl modules are all of the distribution.
>>>>
>>>> HERE I KILL THE NODE:
>>>>
>>>> 2009/10/22 21:03:08 FATAL Can't reach agent on host 'rkdb2'
>>>> 2009/10/22 21:03:10 ERROR Check 'ping' on 'rkdb2' has failed for 11 seconds! Message: ERROR: Could not ping 192.168.71.181
>>>> 2009/10/22 21:03:11 FATAL State of host 'rkdb2' changed from ONLINE to HARD_OFFLINE (ping: not OK, mysql: OK)
>>>> 2009/10/22 21:03:11 ERROR Can't send offline status notification to 'rkdb2' - killing it!
>>>> 2009/10/22 21:03:11 FATAL Could not kill host 'rkdb2' - there may be some duplicate ips now! (There's no binary configured for killing hosts.)
>>>> 2009/10/22 21:03:15 ERROR Check 'mysql' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:15 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:16 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:03:35  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:03:43  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:03:56 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:04:03  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:04:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:04:44 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>>>> 2009/10/22 21:04:52  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:05:11  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>>>> 2009/10/22 21:05:17  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>>>> 2009/10/22 21:05:23 FATAL Agent on host 'rkdb2' is reachable again
>>>> 2009/10/22 21:05:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>>>>
>>>> HERE THE NODE IS ONLINE:
>>>>
>>>> 2009/10/22 21:05:53 FATAL State of host 'rkdb2' changed from HARD_OFFLINE to AWAITING_RECOVERY
>>>>
>>>> HERE I CHANGE TO ONLINE :
>>>>
>>>> 2009/10/22 21:06:01 FATAL Admin changed state of 'rkdb2' from AWAITING_RECOVERY to ONLINE
>>>>
>>>> AND THEN, IT PASS TO REPLICATION DELAY:
>>>>
>>>> 2009/10/22 21:06:02 FATAL State of host 'rkdb2' changed from ONLINE to REPLICATION_DELAY
>>>>
>>>> NOW I RESTART THE MONITOR:
>>>>
>>>> 2009/10/22 21:06:31  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>>>> 2009/10/22 21:06:31  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>>>> 2009/10/22 21:06:31  WARN rkdb1: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:31  WARN rkdb2: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:41  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>>>> 2009/10/22 21:06:41  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>>>> 2009/10/22 21:06:41  WARN rkdb1: auto_increment_increment (1) should be >= 2
>>>> 2009/10/22 21:06:41  WARN rkdb2: auto_increment_increment (1) should be >= 2
>>>>
>>>> AND PASS BYSELF TO ONLINE:
>>>>
>>>> 2009/10/22 21:06:45 FATAL State of host 'rkdb2' changed from REPLICATION_DELAY to ONLINE
>>>>
>> --
>> --------------------------
>>    Uxio Faria Giraldez
>> --------------------------
>>    SISTEMAS
>>    PRISACOM S.A.
>>    +34 913 537 770
>> ==========================
>>
>> --
>> mmm-monitor don't pass to online state
>> https://bugs.launchpad.net/bugs/458907
>> You received this bug notification because you are subscribed to mysql-
>> mmm.
>>
>> Status in MMM for MySQL: New
>>
>> Bug description:
>> when a node dies like "reboot -nf" after restart and pass the node to online status, sometimes
>> the node change automatically  to "replication delay" status but the cluster is doing nothing. and
>> all checks are ok.
>>
>> the only "workaround" is restart mmm-monitor to make that checks again the node ok and
>> pass it to online status.
>>
>> i have debian 5.0.3, the perl modules are all of the distribution.
>>
>> HERE I KILL THE NODE:
>>
>> 2009/10/22 21:03:08 FATAL Can't reach agent on host 'rkdb2'
>> 2009/10/22 21:03:10 ERROR Check 'ping' on 'rkdb2' has failed for 11 seconds! Message: ERROR: Could not ping 192.168.71.181
>> 2009/10/22 21:03:11 FATAL State of host 'rkdb2' changed from ONLINE to HARD_OFFLINE (ping: not OK, mysql: OK)
>> 2009/10/22 21:03:11 ERROR Can't send offline status notification to 'rkdb2' - killing it!
>> 2009/10/22 21:03:11 FATAL Could not kill host 'rkdb2' - there may be some duplicate ips now! (There's no binary configured for killing hosts.)
>> 2009/10/22 21:03:15 ERROR Check 'mysql' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>> 2009/10/22 21:03:15 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>> 2009/10/22 21:03:16 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>> 2009/10/22 21:03:35  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:03:43  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:03:56 ERROR Check 'rep_backlog' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>> 2009/10/22 21:04:03  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:04:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:04:44 ERROR Check 'rep_threads' on 'rkdb2' has failed for 14 seconds! Message: ERROR: Timeout
>> 2009/10/22 21:04:52  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:05:11  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (113)
>> 2009/10/22 21:05:17  WARN Check 'rep_backlog' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>> 2009/10/22 21:05:23 FATAL Agent on host 'rkdb2' is reachable again
>> 2009/10/22 21:05:23  WARN Check 'rep_threads' on 'rkdb2' is in unknown state! Message: UNKNOWN: Connect error (host = 192.168.71.181:3306, user = mmm_monitor)! Can't connect to MySQL server on '192.168.71.181' (111)
>>
>> HERE THE NODE IS ONLINE:
>>
>> 2009/10/22 21:05:53 FATAL State of host 'rkdb2' changed from HARD_OFFLINE to AWAITING_RECOVERY
>>
>> HERE I CHANGE TO ONLINE :
>>
>> 2009/10/22 21:06:01 FATAL Admin changed state of 'rkdb2' from AWAITING_RECOVERY to ONLINE
>>
>> AND THEN, IT PASS TO REPLICATION DELAY:
>>
>> 2009/10/22 21:06:02 FATAL State of host 'rkdb2' changed from ONLINE to REPLICATION_DELAY
>>
>> NOW I RESTART THE MONITOR:
>>
>> 2009/10/22 21:06:31  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>> 2009/10/22 21:06:31  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>> 2009/10/22 21:06:31  WARN rkdb1: auto_increment_increment (1) should be >= 2
>> 2009/10/22 21:06:31  WARN rkdb2: auto_increment_increment (1) should be >= 2
>> 2009/10/22 21:06:41  WARN No binary found for killing hosts (/usr/bin/mysql-mmm//monitor/kill_host).
>> 2009/10/22 21:06:41  WARN auto_increment_offset should be different on both masters (rkdb1: 1 , rkdb2: 1)
>> 2009/10/22 21:06:41  WARN rkdb1: auto_increment_increment (1) should be >= 2
>> 2009/10/22 21:06:41  WARN rkdb2: auto_increment_increment (1) should be >= 2
>>
>> AND PASS BYSELF TO ONLINE:
>>
>> 2009/10/22 21:06:45 FATAL State of host 'rkdb2' changed from REPLICATION_DELAY to ONLINE
>>
> 

-- 
--------------------------
    Uxio Faria Giraldez
--------------------------
    SISTEMAS
    PRISACOM S.A.
    +34 913 537 770
==========================