role IP not applying on target node, mmm_control reports it has applied.

[xcooling]

I have a master master setup with a third host setup as a monitor (monitor host IP is 192.168.14.238).

output for mmm_control show is as follows:

  db1(192.168.14.239) master/ONLINE. Roles: reader(192.168.14.195), writer(192.168.14.194)
  db2(192.168.14.240) master/ONLINE. Roles: reader(192.168.14.196)

Problem is every time I execute a move role say from db1 to db2 the output of mmm_control show changes to the following
which is expected

 db1(192.168.14.239) master/ONLINE. Roles: reader(192.168.14.195)
  db2(192.168.14.240) master/ONLINE. Roles: reader(192.168.14.196), writer(192.168.14.194)

But the address 192.168.14.194 never shows up on 192.168.14.240.
I can't ping it from the monitor either. But if I set it manually on the 14.240 using the confgure_ip script
 configure_ip mmm_agent eth0 192.168.14.194 it shows up in output for the "ip addr list" command which again is expected.

Now the interesting thing is if I move the writer role back to db1 192.168.14.194 disappears from db2 but never shows up on db1.
I tried setting db2 to offline just to check if it moves the reader role to db1 and it does that. It shows up in the output of 'ip addr list" on db1.

I am running on centos 5.5.

my common config file is as follows:

active_master_role writer

<host default>
        cluster_interface eth0
        pid_path /var/run/mmm_agentd.pid
        bin_path /usr/lib/mysql-mmm/
        replication_user repl
        replication_password replpasswd
        agent_user rep_agent
        agent_password rep1agent
</host>
<host db1>
        ip 192.168.14.239
        mode master
        peer db2
</host>
<host db2>
        ip 192.168.14.240
        mode master
        peer db1
</host>
<role writer>
        hosts db1, db2
        ips 192.168.14.194
        mode exclusive
</role>

<role reader>
        hosts db1, db2
        ips 192.168.14.195, 192.168.14.196
        mode balanced
</role>

The above file is the same on all 3 hosts. The monitor config file is as follows:

include mmm_common.conf

<monitor>
        ip 127.0.0.1
        pid_path /var/run/mmm_mond.pid
        bin_path /usr/lib/mysql-mmm/
        status_path /var/lib/misc/mmm_mond.status
        ping_ips 192.168.14.194, 192.168.14.195, 192.168.14.196, 192.168.14.239, 192.168.14.240
</monitor>

<host default>
        monitor_user rep_monitor
        monitor_password rep1monitor
</host>

debug 0

Please let me know if you know what I am doing wrong or any way i can to get your more information in order to resolve this issue.

Thanks

[xcooling (f-root-extremecooling-org)]

I am experiencing the exact same problem on ubuntu 10.10.

The floating IP address for the writer role (10.0.0.20) is not applying on the actual server. However according to mmm_monitor it has.

# mmm_control show (monitor node)
db1(10.0.0.2) master/ONLINE. Roles: reader(10.0.0.21), reader(10.0.0.22), writer(10.0.0.20)

# /var/log/mysql-mmm/mmm_mond.log (monitor node)
2011/01/24 08:49:50 FATAL Admin changed state of 'db1' from ONLINE to ADMIN_OFFLINE
2011/01/24 08:49:50 INFO Removing all roles from host 'db1':
2011/01/24 08:49:50 INFO Removed role 'reader(10.0.0.22)' from host 'db1'
2011/01/24 08:49:50 INFO Removed role 'reader(10.0.0.21)' from host 'db1'
2011/01/24 08:49:50 INFO Removed role 'writer(10.0.0.20)' from host 'db1'
2011/01/24 08:51:07 FATAL Admin changed state of 'db1' from ADMIN_OFFLINE to ONLINE
2011/01/24 08:51:07 INFO Orphaned role 'writer(10.0.0.20)' has been assigned to 'db1'
2011/01/24 08:51:07 INFO Orphaned role 'reader(10.0.0.22)' has been assigned to 'db1'
2011/01/24 08:51:07 INFO Orphaned role 'reader(10.0.0.21)' has been assigned to 'db1'

# ip addr list eth1 (db1 node)
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4096 qdisc pfifo_fast state UNKNO
WN qlen 1000
    link/ether 00:14:d1:17:57:8f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global eth1
    inet 10.0.0.21/32 scope global eth1
    inet 10.0.0.22/32 scope global eth1
    inet6 fe80::214:d1ff:fe17:578f/64 scope link
       valid_lft forever preferred_lft forever

--------------
MMM should be checking to make sure the ARP applies on the target server, to apply the floating ip. If the ip does not apply it should try again.

[shantanuo (shantanu-oak)]

I experienced the same (or similar) problem.
After shutting down one of the mysql services for maintenance, I started it again and MMM showed the correct status after some time, but the writer ip failed to ping. Try restarting the agent demon on masters and see if it helps.

[Marián Mižik (marian-mizik)]

Same problem here. Linux master2 2.6.26-2-amd64 #1 SMP Thu Nov 25 04:30:55 UTC 2010 x86_64 GNU/Linux (Debian Lenny)
MMM 2.2.1-1

[Pascal Hofmann (pascalhofmann)]

Is there some output related to this problem in the agent logfile?

Maybe start the agent in debug mode - put debug 1 in config and restart - (it won't detach from the console in debug mode and print generate a lot of output).

[Jason Centanni (jpcentanni-gmail)]

Just wanted to bump this as I have having the exact same problem. CentOS 5.5 on Monitor and Gentoo on Hosts.

Any ideas?

[shantanuo (shantanu-oak)]

I restarted the network service on the monitoring host and the writer IP could be ping again. I am not sure if that was the reason/ solution to the problem though.

[hueythecat (hueythecat)]

Ditto getting same issue on ubuntu 10.10.
Reader ip assignment fine, Writer assignment reports fine but doesn't happen

[fp (fp)]

I had the same problem, which turned out to be mysql permissions error. Check the /var/log/mysql-mmm/* log on the db servers and make sure there are no errors with mysql permissions.

[Kevin Rohrssen (kevin-rohrssen)]

I had the exact same issue. Review the logs on the agent machines. Moving of the writer role failed because mysql authentication failed on the mmm_agent account. Once I corrected the creditials, everything worked fine.