Gratuitous ARP are not being sent correctly by ns_IPaddr2 script
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Peter Zhurba | ||
6.0.x |
Won't Fix
|
High
|
Denis Meltsaykin | ||
6.1.x |
Fix Released
|
High
|
Denis Meltsaykin | ||
7.0.x |
Fix Released
|
High
|
Peter Zhurba |
Bug Description
Services could not connect to the database with errors:
root@node-3:~# cinder-manage --debug service list
2015-06-09 11:10:56.565 4946 WARNING oslo.db.
2015-06-09 11:13:13.782 4946 WARNING oslo.db.
2015-06-09 11:15:30.997 4946 WARNING oslo.db.
2015-06-09 11:17:48.214 4946 WARNING oslo.db.
2015-06-09 11:20:05.430 4946 WARNING oslo.db.
2015-06-09 11:22:22.645 4946 WARNING oslo.db.
2015-06-09 11:24:39.862 4946 WARNING oslo.db.
2015-06-09 11:26:57.078 4946 WARNING oslo.db.
Steps to reproduce:
OS: Ubuntu
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. ssh on controller and run umm on
6. Wait until controller is rebooting
7. Exit maintenance mode
8. wait for 10 minutes while controller become available
9. Run crm status command
10. run cinder-manage service list
11. run nova-manage service list
Actual result:
on controller where umm mode was enabled and disabled, cinder-scheduler marks with XXX
when we run command cinder-manage service list, it fail to connect to mysql, the same with nova, at the same time crm status shows that db resources started
root@node-3:~# crm status
Last updated: Tue Jun 9 11:34:22 2015
Last change: Tue Jun 9 00:18:39 2015
Stack: corosync
Current DC: node-2.
Version: 1.1.12-561c4cf
3 Nodes configured
33 Resources configured
Online: [ node-1.
Clone Set: clone_p_vrouter [p_vrouter]
Started: [ node-1.
vip__management (ocf::fuel:
vip__public_
vip__managemen
vip__public (ocf::fuel:
Master/Slave Set: master_p_conntrackd [p_conntrackd]
Masters: [ node-2.
Slaves: [ node-1.
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-1.
Clone Set: clone_p_dns [p_dns]
Started: [ node-1.
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-1.
p_ceilometer-
p_ceilometer-
Master/Slave Set: master_
Masters: [ node-2.
Slaves: [ node-1.
Clone Set: clone_p_heat-engine [p_heat-engine]
Started: [ node-1.
Clone Set: clone_ping_
Started: [ node-1.
Clone Set: clone_p_ntp [p_ntp]
Started: [ node-1.
root@node-3:~#
At the same time I can connect to the mysql and use cinder database:
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+------
| Database |
+------
| information_schema |
| cinder |
| glance |
| heat |
| keystone |
| mysql |
| nova |
| performance_schema |
| test |
+------
9 rows in set (0.06 sec)
mysql> use cinder
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> select * from services;
+------
| created_at | updated_at | deleted_at | deleted | id | host | binary | topic | report_count | disabled | availability_zone | disabled_reason |
+------
| 2015-06-08 22:39:11 | 2015-06-09 11:35:37 | NULL | 0 | 1 | node-1.
| 2015-06-08 22:58:01 | 2015-06-09 11:35:29 | NULL | 0 | 3 | node-2.
| 2015-06-08 22:58:30 | 2015-06-09 00:22:05 | NULL | 0 | 6 | node-3.
| 2015-06-08 23:10:39 | 2015-06-09 11:35:37 | NULL | 0 | 9 | node-4.
| 2015-06-08 23:10:40 | 2015-06-09 11:35:34 | NULL | 0 | 12 | node-5.
+------
5 rows in set (0.02 sec)
mysql>
so after that I try to ping management_
arp tables for this ip show old mac (after mm mode vip migraded but arp tables still have old mac adrress)
Last login: Tue Jun 9 00:30:00 2015 from 10.109.5.2
root@node-3:~# arp -a -n
? (10.109.7.2) at 92:23:07:2c:d1:4a [ether] on br-mgmt
? (10.109.6.2) at ba:cc:bf:be:e3:7f [ether] on br-ex
? (10.109.5.2) at 64:01:24:71:bd:f3 [ether] on br-fw-admin
? (10.109.9.3) at 64:e4:d8:1f:89:67 [ether] on br-storage
? (10.109.6.1) at 52:54:00:3d:31:27 [ether] on br-ex
? (10.109.9.2) at 64:cf:b5:d8:3a:01 [ether] on br-storage
? (10.0.0.4) at 64:f0:50:3b:db:bb [ether] on eth3.103
? (10.109.7.7) at 64:ff:30:1d:f2:2b [ether] on br-mgmt
? (10.0.0.3) at 64:f8:0e:98:36:5b [ether] on eth3.103
? (240.0.0.6) at 16:af:4d:65:c5:a4 [ether] on vrouter-host
? (10.109.6.128) at 64:12:61:46:d3:c8 [ether] on br-ex
? (10.109.7.5) at 56:0c:07:3a:6e:a8 [ether] on br-mgmt
? (10.109.5.1) at 52:54:00:01:b6:20 [ether] on br-fw-admin
? (10.109.7.4) at 64:2a:93:8f:8b:0e [ether] on br-mgmt
? (240.0.0.2) at ae:f9:b9:f3:3d:27 [ether] on hapr-host
? (10.109.7.3) at 32:18:dd:fa:ac:f0 [ether] on br-mgmt
? (10.109.6.3) at f2:45:7f:a0:71:bb [ether] on br-ex
? (10.109.7.8) at 64:ed:fe:fc:7e:78 [ether] on br-mgmt
root@node-3:~# ping 10.109.7.2
PING 10.109.7.2 (10.109.7.2) 56(84) bytes of data.
^C
--- 10.109.7.2 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7001ms
Then I remove row in arp tables for 10.109.7.2, ping it and ping passed and arp table was refreshed
root@node-3:~# arp -d 10.109.7.2
root@node-3:~# ping 10.109.7.2
PING 10.109.7.2 (10.109.7.2) 56(84) bytes of data.
64 bytes from 10.109.7.2: icmp_seq=1 ttl=64 time=0.653 ms
64 bytes from 10.109.7.2: icmp_seq=2 ttl=64 time=0.610 ms
64 bytes from 10.109.7.2: icmp_seq=3 ttl=64 time=0.552 ms
^C
--- 10.109.7.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.552/0.
root@node-3:~# arp -a -n
? (10.109.7.2) at d6:1f:1c:2f:df:82 [ether] on br-mgmt
? (10.109.6.2) at ba:cc:bf:be:e3:7f [ether] on br-ex
? (10.109.5.2) at 64:01:24:71:bd:f3 [ether] on br-fw-admin
? (10.109.9.3) at 64:e4:d8:1f:89:67 [ether] on br-storage
? (10.109.6.1) at 52:54:00:3d:31:27 [ether] on br-ex
? (10.109.9.2) at 64:cf:b5:d8:3a:01 [ether] on br-storage
? (10.0.0.4) at 64:f0:50:3b:db:bb [ether] on eth3.103
? (10.109.7.7) at 64:ff:30:1d:f2:2b [ether] on br-mgmt
? (10.0.0.3) at 64:f8:0e:98:36:5b [ether] on eth3.103
? (240.0.0.6) at 16:af:4d:65:c5:a4 [ether] on vrouter-host
? (10.109.6.128) at 64:12:61:46:d3:c8 [ether] on br-ex
? (10.109.7.5) at 56:0c:07:3a:6e:a8 [ether] on br-mgmt
? (10.109.5.1) at 52:54:00:01:b6:20 [ether] on br-fw-admin
? (10.109.7.4) at 64:2a:93:8f:8b:0e [ether] on br-mgmt
? (240.0.0.2) at ae:f9:b9:f3:3d:27 [ether] on hapr-host
? (10.109.7.3) at 32:18:dd:fa:ac:f0 [ether] on br-mgmt
? (10.109.6.3) at f2:45:7f:a0:71:bb [ether] on br-ex
? (10.109.7.8) at 64:ed:fe:fc:7e:78 [ether] on br-mgmt
Thanks Artem Panchenko for the help!!!
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "6.1"
openstack_
api: "1.0"
build_number: "521"
build_id: "2015-06-
nailgun_sha: "4340d55c190293
python-
astute_sha: "7766818f079881
fuel-library_sha: "f43c2ae1af3b49
fuel-ostf_sha: "7c938648a246e0
fuelmain_sha: "bcc909ffc5dd51
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Peter Zhurba (pzhurba) |
description: | updated |
Changed in fuel: | |
status: | New → Confirmed |
Changed in fuel: | |
status: | Confirmed → In Progress |
tags: | added: 6.1rc2 |
tags: | added: on-verification |
summary: |
- After umm mode disabling, services could not connect to the mysql + Gratuitous ARP are not being correctly by ns_IPaddr2 script |
summary: |
- Gratuitous ARP are not being correctly by ns_IPaddr2 script + Gratuitous ARP are not being sent correctly by ns_IPaddr2 script |
tags: | added: on-verification |
First look shows.
This issue appears after vip migration. We reproduce this issue but not on node where umm on off was performed.
umm switching just force vip migration process