DM restart changes irb IP address and old IP is no longer usable

Bug #1772160 reported by Musa Abdin on 2018-05-19
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.1
Fix Committed
Critical
Suresh Balineni
R5.0
Fix Committed
Critical
Suresh Balineni
Trunk
Fix Committed
Critical
Suresh Balineni

Bug Description

In contrail 4.1.0/RHOSP 10 setup: |

restarting Device manager assigns new IP address to the irb interfaces, while the old IPs are no longer available.

[heat-admin@overcloud-contrailcontroller-0 ~]$ neutron subnet-show 24e8e71e-54fd-483f-bb10-fbe625fd0344
+-------------------+----------------------------------------+
| Field | Value |
+-------------------+----------------------------------------+
| allocation_pools | {"start": "8.8.8.2", "end": "8.8.8.5"} |
| cidr | 8.8.8.0/24 |
| created_at | |
| dns_nameservers | 0.0.0.0 |
| enable_dhcp | False |
| gateway_ip | 8.8.8.1 |
| host_routes | |
| id | 24e8e71e-54fd-483f-bb10-fbe625fd0344 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | 24e8e71e-54fd-483f-bb10-fbe625fd0344 |
| network_id | 18067a99-ed12-4a74-8b50-92577dc27bdb |
| tenant_id | db666c41aefc4a4581655e3c2e02627c |
| updated_at | |
+-------------------+----------------------------------------+
[heat-admin@overcloud-contrailcontroller-0 ~]$ curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://192.168.122.11:8082/virtual-network/18067a99-ed12-4a74-8b50-92577dc27bdb/subnet-ip-count -H "Content-Type: application/json" -d '{"subnet_list": ["24e8e71e-54fd-483f-bb10-fbe625fd0344"]}' |python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 79 100 22 100 57 142 370 --:--:-- --:--:-- --:--:-- 372
{
    "ip_count_list": [
        2
    ]
}
[heat-admin@overcloud-contrailcontroller-0 ~]$ sudo systemctl restart contrail-device-manager
[heat-admin@overcloud-contrailcontroller-0 ~]$ curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://192.168.122.11:8082/virtual-network/18067a99-ed12-4a74-8b50-92577dc27bdb/subnet-ip-count -H "Content-Type: application/json" -d '{"subnet_list": ["24e8e71e-54fd-483f-bb10-fbe625fd0344"]}' |python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 79 100 22 100 57 152 395 --:--:-- --:--:-- --:--:-- 398
{
    "ip_count_list": [
        3
    ]
}
[heat-admin@overcloud-contrailcontroller-0 ~]$ neutron port-create --name testsrv01 --fixed-ip subnet_id=24e8e71e-54fd-483f-bb10-fbe625fd0344,ip_address=8.8.8.3 `openstack network show sriov-test -c id -f value`
An unknown exception occurred.
Neutron server returns request_ids: ['req-38e57c68-57ae-40b9-a1a1-58a71bb5536c']
[heat-admin@overcloud-contrailcontroller-0 ~]$

on the mx

[edit]
musa# show groups __contrail__ interfaces irb.7
May 19 11:30:43
family inet {
    /* Allocated IPv4 Address from Subnet: 8.8.8.0/24 */
    address 8.8.8.3/24 {
        virtual-gateway-address 8.8.8.1;
    }
}

[edit]
musa#
May 19 11:32:52

after the restart:

[edit]
musa#
May 19 11:32:52

[edit]
musa# show groups __contrail__ interfaces irb.7
May 19 11:35:05
family inet {
    /* Allocated IPv4 Address from Subnet: 8.8.8.0/24 */
    address 8.8.8.4/24 {
        virtual-gateway-address 8.8.8.1;
    }
}

Musa Abdin (musamustafa) on 2018-05-21
Changed in juniperopenstack:
importance: High → Critical
description: updated
Vineet Gupta (vineetrf) on 2018-05-21
tags: added: hot
tags: added: 2018-0521-0294 hot-case
removed: hot
Jeba Paulaiyan (jebap) on 2018-05-24
tags: added: releaseblocker

Review in progress for https://review.opencontrail.org/43415
Submitter: Suresh Balineni (<email address hidden>)

information type: Proprietary → Public
Musa Abdin (musamustafa) wrote :

Tried the fix,
when restarting the DM, and the ip_count_list doesn't increase anymore, however the previous stale IP's are still there.
is there any way to clean those stale IP's?

Reviewed: https://review.opencontrail.org/43415
Committed: http://github.com/Juniper/contrail-controller/commit/eaaa3d2f01aed4141da4901c35aef59eba0d9f21
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit eaaa3d2f01aed4141da4901c35aef59eba0d9f21
Author: sbalineni <email address hidden>
Date: Wed May 30 14:14:29 2018 -0700

[DM]: Fix for reading DM private DB keyspace columns.

Currently DM invokes get_range() with column count 0.
This call does not return persisted ip-addresses.
Similar to this, ae id allocations also not read properly.
Fixed.

Change-Id: I37ad9dae8aec13a38cd810f15bc1265109c841b2
Closes-Bug: #1772160

Review in progress for https://review.opencontrail.org/43432
Submitter: Suresh Balineni (<email address hidden>)

Review in progress for https://review.opencontrail.org/43433
Submitter: Suresh Balineni (<email address hidden>)

Reviewed: https://review.opencontrail.org/43432
Committed: http://github.com/Juniper/contrail-controller/commit/d9230022f7c9e323aa24f759f5a5a2736af47aa2
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit d9230022f7c9e323aa24f759f5a5a2736af47aa2
Author: sbalineni <email address hidden>
Date: Wed May 30 14:14:29 2018 -0700

[DM]: Fix for reading DM private DB keyspace columns.

Currently DM invokes get_range() with column count 0.
This call does not return persisted ip-addresses.
Similar to this, ae id allocations also not read properly.
Fixed.

Change-Id: I37ad9dae8aec13a38cd810f15bc1265109c841b2
Closes-Bug: #1772160
(cherry picked from commit eaaa3d2f01aed4141da4901c35aef59eba0d9f21)

Himanshu (bhimanshu) on 2018-06-27
tags: removed: hot-case

Reviewed: https://review.opencontrail.org/43433
Committed: http://github.com/Juniper/contrail-controller/commit/7f0794723d857031da74089d52007376138a2933
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 7f0794723d857031da74089d52007376138a2933
Author: sbalineni <email address hidden>
Date: Wed May 30 14:14:29 2018 -0700

[DM]: Fix for reading DM private DB keyspace columns.

Currently DM invokes get_range() with column count 0.
This call does not return persisted ip-addresses.
Similar to this, ae id allocations also not read properly.
Fixed.

Change-Id: I37ad9dae8aec13a38cd810f15bc1265109c841b2
Closes-Bug: #1772160
(cherry picked from commit eaaa3d2f01aed4141da4901c35aef59eba0d9f21)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers