system-peer peer_controller_gateway update doesn't update routes and PGA sync status

[Gustavo Herzmann]

Brief Description
-----------------
When a system peer is created with incorrect peer_controller_gateway IP, when updating it to the correct IP, the routes and the PGA sync status are not updated

Severity
--------
<Major: System/Feature is usable but degraded>

Steps to Reproduce
------------------
1 - Create a system peer with an incorrect systemcontroller_gateway_ip. The IP must be part of the management subnet and outside of the reserved management IP range.
2 - Create a peer group association. After it sync, site2 has the secondary subcloud and a route pointing to the specified IP.
3 - Update the system peer with the correct IP on site1 (the PGA sync status is not set to 'out-of-sync') and then sync it.
4 - A new route is not created on site2 and the subclouds still uses the old IP
5 - Try to migrate the SPG to site2, the rehoming succeeds but the subcloud stays offline due to the missing route.

Expected Behavior
------------------
Route/System controller-gateway IP/ PGA sync status should be updated.

Actual Behavior
----------------
Route/System controller-gateway IP/ PGA sync status are not updated.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Distributed cloud system with geo-redundancy

Branch/Pull Time/Commit
-----------------------
Master (04-18-2024)

Last Pass
---------
Is present since the development of the geo-redundancy feature

Timestamp/Logs
--------------
NA

Test Activity
-------------
Feature Testing

Workaround
----------
Manually create the missing route

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/916139
Committed: https://opendev.org/starlingx/distcloud/commit/d3b7434a5b6279ecdc924f625e19579033868321
Submitter: "Zuul (22348)"
Branch: master

commit d3b7434a5b6279ecdc924f625e19579033868321
Author: Gustavo Herzmann <email address hidden>
Date: Wed Apr 17 15:40:28 2024 -0300

    Fix peer-controller-gateway-address update

    This commit updates the peer group association sync status to
    'out-of-sync' after the user updates the
    peer-controller-gateway-address attribute of the system-peer object.

    This commit also modifies the subcloud update function to update the
    subcloud route whenever the systemcontroller_gateway_address is
    updated on the primary side and synced to the secondary.

    It also adds an informative message to remind the caller to run the
    sync command after updating the peer-controller-gateway-address.

    Test Plan:
    1. PASS: Do the following steps:
             - Create a system peer with an incorrect systemcontroller
             gateway address that's inside the management subcloud, but
             outside the reserved IP range and then create an association.
             Verify that the secondary subcloud and a route was created
             using the incorrect IP.
             - Update the system peer with the correct systemcontroller
             gateway address on the primary site. Verify that the PGA
             sync status is set to 'out-of-sync' on both sites.
             - Sync the PGA and verify that the secondary subcloud
             systemcontroller gateway address was updated and that the
             old route was deleted and a new one using the new address
             was created.
             - Migrate the SPG to the non-primary site and verify that
             it completes successfully and that the subcloud becomes
             online and managed.
    2. PASS: Repeat the first step of test case #1, but use an incorrect
             address that's outside the management subnet. Then create
             a PGA and verify that it fails due to the following
             validation:
             "systemcontroller_gateway_address invalid: Address must be in
             subnet <management subnet>"
    3. PASS: Repeat the first step of test case #1, but use an incorrect
             address that's inside the reserved IP range. Then create
             a PGA and verify that it fails due to the following
             validation:
             "systemcontroller_gateway_address invalid, is within
             management pool <ip range>"
    4. PASS: Create a system peer with a correct systemcontroller gateway
             address for the first time and then create an association.
             Verify that the secondary subcloud and a route was created
             using the correct IP.
    5. PASS: Update an attribute of the subcloud (e.g. the subcloud
             description) on the primary site and verify that the sync
             status chages to 'out-of-sync' on both sites, then run
             the PGA sync operation and verify that the attribute was
             synced to the seconda...