overcloud deployment fails while configuring network routes with "Error: Nexthop has invalid gateway"

[Luca Miccini]

we hit this in our bgp playground based on centos 8-stream and master.
overcloud deployment times out and on the nodes we see:

    WARN : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
    WARN : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
    WARN : [ifup] It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
    RTNETLINK answers: File exists
    WARN : [/etc/sysconfig/network-scripts/ifup-routes] Failed to add route 192.168.1.0/24 via 192.168.1.1 dev enp1s0, using ip route replace instead.
    Error: Nexthop has invalid gateway.
    WARN : [/etc/sysconfig/network-scripts/ifup-routes] Failed to add route 192.168.2.0/24 via 192.168.1.1 dev enp1s0, using ip route replace instead.
    Error: Nexthop has invalid gateway.
    Error: Nexthop has invalid gateway.
    WARN : [/etc/sysconfig/network-scripts/ifup-routes] Failed to add route 192.168.3.0/24 via 192.168.1.1 dev enp1s0, using ip route replace instead.
    Error: Nexthop has invalid gateway.
    Error: Nexthop has invalid gateway.
    WARN : [/etc/sysconfig/network-scripts/ifup-routes] Failed to add route 192.168.4.0/24 via 192.168.1.1 dev enp1s0, using ip route replace instead.
    Error: Nexthop has invalid gateway.

we tracked this down to:

    [root@ctrl-1-0 network-scripts]# rpm -q --changelog network-scripts |less
    * Mon Jan 24 2022 Jan Macku <email address hidden> - 10.00.17-1
    - ifup-routes: Revert quotes
    - ifup-routes: Use `ip replace` only on type `route`
    - ci: introduce GA ci from master to rhel8-branch
    - ci: Setup packit for RHEL8 branch
    - ci: Get rid of old ci files (travis and zanata)

downgrading to http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/network-scripts-10.00.15-1.el8.x86_64.rpm solved the issue.

it may be something we only hit because of our special routing setup, but I guess it's better to have it tracked somewhere.

[Luca Miccini (lmiccini2)]

another possible workaround is to add the "onlink" option to the ip route command. from the man page:

"onlink pretend that the nexthop is directly attached to this link, even if it does not match any interface prefix."