kolla-ansible

From Rocky to Stein, router in one network node creation issue

Bug #1869855 reported by Kevin Zhao on 2020-03-31

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Invalid	Undecided	Unassigned

Bug Description

Upgrade from Rocky to Stein,

After that we find 1 nodes behavior is weird.
The router here are:
root@uk-dc-cavium-01:~# ip netns exec qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

While in other nodes:
root@uk-dc-cavium-02:~# ip netns exec qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2841: ha-be3caa07-49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:69:4c:14 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.21/18 brd 169.254.255.255 scope global ha-be3caa07-49
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe69:4c14/64 scope link
       valid_lft forever preferred_lft forever
2875: qr-d4dab906-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:71:e2:fa brd ff:ff:ff:ff:ff:ff
2941: qg-5e90ae7f-1a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:50:dd:ee brd ff:ff:ff:ff:ff:ff

Or:
root@uk-dc-cavium-03:~# ip netns exec qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
350: ha-eb12605a-94: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:f2:39:3f brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.8/18 brd 169.254.255.255 scope global ha-eb12605a-94
       valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-eb12605a-94
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fef2:393f/64 scope link
       valid_lft forever preferred_lft forever
375: qr-d4dab906-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:71:e2:fa brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 scope global qr-d4dab906-11
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe71:e2fa/64 scope link nodad
       valid_lft forever preferred_lft forever
397: qg-5e90ae7f-1a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:50:dd:ee brd ff:ff:ff:ff:ff:ff
    inet 213.146.141.112/32 scope global qg-5e90ae7f-1a
       valid_lft forever preferred_lft forever
    inet 213.146.141.33/25 scope global qg-5e90ae7f-1a
       valid_lft forever preferred_lft forever
    inet 213.146.155.37/32 scope global qg-5e90ae7f-1a
       valid_lft forever preferred_lft forever
    inet 213.146.155.76/32 scope global qg-5e90ae7f-1a
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe50:ddee/64 scope link nodad
       valid_lft forever preferred_lft forever

=================================================================

In the problem node, when IP addr, we can get a lot of interface:

root@uk-dc-cavium-01:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: control: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:98:97:38 brd ff:ff:ff:ff:ff:ff
    inet 10.20.20.3/16 brd 10.20.255.255 scope global control
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe98:9738/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:98:97:39 brd ff:ff:ff:ff:ff:ff
4: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:98:97:3a brd ff:ff:ff:ff:ff:ff
    inet 213.146.141.4/25 brd 213.146.141.127 scope global public
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe98:973a/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:98:97:3b brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:98:97:3c brd ff:ff:ff:ff:ff:ff
7: internal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether ec:0d:9a:d9:87:f6 brd ff:ff:ff:ff:ff:ff
    inet 10.22.20.1/16 brd 10.22.255.255 scope global internal
       valid_lft forever preferred_lft forever
    inet6 fe80::ee0d:9aff:fed9:87f6/64 scope link
       valid_lft forever preferred_lft forever
8: external: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether ec:0d:9a:d9:87:f7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fed9:87f7/64 scope link
       valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:39:d1:c3:b3 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
10: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:f3:52:7e:8a:9c brd ff:ff:ff:ff:ff:ff
290: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 46:3c:3b:9f:4e:41 brd ff:ff:ff:ff:ff:ff
40: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2a:dd:1f:9d:26:46 brd ff:ff:ff:ff:ff:ff
307: br-ex: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
    link/ether ec:0d:9a:d9:87:f7 brd ff:ff:ff:ff:ff:ff
52: tap40915253-f5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8a:8a:d0:81:f1:73 brd ff:ff:ff:ff:ff:ff
59: tapf2ec4a58-e0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:da:1a:9b:fc:8c brd ff:ff:ff:ff:ff:ff
846: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 92:c0:23:d8:5e:1c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90c0:23ff:fed8:5e1c/64 scope link
       valid_lft forever preferred_lft forever
600: qg-2ac6f1b1-06: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:dd:8e:ff:b1:d4 brd ff:ff:ff:ff:ff:ff
605: tap92dabaa3-b7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 26:00:5b:fc:93:f8 brd ff:ff:ff:ff:ff:ff
606: ha-30a37c5a-45: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 72:ba:c1:8c:4d:4b brd ff:ff:ff:ff:ff:ff
607: ha-48c277e3-dd: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ce:2b:5d:0f:59:9e brd ff:ff:ff:ff:ff:ff
609: ha-6548d93e-15: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether de:fc:7b:24:2e:90 brd ff:ff:ff:ff:ff:ff
611: qg-d8c6dc57-f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 86:7b:8c:83:df:b9 brd ff:ff:ff:ff:ff:ff
620: qr-4f055af8-29: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:1f:5d:78:50:37 brd ff:ff:ff:ff:ff:ff
621: qg-1d13a0c3-58: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:c8:7c:00:16:3d brd ff:ff:ff:ff:ff:ff
624: tap976105c8-88: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 2a:6c:c0:4d:47:02 brd ff:ff:ff:ff:ff:ff
625: qg-536d5df4-c2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3a:54:7f:4f:9d:f5 brd ff:ff:ff:ff:ff:ff
629: ha-12734cef-cc: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 46:39:6c:f6:6a:b8 brd ff:ff:ff:ff:ff:ff
631: ha-0356e11d-64: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 56:17:6d:44:a3:32 brd ff:ff:ff:ff:ff:ff
636: tapa37a33ae-53: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 5a:b2:77:a6:ad:17 brd ff:ff:ff:ff:ff:ff
639: qg-28def6aa-ad: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:db:3d:a8:b5:ae brd ff:ff:ff:ff:ff:ff
641: ha-62e9a3e1-e2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7a:e0:ac:9b:7c:b4 brd ff:ff:ff:ff:ff:ff
642: qg-5e90ae7f-1a: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:02:66:87:a6:ab brd ff:ff:ff:ff:ff:ff
643: qr-15b97210-b5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether be:1b:13:e8:d2:45 brd ff:ff:ff:ff:ff:ff
645: ha-9ad84712-b3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:2d:4a:9c:ae:44 brd ff:ff:ff:ff:ff:ff
647: ha-ca347ff7-c7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6a:e4:5c:45:8b:d8 brd ff:ff:ff:ff:ff:ff
648: qr-894f7b63-82: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f6:ab:4e:32:cc:67 brd ff:ff:ff:ff:ff:ff
652: qg-d36393c8-a1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:be:50:35:2e:3a brd ff:ff:ff:ff:ff:ff
654: qg-a4f5970a-9d: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c6:4f:75:01:39:a3 brd ff:ff:ff:ff:ff:ff
656: tapa673812c-a0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b6:5d:95:92:db:15 brd ff:ff:ff:ff:ff:ff
657: qg-381a8968-50: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 32:cc:0c:78:ba:17 brd ff:ff:ff:ff:ff:ff
658: qg-9e6dea7d-ae: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 8e:6a:53:dc:fa:b0 brd ff:ff:ff:ff:ff:ff
659: qg-8c92a80d-59: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4e:88:a2:2c:1c:c8 brd ff:ff:ff:ff:ff:ff
660: qr-f4c29d10-73: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:43:4c:72:b6:02 brd ff:ff:ff:ff:ff:ff
662: qg-f5035cdb-40: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3a:02:aa:70:08:d9 brd ff:ff:ff:ff:ff:ff
666: qr-36c19935-bd: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 42:48:57:ca:c2:68 brd ff:ff:ff:ff:ff:ff
667: qr-aa9493a4-19: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state D

===================================================

While in common node:
root@uk-dc-cavium-03:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: control: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:94:ab:0a brd ff:ff:ff:ff:ff:ff
    inet 10.20.20.6/16 brd 10.20.255.255 scope global control
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe94:ab0a/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:94:ab:0b brd ff:ff:ff:ff:ff:ff
4: public: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:94:ab:0c brd ff:ff:ff:ff:ff:ff
    inet 213.146.141.5/25 brd 213.146.141.127 scope global public
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe94:ab0c/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:94:ab:0d brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:1b:0d:94:ab:0e brd ff:ff:ff:ff:ff:ff
7: internal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether ec:0d:9a:d9:84:22 brd ff:ff:ff:ff:ff:ff
    inet 10.22.20.3/16 brd 10.22.255.255 scope global internal
       valid_lft forever preferred_lft forever
    inet6 fe80::ee0d:9aff:fed9:8422/64 scope link
       valid_lft forever preferred_lft forever
8: external: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether ec:0d:9a:d9:84:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fed9:8423/64 scope link
       valid_lft forever preferred_lft forever
9: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:10:de:9a:a8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
10: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1a:94:c8:96:03:90 brd ff:ff:ff:ff:ff:ff
13: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 16:d7:77:34:81:49 brd ff:ff:ff:ff:ff:ff
532: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65535 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 06:e7:52:63:79:ce brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4e7:52ff:fe63:79ce/64 scope link
       valid_lft forever preferred_lft forever
31: br-ex: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
    link/ether ec:0d:9a:d9:84:23 brd ff:ff:ff:ff:ff:ff
84: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 82:3d:a9:24:bf:4e brd ff:ff:ff:ff:ff:ff
89: tap4650386a-47: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1a:61:02:a6:4e:75 brd ff:ff:ff:ff:ff:ff

Revision history for this message

Kevin Zhao (kevin-zhao) wrote on 2020-03-31:

Download full text (7.6 KiB)

Common router log:

Common router log:

653216:2020-03-31 10:10:42.439 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167', 'sysctl', '-w', 'net.ipv6.conf.qg-a5a56605-b9.forwarding=0'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
653325:2020-03-31 10:10:43.660 35 DEBUG neutron.agent.l3.ha [-] Closing metadata proxy for router ef38069e-ba1c-41e7-92ff-2b7346cb2167 _update_metadata_proxy /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/ha.py:213
653444:2020-03-31 10:10:44.697 35 DEBUG neutron.agent.l3.router_info [-] Terminating radvd daemon in router device: ef38069e-ba1c-41e7-92ff-2b7346cb2167 disable_radvd /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:550
653445:2020-03-31 10:10:44.699 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/ef38069e-ba1c-41e7-92ff-2b7346cb2167.pid.radvd get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
653446:2020-03-31 10:10:44.701 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/ef38069e-ba1c-41e7-92ff-2b7346cb2167.pid.radvd get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
653447:2020-03-31 10:10:44.702 35 DEBUG neutron.agent.linux.external_process [-] No process started for ef38069e-ba1c-41e7-92ff-2b7346cb2167 disable /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/external_process.py:122
653448:2020-03-31 10:10:44.703 35 DEBUG neutron.agent.linux.ra [-] radvd disabled for router ef38069e-ba1c-41e7-92ff-2b7346cb2167 disable /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ra.py:198
653460:2020-03-31 10:10:45.334 35 DEBUG neutron.agent.l3.ha [-] Updating server with HA routers states {'c30a2186-3606-4feb-9a57-1f44088b858c': 'active', 'a62e219b-b0d8-4204-ab75-a640defab8da': 'active', 'ef38069e-ba1c-41e7-92ff-2b7346cb2167': 'standby'} notify_server /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/ha.py:229
653585:2020-03-31 10:10:48.572 35 DEBUG neutron.agent.l3.agent [req-c3b04c82-5718-4413-81bc-aa6d001df008 - - - - -] Got routers updated notification :[u'ef38069e-ba1c-41e7-92ff-2b7346cb2167'] routers_updated /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:497
653586:2020-03-31 10:10:48.577 35 INFO neutron.agent.l3.agent [-] Starting router update for ef38069e-ba1c-41e7-92ff-2b7346cb2167, action 3, priority 1, update_id d3fa8e74-a463-4856-b323-afd30769ce67. Wait time elapsed: 0.002
653836:2020-03-31 10:10:59.971 35 DEBUG neutron.agent.l3.router_info [-] Process updates, router ef38069e-ba1c-41e7-92ff-2b7346cb2167 process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1183
653849:2020-03-31 10:11:00.745 35 DEBUG oslo_concurrency.lockutils [-] Acquired lock "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265
653850:2020-03-31 10:11:00.748 35 DEBUG oslo_concurrency.lockutils [-] Acquired external semaphore "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:272
653851:2020-03-31 10:11:00.751 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167', 'iptables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
653861:2020-03-31 10:11:01.651 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167', 'ip6tables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
653879:2020-03-31 10:11:02.532 35 DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281
653880:2020-03-31 10:11:02.537 35 DEBUG oslo_concurrency.lockutils [-] Acquired lock "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:265
653881:2020-03-31 10:11:02.540 35 DEBUG oslo_concurrency.lockutils [-] Acquired external semaphore "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:272
653882:2020-03-31 10:11:02.543 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167', 'iptables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
653895:2020-03-31 10:11:03.617 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167', 'ip6tables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
653911:2020-03-31 10:11:04.722 35 DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-ef38069e-ba1c-41e7-92ff-2b7346cb2167" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281
653912:2020-03-31 10:11:04.724 35 DEBUG neutron.agent.l3.ha_router [-] Processing HA router with HA port: {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2020-03-31T08:46:37Z', u'device_owner': u'network:router_ha_interface', u'revision_number': 18, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'61792fb4-1e40-4c06-8670-57f8d0dff457', u'prefixlen': 18, u'ip_address': u'169.254.192.4'}], u'id': u'0607aabc-6d0f-4d80-924a-0e0ef2cb6366', u'security_groups': [], u'binding:vif_details': {u'port_filter': True, u'datapath_type': u'system', u'ovs_hybrid_plug': True}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'ovs', u'mac_address': u'fa:16:3e:d9:df:3c', u'project_id': u'', u'status': u'ACTIVE', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': None, u'cidr': u'169.254.192.0/18', u'id': u'61792fb4-1e40-4c06-8670-57f8d0dff457', u'subnetpool_id': None}], u'binding:host_id': u'uk-dc-cavium-01', u'description': u'', u'tags': [], u'device_id': u'ef38069e-ba1c-41e7-92ff-2b7346cb2167', u'name': u'HA port tenant bd1fa0e6c41a49bd805977dba997c230', u'admin_state_up': True, u'network_id': u'de56d4d5-aa3c-4ed7-930e-6e420e7a41f0', u'tenant_id': u'', u'created_at': u'2020-03-02T02:36:27Z', u'mtu': 1450, u'extra_subnets': []} process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py:488
653914:2020-03-31 10:11:04.728 35 DEBUG neutron.agent.linux.keepalived [-] Router ef38069e-ba1c-41e7-92ff-2b7346cb2167 keepalived config: global_defs {
653920:    script "/var/lib/neutron/ha_confs/ef38069e-ba1c-41e7-92ff-2b7346cb2167/ha_check_script_1.sh"
653966:2020-03-31 10:11:05.611 35 DEBUG neutron.agent.linux.keepalived [-] Keepalived spawned with config /var/lib/neutron/ha_confs/ef38069e-ba1c-41e7-92ff-2b7346cb2167/keepalived.conf spawn /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py:446
653970:2020-03-31 10:11:05.617 35 INFO neutron.agent.l3.agent [-] Finished a router update for ef38069e-ba1c-41e7-92ff-2b7346cb216

Revision history for this message

Kevin Zhao (kevin-zhao) wrote on 2020-03-31:

Download full text (16.2 KiB)

Issue router log in l3:
656707:2020-03-31 10:13:50.428 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'iptables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
656714:2020-03-31 10:13:51.254 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'ip6tables-save'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
656722:2020-03-31 10:13:51.992 35 DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1" lock /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:281
656723:2020-03-31 10:13:51.994 35 DEBUG neutron.agent.l3.ha_router [-] Processing HA router with HA port: {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2020-03-31T09:13:41Z', u'device_owner': u'network:router_ha_interface', u'revision_number': 24, u'port_security_enabled': False, u'binding:profile': {}, u'binding:vnic_type': u'normal', u'fixed_ips': [{u'subnet_id': u'0bc00bb1-9c15-47e0-83c5-706134235cd4', u'prefixlen': 18, u'ip_address': u'169.254.192.4'}], u'id': u'd05bc2a7-a29a-46b7-8c4f-238a1fc0b0bc', u'security_groups': [], u'binding:vif_details': {u'port_filter': True, u'datapath_type': u'system', u'ovs_hybrid_plug': True}, u'address_scopes': {u'4': None, u'6': None}, u'binding:vif_type': u'ovs', u'mac_address': u'fa:16:3e:a2:b5:66', u'project_id': u'', u'status': u'ACTIVE', u'subnets': [{u'dns_nameservers': [], u'ipv6_ra_mode': None, u'gateway_ip': None, u'cidr': u'169.254.192.0/18', u'id': u'0bc00bb1-9c15-47e0-83c5-706134235cd4', u'subnetpool_id': None}], u'binding:host_id': u'uk-dc-cavium-01', u'description': u'', u'tags': [], u'device_id': u'44cc73d8-fb3c-408b-be22-682931cb1fc1', u'name': u'HA port tenant a9b848288ea842b6a3ee1966fe96aa4b', u'admin_state_up': True, u'network_id': u'cc4b14d5-2864-4f7b-823b-33898f939b01', u'tenant_id': u'', u'created_at': u'2018-09-05T21:55:38Z', u'mtu': 8950, u'extra_subnets': []} process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py:488
656725:2020-03-31 10:13:51.997 35 DEBUG neutron.agent.linux.keepalived [-] Router 44cc73d8-fb3c-408b-be22-682931cb1fc1 keepalived config: global_defs {
656731:    script "/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/ha_check_script_1.sh"
656768:2020-03-31 10:13:52.001 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
656769:2020-03-31 10:13:52.004 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
656770:2020-03-31 10:13:52.007 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
656783:2020-03-31 10:13:52.857 35 DEBUG neutron.agent.linux.keepalived [-] Keepalived spawned with config /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf spawn /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/keepalived.py:446
656787:2020-03-31 10:13:52.863 35 INFO neutron.agent.l3.agent [-] Finished a router update for 44cc73d8-fb3c-408b-be22-682931cb1fc1, update_id 532d198f-c43a-4492-80eb-d01ddde4d86c. Time elapsed: 9.846
657149:2020-03-31 10:14:54.864 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657150:2020-03-31 10:14:54.866 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
657151:2020-03-31 10:14:54.867 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
657152:2020-03-31 10:14:54.868 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657153:2020-03-31 10:14:54.869 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657154:2020-03-31 10:14:54.870 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
657376:2020-03-31 10:15:54.814 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657377:2020-03-31 10:15:54.815 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
657378:2020-03-31 10:15:54.816 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
657379:2020-03-31 10:15:54.817 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657380:2020-03-31 10:15:54.818 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657381:2020-03-31 10:15:54.819 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
657604:2020-03-31 10:16:55.152 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657605:2020-03-31 10:16:55.153 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
657606:2020-03-31 10:16:55.154 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
657607:2020-03-31 10:16:55.155 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657608:2020-03-31 10:16:55.156 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657609:2020-03-31 10:16:55.158 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
657831:2020-03-31 10:17:55.162 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657832:2020-03-31 10:17:55.163 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
657833:2020-03-31 10:17:55.164 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
657834:2020-03-31 10:17:55.165 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657835:2020-03-31 10:17:55.166 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
657836:2020-03-31 10:17:55.167 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
658059:2020-03-31 10:18:55.297 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658060:2020-03-31 10:18:55.299 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
658061:2020-03-31 10:18:55.300 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
658062:2020-03-31 10:18:55.301 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658063:2020-03-31 10:18:55.302 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658064:2020-03-31 10:18:55.303 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
658286:2020-03-31 10:19:54.851 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658287:2020-03-31 10:19:54.852 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
658288:2020-03-31 10:19:54.853 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
658289:2020-03-31 10:19:54.854 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658290:2020-03-31 10:19:54.855 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658291:2020-03-31 10:19:54.856 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
658514:2020-03-31 10:20:55.528 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658515:2020-03-31 10:20:55.529 35 ERROR neutron.agent.linux.external_process [-] keepalived for router with uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1 not found. The process should not have died
658516:2020-03-31 10:20:55.530 35 WARNING neutron.agent.linux.external_process [-] Respawning keepalived for uuid 44cc73d8-fb3c-408b-be22-682931cb1fc1
658517:2020-03-31 10:20:55.531 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658518:2020-03-31 10:20:55.531 35 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp get_value_from_file /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:261
658519:2020-03-31 10:20:55.533 35 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-44cc73d8-fb3c-408b-be22-682931cb1fc1', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived', '-r', '/var/lib/neutron/ha_confs/44cc73d8-fb3c-408b-be22-682931cb1fc1.pid.keepalived-vrrp', '-D'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:87
root@uk-dc-cavium-01:~#

Revision history for this message

Michal Nasiadka (mnasiadka) wrote on 2023-10-02:

Stein EOL

Changed in kolla-ansible:
status:	New → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.