I was able to access the system (thanks Marius!) and here are some details:

- the bridge through which all traffic flows (br-infra) is indeed configured as a bridge mapping with the neutron ovs agent. This means that the agent will set fail_mode to secure, making it essential that the neutron ovs agent is running or things will start flowing as soon as ovs notices
- the agent itself looks to be have been relatively cleanly brought down but not brought back. 
- the yum update itself seems to have been hung from about the time the neutron ovs agent was updated and stopped, a quick grep for tripleo_upgrade_node.sh process on that node indicates that it too is hung. 
- now here is the pretty: the hang seems to be on a try-restart for openstack-nova-compute and it seems to be stuck because it cannot reach the rabbit server, which is listening on the internal API network, which is not functional because bridge on which that traffic flows is non-functional because the neutron ovs agent is not running.

I'm not sure what stops the neutron ovs agent without starting it again, but it appears to be linked with a check on ovs that ends up stopping ovs and the neutron agent and while it restarts ovs, it doesn't do the same for the neutron agent. 


Feb 14 16:14:56 localhost systemd: Started Session 38 of user heat-admin.
Feb 14 16:14:56 localhost systemd-logind: New session 38 of user heat-admin.
Feb 14 16:14:56 localhost systemd: Starting Session 38 of user heat-admin.
Feb 14 16:14:59 localhost os-collect-config: /var/lib/os-collect-config/local-data not found. Skipping
Feb 14 16:14:59 localhost os-collect-config: No local metadata found (['/var/lib/os-collect-config/local-data'])
Feb 14 16:15:07 localhost systemd: Reloading.
Feb 14 16:15:07 localhost systemd: [/usr/lib/systemd/system/microcode.service:10] Trailing garbage, ignoring.
Feb 14 16:15:07 localhost systemd: microcode.service lacks both ExecStart= and ExecStop= setting. Refusing.
Feb 14 16:15:07 localhost systemd: Stopping Open vSwitch Internal Unit...
Feb 14 16:15:07 localhost systemd: Stopping OpenStack Neutron Open vSwitch Agent...
Feb 14 16:15:07 localhost ovs-ctl: ovs-vswitchd is not running.
Feb 14 16:15:07 localhost ovs-ctl: ovsdb-server is not running.
Feb 14 16:15:07 localhost systemd: Stopped Open vSwitch Internal Unit.
Feb 14 16:15:08 localhost systemd: Stopped OpenStack Neutron Open vSwitch Agent.
Feb 14 16:15:08 localhost systemd: Stopping Open vSwitch...
Feb 14 16:15:08 localhost systemd: Stopped Open vSwitch.
Feb 14 16:15:08 localhost systemd: Cannot add dependency job for unit microcode.service, ignoring: Unit is not loaded properly: Invalid argument.
Feb 14 16:15:08 localhost systemd: Starting Open vSwitch Database Unit...
Feb 14 16:15:08 localhost ovs-ctl: Backing up database to /etc/openvswitch/conf.db.backup7.12.1-2211824403 [  OK  ]
Feb 14 16:15:08 localhost ovs-ctl: Compacting database [  OK  ]
Feb 14 16:15:08 localhost ovs-ctl: Converting database schema [  OK  ]
Feb 14 16:15:08 localhost ovs-ctl: Starting ovsdb-server [  OK  ]
Feb 14 16:15:08 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.14.0
Feb 14 16:15:08 localhost ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.6.1 "external-ids:system-i>
59-ca5eb2f08ed8\"" "external-ids:hostname=\"overcloud-novacompute-1.localdomain\"" "system-type=\"centos\"" "system-version=\"7\""
Feb 14 16:15:08 localhost ovs-ctl: Configuring Open vSwitch system IDs [  OK  ]
Feb 14 16:15:08 localhost systemd: Started Open vSwitch Database Unit.
Feb 14 16:15:08 localhost systemd: Starting Open vSwitch Forwarding Unit...
Feb 14 16:15:08 localhost kernel: device gre_sys left promiscuous mode
Feb 14 16:15:08 localhost kernel: device vxlan_sys_4789 left promiscuous mode
Feb 14 16:15:08 localhost kernel: device vxlan_sys_4789 entered promiscuous mode
Feb 14 16:15:08 localhost kernel: device gre_sys entered promiscuous mode
Feb 14 16:15:08 localhost ovs-ctl: Starting ovs-vswitchd [  OK  ]
Feb 14 16:15:08 localhost ovs-ctl: Enabling remote OVSDB managers [  OK  ]
Feb 14 16:15:08 localhost systemd: Started Open vSwitch Forwarding Unit.
Feb 14 16:15:08 localhost systemd: Starting Open vSwitch...
Feb 14 16:15:08 localhost systemd: Started Open vSwitch.