After restart ovs-vswitchd, the ports should still exist on the ovs bridges. So, you may show us the output of "ovs-vsctl show". All the ports of fip and qrouter should still be printed.
And, I guess, this looks like a container issue, the devices (qr-dev or fg-dev) are not added to namespace after restart the vswitchd container. When you restart the L3-agent, the devices will be reprocessed. So you can see it back to right namespace.
After restart ovs-vswitchd, the ports should still exist on the ovs bridges. So, you may show us the output of "ovs-vsctl show". All the ports of fip and qrouter should still be printed.
And, I guess, this looks like a container issue, the devices (qr-dev or fg-dev) are not added to namespace after restart the vswitchd container. When you restart the L3-agent, the devices will be reprocessed. So you can see it back to right namespace.