OVN metadata agent can be slow with large amount of subnets
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
High
|
Miro Tomaska |
Bug Description
OVN metadata agent can take very long time (observed ~40s) to add cidrs under a metadata namespace tap interface when a network consist of many subnets (observed ~1700 subnets). The long processing time can result in ovn-metada-agent not having haproxy ready by the time the first VM cloud-init requests for its metadata. Thus resulting in VM missing metadata for proper operation.
Reproducing step:
- Create a network with hundreds or thousands of subnets under this network. The more subnets the more obvious the problem is
- Create a VM connected to the network from above. Make sure this is the first VM on the deployed compute node(hypervisor).
- Once VM is created, observe that VM's cloud-init request time out due to no response from 169.256.
- Inspect ovn-metadata-agent log and notice this is due to ovn-metadata-agent taking very long time to process [1]
Possible solutions:
1. (Low hanging fruit?) See if there is a way to improve execution time of `ip.add` call. Perhaps passing a list of cidrs instead of a single cidr at the time can improve performance?
2. (more involved) refactor the code such that ovn-metadata-agent only adds a single cidr which belongs to the VM being created. Instead of unconditionally adding all cidrs for the network when the first VM is created(current implementation)
Changed in neutron: | |
assignee: | nobody → Miro Tomaska (mtomaska) |
description: | updated |
Changed in neutron: | |
importance: | Undecided → High |
status: | New → Confirmed |
Fix proposed to branch: master /review. opendev. org/c/openstack /neutron/ +/855677
Review: https:/