Add Monitoring for amphora failing to build due to quota/IP exhaustion

[Drew Freiberger]

There are several failure scenarios when a site has multiple loadbalancers deployed on openstack (for instance, a cloud with k8s on top of openstack) where loadbalancers fail to create due to either IP exhaustion on the chosen VIP network, or quota limitations on the services project (see lp#1850985.

The octavia API interface doesn't expose these reasons for LB creation failure, but they're easily found within the octavia logs and should be scraped by nagios for operator notification.

I'll include some sample errors in the comments of this bug for inclusion in these checks.

[Drew Freiberger (afreiberger)]

In the octavia-worker.log:

Quota exceeded message for RAM:

|__Flow 'octavia-create-loadbalancer-flow': octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: Quota exceeded for ram: Requested 4096, but already used 49152 of 51200 ram (HTTP 403) (Request-ID: req-xyz)

Quota Exhaustion for vCPU and Instances:

                                         |__Flow 'octavia-create-loadbalancer-flow': octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: Quota exceeded for cores, instances: Requested 2, 1, but already used 20, 10 of 20, 10 cores, instances (HTTP 403) (Request-ID: req-55ed3f13-03ad-4312-8d19-7cf087afcc05)

Searching for "Quota exceeded for (.*):" and reporting the (.*) quota buckets in the check would be quite useful.

Re: VIP Exhaustion:

2020-04-29 19:55:11.946 12954 ERROR octavia.network.drivers.neutron.allowed_address_pairs [req-f4c079ef-a110-45f4-8445-0363d98a7d37 - d4605f6a55ad47fabc406331ac0d94b2 - - -] Error creating the base (VRRP) port for the VIP with port details: {'port': {'name': 'octavia-lb-vrrp-7c9b12a6-3245-482c-87e0-62e1036ba0c1', 'network_id': '6f05a9de-4fc9-41f5-9c51-d5f43cd244b9', 'fixed_ips': [{'subnet_id': '4ab2ec1c-50bf-4d13-9fcc-dede97e5178c'}], 'admin_state_up': True, 'device_owner': 'Octavia'}}: neutronclient.common.exceptions.IpAddressGenerationFailureClient: No more IP addresses available on network 6f05a9de-4fc9-41f5-9c51-d5f43cd244b9.

Searching for "No more IP addresses available on network (.*)\." and reporting the matching network ID that has IP exhaustion would be helpful for the operator.

[Drew Freiberger (afreiberger)]

This may be something to be addressed by a grok-exporter or graylog feed into alertmanager.