lb-manager role missing required right Microsoft.Network/networkSecurityGroups/read

[Drew Freiberger]

Action "Microsoft.Network/networkSecurityGroups/read" is needed for the EnsureLoadBalancer process within kube-controller-manager when running on Azure while using network security groups.

Please add this action into the lb-manager.json:
https://github.com/juju-solutions/charm-azure-integrator/blob/master/files/roles/lb-manager.json

Missing the configuration results in errors like:
  Warning UpdateLoadBalancerFailed 4m7s (x170 over 4h27m) service-controller Error updating load balancer with new hosts map[prd-aifactory-k8s-worker-vm-0:{} prd-aifactory-k8s-worker-vm-1:{} prd-aifactory-k8s-worker-vm-2:{} prd-aifactory-k8s-worker-vm-3:{} prd-aifactory-k8s-worker-vm-4:{} prd-aifactory-k8s-worker-vm-5:{} prd-aifactory-k8s-worker-vm-6:{} prd-aifactory-k8s-worker-vm-7:{}]: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '$clientuuid' with object id '$clientuuid' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/$MYSUBUUID/resourceGroups/$mynetwork/providers/Microsoft.Network/networkSecurityGroups/juju-internal-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

The charm continues to update this and reverts manual changes made to the role definition causing recurring production outages for loadbalancers and ends up flooding the API with too many calls to EnsureLoadBalancer.

[Drew Freiberger (afreiberger)]

Adding field-high as the charm reverts manual changes and affects LB creation/update and causes Azure API denial of service due to retries of failed LB updates.

[Drew Freiberger (afreiberger)]

Workaround:
Find the LB-manager role definition:

az role definition list -o table|grep juju-charm-lb-manager
az role definition list --name $role-name-uuid --output json > lb-role.json
vi lb-role.json
  # remove the surrounding [ ] list brackets from top and bottom lines
  # add a comma after the last Actions section item, and add this line within Actions:
  "Microsoft.Network/networkSecurityGroups/read"
  # save the file and quit vi
az role definition update --role-definition ./lb-role.json

[Drew Freiberger (afreiberger)]

PR submitted for review. https://github.com/juju-solutions/charm-azure-integrator/pull/31

[George Kraft (cynerva)]

Cherry-picked to stable branch: https://github.com/charmed-kubernetes/charm-azure-integrator/commit/03cb276452aecc8361d28e1ab765d069206e38bd

[Cory Johns (johnsca)]

This was released on May 28th with azure-integrator rev 97.