configure-resources has to be run twice to succeed

[Przemyslaw Hausman]

This is a fresh OpenStack deployment of bionic-stein. Octavia charm revision: 15.

Running configure-resources juju action for the first time always fails. Running it again, always succeeds.

FIRST RUN:

juju run-action --wait octavia/leader configure-resources
unit-octavia-0:
  UnitId: octavia/0
  id: "110"
  message: '''NoneType'' object is not iterable'
  results: {}
  status: failed
  timing:
    completed: 2020-03-04 20:05:45 +0000 UTC
    enqueued: 2020-03-04 20:05:36 +0000 UTC
    started: 2020-03-04 20:05:38 +0000 UTC

juju debug-log -i octavia/0
unit-octavia-0: 20:05:44 INFO unit.octavia/0.juju-log Created router 86ecbb45-6373-4d1f-a8ce-551bbc1e47fb
unit-octavia-0: 20:05:44 INFO unit.octavia/0.juju-log DEPRECATION WARNING: Function action_fail is being removed : moved to function_fail()

SECOND RUN:

juju run-action --wait octavia/leader configure-resources
unit-octavia-0:
  UnitId: octavia/0
  id: "111"
  results:
    Stderr: |
      Warning: Stopping systemd-networkd.service, but it can still be activated by:
        systemd-networkd.socket
    Stdout: |
      active
      active
      active
      active
  status: completed
  timing:
    completed: 2020-03-04 20:06:50 +0000 UTC
    enqueued: 2020-03-04 20:06:12 +0000 UTC
    started: 2020-03-04 20:06:15 +0000 UTC

juju debug-log -i octavia/0
unit-octavia-0: 20:06:28 INFO unit.octavia/0.juju-log Created port 4c575cfb-5efd-48d1-bada-4a43fd19e1cf
unit-octavia-0: 20:06:28 INFO unit.octavia/0.juju-log add OVS port
unit-octavia-0: 20:06:28 INFO unit.octavia/0.juju-log toggling port 4c575cfb-5efd-48d1-bada-4a43fd19e1cf (admin_state_up: False status: DOWN)
unit-octavia-0: 20:06:33 DEBUG unit.octavia/0.juju-log Writing file /etc/systemd/network/99-charm-octavia-o-hm0.network root:octavia 640
unit-octavia-0: 20:06:34 INFO unit.octavia/0.juju-log Making dir /etc/octavia/certs root:octavia 750
unit-octavia-0: 20:06:34 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33056 -> 288
unit-octavia-0: 20:06:35 INFO unit.octavia/0.juju-log Making dir /etc/octavia/certs root:octavia 750
unit-octavia-0: 20:06:35 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33056 -> 288
unit-octavia-0: 20:06:35 INFO unit.octavia/0.juju-log Making dir /etc/octavia/certs root:octavia 750
unit-octavia-0: 20:06:35 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33056 -> 288
unit-octavia-0: 20:06:35 INFO unit.octavia/0.juju-log Making dir /etc/octavia/certs root:octavia 750
unit-octavia-0: 20:06:35 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33056 -> 288
unit-octavia-0: 20:06:35 INFO unit.octavia/0.juju-log Making dir /etc/octavia/certs root:octavia 750
unit-octavia-0: 20:06:35 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33056 -> 288
unit-octavia-0: 20:06:36 DEBUG unit.octavia/0.juju-log Writing file /etc/octavia/octavia.conf root:octavia 640
unit-octavia-0: 20:06:36 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33184 -> 416
unit-octavia-0: 20:06:37 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33184 -> 416
unit-octavia-0: 20:06:37 WARNING unit.octavia/0.juju-log Not adding haproxy listen stanza for octavia-api_int port is already in use
unit-octavia-0: 20:06:37 WARNING unit.octavia/0.juju-log Not adding haproxy listen stanza for octavia-api_public port is already in use
unit-octavia-0: 20:06:37 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33184 -> 416
unit-octavia-0: 20:06:38 DEBUG unit.octavia/0.juju-log Changing permissions on existing content: 33184 -> 416
unit-octavia-0: 20:06:46 DEBUG unit.octavia/0.configure-resources Warning: Stopping systemd-networkd.service, but it can still be activated by:
unit-octavia-0: 20:06:46 DEBUG unit.octavia/0.configure-resources systemd-networkd.socket
unit-octavia-0: 20:06:49 DEBUG unit.octavia/0.configure-resources active
unit-octavia-0: 20:06:49 DEBUG unit.octavia/0.configure-resources active
unit-octavia-0: 20:06:49 DEBUG unit.octavia/0.configure-resources active
unit-octavia-0: 20:06:49 DEBUG unit.octavia/0.configure-resources active

[Przemyslaw Hausman (phausman)]

Before running configure-resources I configure OpenStack resources as follows:

# Get service project ID from service_domain
SERVICE_DOMAIN_ID=$(openstack domain list | \
  grep service_domain | awk '{print $2}')
PROJECT_ID=$(openstack project list --long | grep services | \
  grep ${SERVICE_DOMAIN_ID} | awk '{print $2}')

# Create management network
openstack network create --tag charm-octavia --project ${PROJECT_ID} \
  --provider-network-type gre lb-mgmt-net

# Create a subnet
openstack subnet create --tag charm-octavia --subnet-range 192.168.254.0/24 \
  --dhcp --network lb-mgmt-net --project ${PROJECT_ID} lb-mgmt-subnetv4

# Create Security Groups
openstack security group create --project ${PROJECT_ID} \
  --tag charm-octavia lb-mgmt-sec-grp
openstack security group create --project ${PROJECT_ID} \
--tag charm-octavia-health lb-health-mgr-sec-grp

# Configure Security Groups
openstack security group rule create --protocol icmp \
  --remote-ip 0.0.0.0/0 lb-health-mgr-sec-grp
openstack security group rule create --protocol udp \
  --remote-ip 0.0.0.0/0 lb-health-mgr-sec-grp
openstack security group rule create --protocol tcp \
  --remote-ip 0.0.0.0/0 lb-health-mgr-sec-grp
openstack security group rule create --protocol icmp \
  --remote-ip 0.0.0.0/0 lb-mgmt-sec-grp
openstack security group rule create --protocol udp \
  --remote-ip 0.0.0.0/0 lb-mgmt-sec-grp
openstack security group rule create --protocol tcp \
  --remote-ip 0.0.0.0/0 lb-mgmt-sec-grp

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/711873

[Przemyslaw Hausman (phausman)]

@pguimaraes noticed that I was missing "create-mgmt-network: False" in the config of the octavia charm. Adding this config option makes this issue go away.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711873
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=9e1e95a8daba0254268ebfca74cb7c82e1fc4e38
Submitter: Zuul
Branch: master

commit 9e1e95a8daba0254268ebfca74cb7c82e1fc4e38
Author: Frode Nordahl <email address hidden>
Date: Mon Mar 9 08:58:55 2020 +0100

    Log traceback on action failure

    The information logged by the charm is insuficcient in the event
    of an action failure.

    Change-Id: I0378220915dc49908ca841e58e0dc3869245accc
    Related-Bug: #1866126

[Billy Olsen (billy-olsen)]

I encountered this bug when deploying octavia as well; the traceback added in the previous comment captured the error quite clearly. It seems that if subnets exist but a router does not (at least in my case), the attempts code creates a router and attempts to iterate over the list of subnets which was not referenced.

Log excerpt:

2021-07-28 01:56:20 INFO juju-log Created router a2f71244-eee6-4697-b302-807346e56cbe
2021-07-28 01:56:20 ERROR juju-log action "configure-resources" failed: "'NoneType' object is not iterable" "Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/charm/actions/configure-resources", line 126, in main
    action(args)
  File "/var/lib/juju/agents/unit-octavia-0/charm/actions/configure-resources", line 62, in configure_resources
    (network, secgrp) = api_crud.get_mgmt_network(
  File "/var/lib/juju/agents/unit-octavia-0/charm/lib/charm/openstack/api_crud.py", line 628, in get_mgmt_network
    for subnet in subnets:
TypeError: 'NoneType' object is not iterable
"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/802651

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/802651
Committed: https://opendev.org/openstack/charm-octavia/commit/07b5dc8ea8d289bb514ae0125d6f0f25919cb078
Submitter: "Zuul (22348)"
Branch: master

commit 07b5dc8ea8d289bb514ae0125d6f0f25919cb078
Author: Billy Olsen <email address hidden>
Date: Tue Jul 27 20:34:31 2021 -0700

    Handle create_router when network and subnet exist

    Subnets were not assigned if they already existed and were not created,
    but the router creation code for the configure-resources depends on the
    subnets variable for iteration. Initialize the subnets from the query
    for the subnets for the case of existing subnets.

    Additionally, the octavia charm uses a service user which can see other
    subnets that are tagged 'charm-octavia'. Narrow the scope of the subnet
    listing to the network being used by the charm.

    Closes-Bug: #1866126
    Change-Id: Id88a68a398a50532e11daff33fb774caf65a0c0a
    Co-authored-by: Aurelien Lourot <email address hidden>