[RFE] ironic conductor - node affinity placement

[Sam Betts]

RFE https://bugs.launchpad.net/bugs/1734876 proposes adding some additional awareness to the ironic conductor hashring regarding which physical networks a node and conductor have access to.

However, this might not be enough to place nodes in a sane when in a large ironic deployment. This RFE proposes an arbitrary node to conductor placement method based on a set of tags.

Conductor will have a new configuration option which defines their specific tags.

Nodes will have a new field which defines their specific tags.

Placement will be done based on a best match system. When a node needs to be placed on a conductor, it will score the conductors based on the number of tags which match the node's tags. It will then place the node onto the conductor with the highest score. If there is more than one conductor with the same highest score, then hashring placement will be used to distribute the node between the conductors.

For example:

node1 tags: "rack10, network1, datacenter1"

conductor1: "rack10, rack11, network1, network2, datacenter1"

conductor2: "rack11, rack12, network1, datacenter1"

conductor3: "network1, datacenter2"

With this system conductor1 would be scored 3, conductor2 would be scored 2, conductor3 would be score 1. So node1 would be placed onto conductor1.

However say conductor1 dies, and the placement of the node's need to be reshuffled. Now conductor2 is the highest scorer and node1 is placed on the conductor2.

If we now add a new node:

node2 tags: "rack11"

conductor1 and conductor2 both score 1, and conductor3 scores 0. So node2 is hashring distributed among conductor1 and conductor2. Now if both conductors 1 and 2 die, and the hashring needs to be resuffled, even though conductor 3 scored 0, its still the highest scorer, so node2 is placed on the conductor3.

**Edit:**

Initial thoughts on anti-affinity:

Conductors will have another configuration option which defines "anti-tags"

During the placement, if a conductor's "anti-tags" match any of the node's tags that conductor will be removed from the placement process. Extending the above example:

conductor2 anti-tags: customer1

node3 tags: "rack11, customer1"

Just like node2, conductor1 and conductor2 both score 1, however this time conductor2's anti-tags match one of node3's tags "customer1". So conductor2 is removed from the placement selection.

This leaves only conductor1 with score of 1, so node3 is placed onto conductor1. If conductor 1 dies, and the hashring needs resuffling, then conductor2 scores 1, and conductor3 scores 0, but conductor2 is removed because it has matching anti-tags, so the node is placed onto conductor3.

**Edit2:**

After further thought here are some additional enhancements to this idea which
should make affinity and anti-affinity work better.

Conductors should have 4 new configuration options:

- hard-anti-affinity-tags
- hard-affinity-tags
- soft-anti-affinity-tags
- soft-affinity-tags

Nodes will have the new node.affinity-tags field as described above.

The conductor hard-anti-affinity-tags define which node.tags should *never* be
placed onto this conductor.

The conductor hard-affinity-tags define what tags a node *must* have to be
placed on this conductor.

The conductor soft-affinity and soft-anti-affinity tags contribute to the
conductors placement score, increasing or decreasing it respectively.

Psudo code for this is:

for node in nodes:
    valid_conductors_and_scores = {}
    for conductors in conductors:
        if any node.affinity_tags in conductor.hard_anti_affinity_tags:
            # node doesn't match conductor *must not have any* tags
            continue
        else if any conductor.hard_affinity_tags not in node.affinity_tags:
            # node doesn't match conductor *must have* tags
            continue
        else
            # Valid conductor but whats its score
            score = 0
            for tag in node.affinity_tags:
                if tag in conductor.soft_anti_affinity_tags:
                    score = score - 1
                elif tag in conductor.soft_affinity_tags:
                    score = score + 1
            valid_conductors_and_scores[conductor] = score
    highest_scoring_conductors = find_highest_scoring_conductors(valid_conductors_and_scores)
    hash_ring_place_node_onto_conductors(node, highest_scoring_conductors)

A working example:

node1-affinity-tags: "rack10, network1, datacenter1"

node2-affinity-tags: "rack20, network3, datacenter1"

node3-affinity-tags: "rack30, network1, datacenter2"

conductor1:
  hard-anti-affinity-tags:
  hard-affinity-tags: "datacenter1" # This conductor is located in datacenter1 so only place nodes in the same datacenter
  soft-anti-affinity-tags: "rack20" # This conductor is far away from rack20
  soft-affinity-tags: "rack10 network1 network2" # This conductor is close to rack10 and has direct access to network1 and network2

conductor2:
  hard-anti-affinity-tags: "network3" # This conductor doesn't have access to network3 at all
  hard-affinity-tags: "datacenter1" # This conductor is located in datacenter1 so only place nodes in the same datacenter
  soft-anti-affinity-tags: "rack10" # This conductor is far away from rack10
  soft-affinity-tags: "rack20 network1 network2" # This conductor is close to rack20 and has direct access to network1 and network2

conductor3:
  hard-anti-affinity-tags: "network2 network3" # This conductor doesn't have access to network3 or 2 at all
  hard-affinity-tags: "datacenter2" # This conductor is located in datacenter2 so only place nodes in the same datacenter
  soft-anti-affinity-tags:
  soft-affinity-tags: "rack30 network1 network2" # This conductor is close to rack10 and has direct access to network1

When placing node1:

Conductor3 is removed from selection because the conductor hard-affinity-tag
datacenter2 not in node.affinity_tags, leaving conductors 1 and 2. Conductors 1
and 2 are both valid, so then they get scored. Both have network1 so that gets
them a score of 1 each, but conductor2 has a soft-anti-affinity-tag to rack10,
so that earns it -1 from its score, resulting in a total score of 0. Conductor1
now wins with the highest score of 1 against conductor2s score of 0. So node1
is placed on conductor1.

When placing node2:

As before conductor3 is removed from selection. This time conductor2 is also
removed from selection because node2 has the network3 tag and condcutor2 is
hard-anti-affinity to network3. Leaving conductor1, which gets a score of -1
because the anti-affintiy rack20. Even thought it got a negative soft affinity
score, as its the only conductor left it has the highest score so node2 is
placed on conductor1.

When placing node3:

Condcutor1 and 2 are removed by the hard-affinity-tag datacenter2. So the node
is placed onto conductor3.