During the discussion around deduplication in mitigating missed actions during PE failover, we've been assuming that data comes from multiple sources (say DSDs spread over several DseNodes). But since we have adopted a design that puts all the DSDs on a single DseNode, we can sidestep a lot of the deduplication issues having to do with different PEs getting updates in different orders.
Basically, we can add a global sequencer at the DseNode which sequences all the data published from that node (currently we have independent sequencer per DSD).
The big advantage is it gives us a simple way to perfectly execute all action requests without any hiccup during a PE failover (see leaderless deduplication in spec).
With global update sequencing, two action requests are duplicates if and only if the action and args and the global sequence number of the update that triggered the action all match.
The main disadvantage is that this approach is not easy to scale out to having DSDs distributed across multiple PE nodes.
Any thoughts on whether to implement this leaderless deduplication via global sequencing in the DseNode?
During the discussion around deduplication in mitigating missed actions during PE failover, we've been assuming that data comes from multiple sources (say DSDs spread over several DseNodes). But since we have adopted a design that puts all the DSDs on a single DseNode, we can sidestep a lot of the deduplication issues having to do with different PEs getting updates in different orders.
Basically, we can add a global sequencer at the DseNode which sequences all the data published from that node (currently we have independent sequencer per DSD).
The big advantage is it gives us a simple way to perfectly execute all action requests without any hiccup during a PE failover (see leaderless deduplication in spec).
With global update sequencing, two action requests are duplicates if and only if the action and args and the global sequence number of the update that triggered the action all match.
The main disadvantage is that this approach is not easy to scale out to having DSDs distributed across multiple PE nodes.
Any thoughts on whether to implement this leaderless deduplication via global sequencing in the DseNode?