© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
73416ba
(Get the code!)
OK, so some findings on this one.
So, this is a bug indeed, but there is no easy way to fix it properly. The problem happens due to a real collision when several concurrent deployments attempt to pick a cidr using a pseudo-random algorithm. At the default setting the probability of the conflict is ideally ~ 3% (in practise - worse, because the algorithm is not truly random), but it growths up significantly with the number of concurrent deployments (at 10 concurrent deployments it is about 80%)
We have the following options to consider:
1. increase the max_environments setting from 20 to 200 (or, actually, to anything from 128 to 255 - it will have the same effect): this will increase the number of available CIDRs in 8 times, so the probability of the collision will be much lower. But such solution cannot be considered as a real fix, as it does not eliminate the problem, it just makes it less probable to happen.
2. pre-allocate CIDRs by murano-api instead of murano engine. The API can synchronise the environment creation, as it has access to the database. This will be a real fix, i.e. it will prevent the conflicts from ever happening. Its cons: it requires us to store CIDRs as database columns (so we may efficiently fetch them), and that is the abstraction leak from the muranopl class of Environment into the DB model of murano-api, which is not very good from overall architecture point of view. However, this is a good approach to take in Mitaka and probably backport to stable/liberty as well.
3. Use some kind of distributed cache (Redis or similar) to synchornise the state between different engines. This will allow us to fix the issue right in place, without abstraction leakage. But this is a major architecture redesign (useful for lots of other good features) and is unlikely to be backported into Liberty branch (and also unlikely to happen in Mitaka as well, as we have quite a lengthy backlog already).
I propose to do p1 for Liberty and plan doing p2 for Mitaka, while keeping p3 in mind.