Juju is unreliable on Joyent because of undeleted firewalls

[Curtis Hovey]

CI has observed several cases where Juju on Joyent is reliable, but reliability and repeatability decline after a week. Manually deleting firewalls after we teardown the env appears to restore reliability.

Joyent firewalls were introduced after we created the joyent-provider. After the last upgrade to Joyent cloud, the feature is now always on in. We saw Joyent become the most reliable to deploy on. We also noted that we were getting successes when we had mixed networks. Then over a week we saw that that service machine could not contact the state server to download agents. We can also see that all machines were on same 72.* or 165.* network. Our own personal experience with Joyent showed it was still very reliable.

We discovered in just a week, CI had added 1000 firewall rules. We deleted all the rules, Joyent was better for a time. We then added a rule to several CI jobs to delete firewalls after destroy-environment. CI appears to be very happy.

If Joyent continues to be happy without an accumulation of firewall rules, I think we need to update juju to alway delete firewalls when destroying an env to ensure repeatability.

[Curtis Hovey (sinzui)]

We have enough evidence to conclude that deleting firewalls makes juju reliable:
This Week:
    95% success rate for series and bundles
    precise is 100% reliable
    The 10% bundle failures were caused by charms
    No machine agents failed to download
    No actions were take to prevent 72.* addresses.

Last week:
    40% success rate for all series and bundles.
    precise was 25% reliable
    bundles failed 30% of the time because agents failed to download
    80% of failures were because agents cold not be downloaded from the state-server
    We kept 72.* addresses tied to running machines to keep them out of tests

Joyent's firewall rules left beta earlier this year. AS major update to the regions happened in the last 4 weeks. We saw an immediate improvement after the updates, but reliability declined until we started manually deleting firewall rules tagged with the Juju env.

[Mark Ramm (mark-ramm)]

These kinds of "failure to tear down" issues have been discussed as something which should have CI tests to check that everything is torn down properly, and would then curse the build.

[Tim McNamara (tim-clicks)]

Marking as wont-fix as we have removed support for Joyent's public cloud.