gce: bootstrap instance has no network rule for API

[Andrew Wilkins]

I've just bootstrapped the GCE provider on master (0f0986bb9771f974bcb52f1cf9543e5f9f521026). Bootstrap succeeds, but then "juju status" hangs as the API server port cannot be connected to.

I investigated in the GCE console, and found that the instance that was created was associated with the "default" network, and for that network there was no rule allowing port 17070.

[Andrew Wilkins (axwalk)]

While destroying the environment, I got:

ERROR while removing instance "juju-9eddd2c8-55dc-4e48-89a0-99971c9bf984-machine-0": googleapi: Error 404: The resource 'projects/sunlit-inquiry-89505/global/firewalls/juju-9eddd2c8-55dc-4e48-89a0-99971c9bf984-machine-0' was not found, notFound
ERROR some instance removals failed: [juju-9eddd2c8-55dc-4e48-89a0-99971c9bf984-machine-0]

may well be related.

[Dimiter Naydenov (dimitern)]

I signed up for a GCE free trial account.

After some trial and error, I managed to bootstrap an environment with 1.23 successfully.

I was able to reproduce the issue (bootstrap succeeds, then juju status hangs).

Looking the GCE web console, I can see a firewall rule added for the instance: source 0.0.0.0/0, tcp:17017

So the provider does add a rule, but for some reason it's not taking effect.
I'm still investigating.

[Dimiter Naydenov (dimitern)]

I found the problem and proposed a fix for it - http://reviews.vapour.ws/r/1282/

In summary, the issue is that the firewall rule for the api port (17070) is getting created as an instance-specific rule, unlike the default global rules GCE creates for each project (default-allow-icmp, default-allow-ssh, default-allow-internal).

Because the rule about opening 17070/tcp is an instance rule, rather than environment-level rule, this interacts badly with the firewaller when the default FwInstance mode is used.

I can see in the machine-0.log that once the firewaller starts, it almost immediately closes the 17070/tcp port on machine 0. While analyzing why this happens and why it's not affecting other providers I realized that's some fallout from the port ranges work, which I'll file a separate bug about.

Basically, port 17070/tcp is not opened of machine 0 in state, and the firewaller initially compares what ports a machine needs (based on what units are assigned to it and whether any of them called OpenPorts), then computes the difference between what it can see in state and what it can get from the provider and opens/closes ports to equalize one with the other. More details and logs will be added to the related bug I'll file.

So my fix for GCE is to just always create a global firewall rule (named after the environment UUID + "juju-" prefix) for port 17070/tcp, rather than a rule just for machine 0. Live tested to work.

[Dimiter Naydenov (dimitern)]

Fix ported to master as well - http://reviews.vapour.ws/r/1283/

[Eric Snow (ericsnowcurrently)]

Why can't the firewaller know not to close the API port on a state server? If it goes strictly by what ports are indicated for the machine in the state DB then why isn't the API port added already?

I would think that the firewaller is making an Instance.Ports call on each of the instances returned by Environ.StateServerInstances for a given provider. If the firewaller isn't the right place for that then I'd expect it to happen as part of bootstrap and the result stored in state for the machine, where the firewaller would later see it.

This implies that that the provider's bootstrap handling is responsible for opening the state ports it needs (i.e. the status quo). That means each provider must build in it's own logic for something they all have to do. The better approach would be to have this handled in the provider/common code. Better yet, at bootstrap the state-server-related ports would be recorded in the DB for the new state server machine, providers wouldn't have to worry about it, and the firewaller can just keep doing what it already does. :)

Of course the same considerations would apply for ensure-availability as for bootstrap.

[Andrew Wilkins (axwalk)]

The way the Azure provider does this is by just not reporting the API port in its Instance.Ports method. That way, the firewaller never thinks it needs to close the port, and API ports are opened only for state server instances.

[Dimiter Naydenov (dimitern)]

Yes, the provider is responsible for making sure the API server port is open. That haven't changed, but I agree it can be improved.

How ports management and firewalling work will need to change and there are steps to get there, which I'll try to summarize in the bug I'm about to file about it.