On juju upgrade the security group lost ports for the exposed services

[Tim Kuhlman]

After a recent Juju upgrade from 1.20.14-> 1.22.8 which seemed to go flawless the site was down. Upon investigation the security group port 80 and 443 for the exposed apache2 service even though the service still said it was exposed. Manually adding in the ports to the security group fixed the site.

This is a view of the working security group rules, the rules besides port 80 and 443 were retained:
(mojo-how-to-ps45)prodstack-is@wekufe:~$ nova secgroup-list-rules juju-prod45-is-mojo-how-to
+-------------+-----------+---------+-----------------+----------------------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------------+----------------------------+
| tcp | 22 | 22 | 10.172.126.0/23 | |
| icmp | -1 | -1 | 0.0.0.0/0 | |
| tcp | 37017 | 37017 | 0.0.0.0/0 | |
| udp | 1 | 65535 | | juju-prod45-is-mojo-how-to |
| tcp | 22 | 22 | 91.189.90.46/32 | |
| tcp | 1 | 65535 | | juju-prod45-is-mojo-how-to |
| tcp | 5666 | 5666 | 91.189.90.46/32 | |
| tcp | 80 | 80 | 0.0.0.0/0 | |
| icmp | -1 | -1 | | juju-prod45-is-mojo-how-to |
| tcp | 22 | 22 | 10.172.254.0/23 | |
| tcp | 22 | 22 | 10.172.62.0/23 | |
| tcp | 22 | 22 | 10.172.192.0/23 | |
| tcp | 17070 | 17070 | 0.0.0.0/0 | |
| tcp | 443 | 443 | 0.0.0.0/0 | |
| tcp | 873 | 873 | 91.189.90.46/32 | |
+-------------+-----------+---------+-----------------+----------------------------+

[Tim Kuhlman (timkuhlman)]

Oddly the all-machines hasn't had any updates to it for 3 days. I attached the machine-0.log which does have data from the update.

[Martin Packman (gz)]

It thinks it works, but this upgrade step may well have screwed up:

2015-10-15 17:15:28 INFO juju.upgrade upgrade.go:132 running upgrade step: migrate individual unit ports to openedPorts collection
2015-10-15 17:15:28 INFO juju.state.upgrade upgrades.go:305 migrating legacy ports to port ranges for all 14 units
2015-10-15 17:15:28 INFO juju.state.upgrade upgrades.go:308 migrating ports for unit "apache2/0"
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:309 raw ports for unit "apache2/0": [80/tcp 443/tcp]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:116 merged raw port ranges for unit "apache2/0": [80/tcp 443/tcp]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:156 unit "apache2/0" valid merged ranges: [80-80/tcp ("apache2/0") 443-443/tcp ("apache2/0")]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:321 unit "apache2/0" assigned to machine "1"
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:333 existing port ranges for unit "apache2/0"'s machine "1": map[]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:245 unit "apache2/0" port ranges to migrate: map[80-80/tcp ("apache2/0"):true 443-443/tcp ("apache2/0"):true]
2015-10-15 17:15:28 INFO juju.state.upgrade upgrades.go:357 unit "apache2/0"'s ports (ranges) migrated: total 2(2); ok 2(2); skipped 0(0)
2015-10-15 17:15:28 INFO juju.state.upgrade upgrades.go:308 migrating ports for unit "apache2/1"
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:309 raw ports for unit "apache2/1": [80/tcp 443/tcp]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:116 merged raw port ranges for unit "apache2/1": [80/tcp 443/tcp]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:156 unit "apache2/1" valid merged ranges: [80-80/tcp ("apache2/1") 443-443/tcp ("apache2/1")]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:321 unit "apache2/1" assigned to machine "2"
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:333 existing port ranges for unit "apache2/1"'s machine "2": map[]
2015-10-15 17:15:28 DEBUG juju.state.upgrade upgrades.go:245 unit "apache2/1" port ranges to migrate: map[80-80/tcp ("apache2/1"):true 443-443/tcp ("apache2/1"):true]
2015-10-15 17:15:28 INFO juju.state.upgrade upgrades.go:357 unit "apache2/1"'s ports (ranges) migrated: total 2(2); ok 2(2); skipped 0(0)

[Gareth Woolridge (moon127)]

This looks to have affected us on another environment also upgraded from 1.20.14 to 1.24.7.

Symptoms being missing secgroup rules for the exposed apache2 service.

Manually worked around this by manually adding secgroup rules as juju unexpose apache2; juju expose apache2 toggles in juju status output but does not update nova secgroups.

Also note, restarting jujud-machine-0 removes the secgroup rules again despite juju status showing the service as exposed.

juju status --format tabular
[Services]
NAME STATUS EXPOSED CHARM
apache2 true local:trusty/apache2-0
content-fetcher false local:trusty/content-fetcher-0
ksplice false local:trusty/ksplice-3
landscape-client false local:trusty/landscape-client-15
nrpe-external-master false local:trusty/nrpe-external-master-7
ubuntu-basenode false local:trusty/ubuntu-basenode-1

[Units]
ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
apache2/0 idle 1.24.7 1 162.213.33.177
  content-fetcher/1 idle 1.24.7 162.213.33.177
  ksplice/0 idle 1.24.7 162.213.33.177
  landscape-client/4 idle 1.24.7 162.213.33.177
  nrpe-external-master/0 idle 1.24.7 162.213.33.177
apache2/1 idle 1.24.7 2 162.213.33.178
  content-fetcher/0 idle 1.24.7 162.213.33.178
  ksplice/1 idle 1.24.7 162.213.33.178
  landscape-client/5 idle 1.24.7 162.213.33.178
  nrpe-external-master/1 idle 1.24.7 162.213.33.178
ubuntu-basenode/1 idle 1.24.7 0 10.25.12.102
  ksplice/2 idle 1.24.7 10.25.12.102
  landscape-client/3 idle 1.24.7 10.25.12.102
  nrpe-external-master/2 idle 1.24.7 10.25.12.102

[Machines]
ID STATE VERSION DNS INS-ID SERIES HARDWARE
0 started 1.24.7 10.25.12.102 5bc09c81-680b-4352-a1be-6dfb4d4dd3c3 trusty arch=amd64 cpu-cores=1 mem=2048M root-disk=10240M availability-zone=prodstack-zone-1
1 started 1.24.7 162.213.33.177 a31edd0f-e6a5-4738-b5fa-7d437b195798 trusty arch=amd64 cpu-cores=1 mem=2048M root-disk=10240M availability-zone=prodstack-zone-2
2 started 1.24.7 162.213.33.178 ad91da6c-9d80-449c-aa4f-da5f2d4d67c6 trusty arch=amd64 cpu-cores=1 mem=2048M root-disk=10240M availability...

[Gareth Woolridge (moon127)]

machine-0.log - 2015-11-18 20:16 looks to be when the juju upgrade occurred.

[Tim Kuhlman (timkuhlman)]

Saw this again in another environment, I manually added the rules for machines 1 and 2.
(prod45-ubuntu-assets-manager)prodstack-comms@wekufe:~/juju-upgrade$ juju status --format tabular
[Services]
NAME STATUS EXPOSED CHARM
apache2-assets-manager true local:trusty/apache2-0
assets-manager false local:trusty/wsgi-app-2
assets-manager-haproxy false local:trusty/haproxy-28
gunicorn false local:trusty/gunicorn-4
ksplice false local:trusty/ksplice-3
landscape-client false local:trusty/landscape-client-15
nrpe false local:trusty/nrpe-external-master-7
postgresql false local:trusty/postgresql-0
ubuntu-basenode false local:trusty/ubuntu-basenode-2

[Units]
ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
apache2-assets-manager/0 idle 1.24.7 1 162.213.33.119
  ksplice/1 idle 1.24.7 162.213.33.119
  landscape-client/5 idle 1.24.7 162.213.33.119
  nrpe/1 idle 1.24.7 162.213.33.119
apache2-assets-manager/1 idle 1.24.7 2 162.213.33.120
  ksplice/2 idle 1.24.7 162.213.33.120
  landscape-client/4 idle 1.24.7 162.213.33.120
  nrpe/0 idle 1.24.7 162.213.33.120
assets-manager-haproxy/0 idle 1.24.7 5 10.25.12.124
  ksplice/7 idle 1.24.7 10.25.12.124
  landscape-client/6 idle 1.24.7 10.25.12.124
  nrpe/3 idle 1.24.7 10.25.12.124
assets-manager-haproxy/1 idle 1.24.7 6 10.25.12.125
  ksplice/6 idle 1.24.7 10.25.12.125
  landscape-client/7 idle 1.24.7 10.25.12.125
  nrpe/2 idle 1.24.7 10.25.12.125
assets-manager/0 idle 1.24.7 3 8080/tcp 10.25.12.122
  gunicorn/0 idle 1.24.7 10.25.12.122
  ksplice/4 idle 1.24.7 10.25.12.122
  landscape-client/1 idle 1.24.7 10.25.12.122
  nrpe/4 idle 1.24.7 10.25.12.122
assets-manager/1 idle 1.24.7 4 8080/tcp 10.25.12.123
  gunicorn/1 idle 1.24.7 10.25.12.123
  ksplice/3 idle 1.24.7 10.25.12.123
  landscape-client/3 idle 1.24.7 10.25.12.12...

[Tim Kuhlman (timkuhlman)]

prod45-ubuntu-assests-manager machine-0.log

[Tim Kuhlman (timkuhlman)]

Saw it on another environment, this time for haproxy rather than apache.
(prod45-ubuntu-china)prodstack-comms@wekufe:~/juju-upgrade$ juju status --format tabular
[Services]
NAME STATUS EXPOSED CHARM
gunicorn false local:trusty/gunicorn-12
ksplice false local:trusty/ksplice-9
landscape false local:trusty/landscape-client-23
nrpe false local:trusty/nrpe-15
swift-log-archive false local:trusty/swift-log-archive-89
ubuntu-basenode false local:trusty/ubuntu-basenode-5
ubuntu-china-app false local:trusty/wsgi-app-16
ubuntu-china-haproxy true local:trusty/haproxy-42

[Units]
ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
ubuntu-basenode/0 idle 1.24.7 0 10.25.12.59
  ksplice/4 idle 1.24.7 10.25.12.59
  landscape/5 idle 1.24.7 10.25.12.59
ubuntu-china-app/0 idle 1.24.7 1 8080/tcp 10.25.12.71
  gunicorn/1 idle 1.24.7 10.25.12.71
  ksplice/1 idle 1.24.7 10.25.12.71
  landscape/9 idle 1.24.7 10.25.12.71
  nrpe/1 idle 1.24.7 10.25.12.71
  swift-log-archive/1 idle 1.24.7 10.25.12.71
ubuntu-china-app/1 idle 1.24.7 2 8080/tcp 10.25.12.72
  gunicorn/0 idle 1.24.7 10.25.12.72
  ksplice/0 idle 1.24.7 10.25.12.72
  landscape/8 idle 1.24.7 10.25.12.72
  nrpe/3 idle 1.24.7 10.25.12.72
  swift-log-archive/0 idle 1.24.7 10.25.12.72
ubuntu-china-haproxy/8 idle 1.24.7 11 162.213.33.107
  ksplice/3 idle 1.24.7 162.213.33.107
  landscape/6 idle 1.24.7 162.213.33.107
  nrpe/0 idle 1.24.7 162.213.33.107
  swift-log-archive/2 idle 1.24.7 162.213.33.107
ubuntu-china-haproxy/9 idle 1.24.7 12 162.213.33.108
  ksplice/2 idle 1.24.7 162.213.33.108
  landscape/7 idle 1.24.7 162.213.33.108
  nrpe/2 idle 1.24.7 162.213.33.108
  swift-log-archive/3 idle 1.24.7 162.213.33.108

[Machines]
ID STATE VERSION DNS INS-ID SERIES HARDWARE ...

[Tim Kuhlman (timkuhlman)]

I tried restarting the jujud-machine-0 daemon for ubuntu-asset-manager and confirmed that is enough to drop the security group rules.

[Tim Kuhlman (timkuhlman)]

Here is the machine-0.log from ubuntu-assets-manager when I restared jujud

[Martin Packman (gz)]

Looks like this change is aimed at fixing this:

<https://github.com/juju/juju/pull/3894>

[Cheryl Jennings (cherylj)]

Users will be required to upgrade through 1.25.2 before going to 2.0, so this upgrade step doesn't need to be included in 2.0.