Set up elaborated EC2 security policy for new ci.linaro.org

[Paul Sokolovsky]

From: Loïc Minier <email address hidden>
Subject: Re: ci.linaro.org migrated to medium EC2 instance
Date: Tue, 27 Mar 2012 22:55:18 +0200

        Hey

 Alexander noticed that ci.linaro.org git jobs were failing and asked
 about security groups. Paul, I could try fixing this, but it's
 probably best if you fix this since you have all the steps fresh in
 your mind and Alexander said this could wait a day.

 Indeed it seems the new ci.linaro.org instance is only in
 jenkins-master and not in git-mirror like the "old ci.linaro.org" still
 is.

 What git-mirror gives is access to ports 3128, 3129, 8080 and 9418 to
 our others instances in the "default" security group; it also opens
 port 22 to the world.

 jenkins-master gives ports 22, 80 and 443.

 These groups were created a relatively long time ago in the hope that
 we'd have specific functions (such as web server, git server etc.) on
 each instance and that we'd launch instances with lists of functions
 they need to perform. However now that we have some longer history, I
 think the other choice would have been nicer: we could create a
 security group per service-role. e.g. ci.linaro.org-master,
 ci.linaro.org-slave, bip.linaro.org etc.

 So I would suggest that rather than relaunching ci.linaro.org with
 jenkins-master + git-mirror, we create a new ci.linaro.org-master sg
 with ports 22, 80, 443 open to the world and port 3128, 3129, 8080 and
 9418 open to our instances in the default sg. The main difference is
 that we stop reusing sg across different Linaro services (event if they
 look similar) and the big win is that we can change them the filters
 without rebooting the instances (changing a sg takes effect
 immediately, changing the sgs of an instance requires stop-ing it).

   Cheers,

On Thu, Mar 22, 2012, Paul Sokolovsky wrote:
> Hello,
>
> Infrastructure team yesterday finished migration of Linaro CI Jenkins
> service from i-c2f400ad (small instance) to i-4b36752f (medium
> instance). This happened verification that old instance had severe Java
> heap contention which led to system unresponsiveness and lock up with
> the amount of load it it started to process recently. Midium instance
> (2x RAM) allowed us double Java heap size and solved all the
> responsiveness issues.
>
> In the process of migration, we also created EBS volume
> for old instance, and then snapshotted/cloned volumes for new instance.
> "50.17.200.206" Elastic IP (pointed by "ci.linaro.org" DNS) was
> reassociated to the new instance.
>
> I'm going to submit updates for EC2 monitoring scripts to account for
> the new instance shortly. The old instance is expected to stay around
> for 1-2 weeks (TBD) before decommissioning.
>
> Please let me know if there're questions or suggestions regarding these
> changes.
>
>
> Thanks,
> Paul
>
> Linaro.org | Open source software for ARM SoCs
> Follow Linaro: http://www.facebook.com/pages/Linaro
> http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

--
Loïc Minier

[Milo Casagrande (milo)]

Paul, can you please take a look at this and update with the correct status?
Is it still valid or was it already implemented with latest Jenkins changes/updates?

[Paul Sokolovsky (pfalcon)]

Well, we don't have problems with current setup. There's definitely ways to improve it along the lines of Loic writing, but there're a lot more priority stuff to fix/improve. And the latest trend is people laughing and complaining of EC2 performance, so maybe we'll be getting more physical slaves instead. Setting as Wishlist, and actually can be closed as Wontfix IMHO.