Configurable persistent backing storage

[Dustin Kirkland ]

Following on the discussion on the mailing list, about storage:
 * https://lists.ubuntu.com/archives/ensemble/2011-July/000223.html
I'm filing this bug.

There are many, many difference configurations and considerations, so this bug might be split into multiple bugs or blueprints.

At a basic level, the default deployment of most current Ensemble formulas is rather ephemeral. Ensemble provides quite a nice way to deploy and relate services quickly and reliably. But there is no native support for persistence. It's up to the formula writer/user to manually configure persistent storage (EBS in the AWS case).

Jorge writes:

$ ensemble deploy --respository=examples amazon-ebs
$ ensemble add-relation mysql amazon-ebs

I would then treat "amazon-ebs" as any other resource and set up
relationships to the storage and do other things like:

$ ensemble snapshot // Point in time snapshot the entire thing to S3
$ ensemble backup blah // Prompt me to snag my config and EBS stuff
so I can keep it offsite.
$ ensemble import blah // Import the configs and EBS stuff back to EC2

Of course EBS is just one storage thing that is obvious, I'd want to
be able to treat a local SAN the same way, etc. Thoughts?

[Dustin Kirkland  (kirkland)]

I personally like the idea of treating "storage" as just another service that Ensemble is able to manage and relate. This would also allow many different types of storage to be supported as different implementations of a generic storage service.

[Clint Byrum (clint-fewbar)]

EBS volumes can only have one instance attached at one time, so I'm not sure they are a good analog for services which can be related to many things. This is why I think storage is fundamental, and needs to be modeled at the provider level.

Until then, a stop gap would be to add an ensemble sub-command to pre-create a machine, then use this branch:

https://code.launchpad.net/~clint-fewbar/ensemble/reuse-machines/+merge/67417

To assign a deployed service to the machine after you have resized its EBS root or attached any extra EBS volumes to it.

[Dustin Kirkland  (kirkland)]

Howdy, Ensemblers! Could anyone speak to the viability of fixing this bug, or providing a suitable workaround for 11.10?

[Dustin Kirkland  (kirkland)]

Note that this is similar, and possibly duplicate to Bug #830999.

[Kapil Thangavelu (hazmat)]

Excerpts from Dustin Kirkland's message of Thu Sep 01 22:06:02 UTC 2011:
> Howdy, Ensemblers! Could anyone speak to the viability of fixing this
> bug, or providing a suitable workaround for 11.10?
>

We won't have volume management addressed with 11.10. Currently all instances
launched by ensemble against ec2 are using ebs backed volumes, so there is some
notion of data persistence, just completely unexposed via ensemble.

I'd ideally like to see a volume per unit, against environment/provider level
volume management services [1, 2]. Ideally we could drive physical installations
towards openstack/orchestra and utilize the volume capabilities available there.

[1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/4
[1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/5

[Juan L. Negron (negronjl)]

Are you guys sure about the ebs backed instances?

I explain ....

Currently for oneiric, I am using ami-f1448498 which is an instance store
ami so, am I missing something here or is ensemble by "default" using
ebs-root instances and I just changed them to instance store ones ?

If indeed I am using instance-store images that are not being backed
anywhere, it would be helpful to document this somewhere so not to confuse
users.

Thanks,

Juan

On Thu, Sep 1, 2011 at 4:24 PM, Kapil Thangavelu
<email address hidden>wrote:

> Excerpts from Dustin Kirkland's message of Thu Sep 01 22:06:02 UTC 2011:
> > Howdy, Ensemblers! Could anyone speak to the viability of fixing this
> > bug, or providing a suitable workaround for 11.10?
> >
>
> We won't have volume management addressed with 11.10. Currently all
> instances
> launched by ensemble against ec2 are using ebs backed volumes, so there is
> some
> notion of data persistence, just completely unexposed via ensemble.
>
> I'd ideally like to see a volume per unit, against environment/provider
> level
> volume management services [1, 2]. Ideally we could drive physical
> installations
> towards openstack/orchestra and utilize the volume capabilities available
> there.
>
> [1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/4
> [1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/5
>
> --
> You received this bug notification because you are a member of
> Canonical-SIG, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/811226
>
> Title:
> Configurable persistent backing storage
>
> Status in Ensemble:
> Confirmed
>
> Bug description:
> Following on the discussion on the mailing list, about storage:
> * https://lists.ubuntu.com/archives/ensemble/2011-July/000223.html
> I'm filing this bug.
>
> There are many, many difference configurations and considerations, so
> this bug might be split into multiple bugs or blueprints.
>
> At a basic level, the default deployment of most current Ensemble
> formulas is rather ephemeral. Ensemble provides quite a nice way to
> deploy and relate services quickly and reliably. But there is no
> native support for persistence. It's up to the formula writer/user to
> manually configure persistent storage (EBS in the AWS case).
>
> Jorge writes:
>
> $ ensemble deploy --respository=examples amazon-ebs
> $ ensemble add-relation mysql amazon-ebs
>
> I would then treat "amazon-ebs" as any other resource and set up
> relationships to the storage and do other things like:
>
> $ ensemble snapshot // Point in time snapshot the entire thing to S3
> $ ensemble backup blah // Prompt me to snag my config and EBS stuff
> so I can keep it offsite.
> $ ensemble import blah // Import the configs and EBS stuff back to EC2
>
> Of course EBS is just one storage thing that is obvious, I'd want to
> be able to treat a local SAN the same way, etc. Thoughts?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ensemble/+bug/811226/+subscriptions
>
>

[Kapil Thangavelu (hazmat)]

Hi Juan,

your right, manually selecting an ephemeral storage ami-id in environments.yaml
config will bypass ensemble picking an ebs ami-id.

cheers,

Kapil

Excerpts from Juan L. Negron's message of Fri Sep 02 00:09:16 UTC 2011:
> Are you guys sure about the ebs backed instances?
>
> I explain ....
>
> Currently for oneiric, I am using ami-f1448498 which is an instance store
> ami so, am I missing something here or is ensemble by "default" using
> ebs-root instances and I just changed them to instance store ones ?
>
> If indeed I am using instance-store images that are not being backed
> anywhere, it would be helpful to document this somewhere so not to confuse
> users.
>
>
> Thanks,
>
> Juan
>
>
> On Thu, Sep 1, 2011 at 4:24 PM, Kapil Thangavelu
> <email address hidden>wrote:
>
> > Excerpts from Dustin Kirkland's message of Thu Sep 01 22:06:02 UTC 2011:
> > > Howdy, Ensemblers! Could anyone speak to the viability of fixing this
> > > bug, or providing a suitable workaround for 11.10?
> > >
> >
> > We won't have volume management addressed with 11.10. Currently all
> > instances
> > launched by ensemble against ec2 are using ebs backed volumes, so there is
> > some
> > notion of data persistence, just completely unexposed via ensemble.
> >
> > I'd ideally like to see a volume per unit, against environment/provider
> > level
> > volume management services [1, 2]. Ideally we could drive physical
> > installations
> > towards openstack/orchestra and utilize the volume capabilities available
> > there.
> >
> > [1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/4
> > [1] https://bugs.launchpad.net/ensemble/+bug/806241/comments/5
> >
> > --
> > You received this bug notification because you are a member of
> > Canonical-SIG, which is subscribed to the bug report.
> > https://bugs.launchpad.net/bugs/811226
> >
> > Title:
> > Configurable persistent backing storage
> >
> > Status in Ensemble:
> > Confirmed
> >
> > Bug description:
> > Following on the discussion on the mailing list, about storage:
> > * https://lists.ubuntu.com/archives/ensemble/2011-July/000223.html
> > I'm filing this bug.
> >
> > There are many, many difference configurations and considerations, so
> > this bug might be split into multiple bugs or blueprints.
> >
> > At a basic level, the default deployment of most current Ensemble
> > formulas is rather ephemeral. Ensemble provides quite a nice way to
> > deploy and relate services quickly and reliably. But there is no
> > native support for persistence. It's up to the formula writer/user to
> > manually configure persistent storage (EBS in the AWS case).
> >
> > Jorge writes:
> >
> > $ ensemble deploy --respository=examples amazon-ebs
> > $ ensemble add-relation mysql amazon-ebs
> >
> > I would then treat "amazon-ebs" as any other resource and set up
> > relationships to the storage and do other things like:
> >
> > $ ensemble snapshot // Point in time snapshot the entire thing to S3
> > $ ensemble backup blah // Prompt me to snag my config and EBS stuff
> > so I can keep it offsite.
> > $ ensemble import blah // Import the configs and EBS stuff back to EC2
> >
> > Of c...

[Clint Byrum (clint-fewbar)]

Excerpts from Juan L. Negron's message of Fri Sep 02 00:09:16 UTC 2011:
> Are you guys sure about the ebs backed instances?
>
> I explain ....
>
> Currently for oneiric, I am using ami-f1448498 which is an instance store
> ami so, am I missing something here or is ensemble by "default" using
> ebs-root instances and I just changed them to instance store ones ?
>
> If indeed I am using instance-store images that are not being backed
> anywhere, it would be helpful to document this somewhere so not to confuse
> users.
>

Agreed Juan, its just "by default" using EBS backed AMI's. If you switch,
you have no safety net.

Also it turns out its not that much of a safety net anyway. The "big
meltdown" on EC2 a few months ago happened because people were using
snapshots as backups. But snapshots live in the same availability zone
as your instance, so you may not be able to reach the snapshot when your
instance has died (as many people found out when they tried to enact
their disaster recovery plan of pulling their snapshots onto other hosts).

Its ok though, we can simply make it clear that Ensemble doesn't take
care of this for you, and you need to manage your own data storage and
backups. Having co-located formulas so people can roll out a backup
service to all their nodes would be a nice start at this.

[Clint Byrum (clint-fewbar)]

Excerpts from Kapil Thangavelu's message of Thu Sep 01 23:24:22 UTC 2011:
> Excerpts from Dustin Kirkland's message of Thu Sep 01 22:06:02 UTC 2011:
> > Howdy, Ensemblers! Could anyone speak to the viability of fixing this
> > bug, or providing a suitable workaround for 11.10?
> >
>
> We won't have volume management addressed with 11.10. Currently all instances
> launched by ensemble against ec2 are using ebs backed volumes, so there is some
> notion of data persistence, just completely unexposed via ensemble.
>
> I'd ideally like to see a volume per unit, against environment/provider level
> volume management services [1, 2]. Ideally we could drive physical installations
> towards openstack/orchestra and utilize the volume capabilities available there.
>

Its more than that though. There's the notion of available space
too. Having a single instance of Cassandra that can only put 10G of data
in it is pretty worthless. I need to start it with a 100G EBS. Ensemble
offers me no ability to coordinate this before install of the formula.

[Данило Шеган (danilo)]

As far as I know, this is not very high on our roadmap. Lowering the priority.

[Cheryl Jennings (cherylj)]

Support for storage landed in 1.25. There is currently a bug to track allowing users to keep storage after its associated unit has been removed: bug 1592887