Fetching additional cloud-init from object storage - such as S3

[Streich-reinis]

Feature request: user-data in some cloud providers is limited to 16kb, even with compression is not enough sometimes.
Adding option to fetching/loading cloud-init from remote object storage would greatly solve this problem (For example S3 bucket)

[Scott Moser (smoser)]

Use

#include <your-url-here>

or

#include-once <your-expiring-url-here>

[Scott Moser (smoser)]

https://cloudinit.readthedocs.io/en/latest/topics/format.html?highlight=include%20file

[Dan Watkins (oddbloke)]

#include only supports HTTP(S), not the S3 API. The reason that using S3 directly is interesting is that EC2 instances can be granted roles which give them S3 fetch permissions but (according to user reports) these don't apply to HTTP(S) GETs, only to API hits. This means that users can protect their user-data without having to supply credentials in their user-data.

(The relevant IRC conversation is https://irclogs.ubuntu.com/2020/04/23/%23cloud-init.html#t18:38)

[Scott Moser (smoser)]

The '#include-once' was aimed at a similar goal.
There, you can just provide an expiring url in the #include-once
cloud-init will only ever download it once.

I'm sure its different
 https://medium.com/runascloud/creating-a-secure-url-with-an-expiration-date-from-s3-11c7b2fe7206

[Streich-reinis (streich-reinis)]

@Scott, Main issue the the pre-signed url has expiration date which can't be set to "forever".

Common use-case is to use Auto Scaling Group where instances scale-in & scale-out based on load/requirements, if I were to use url, then I would have to re-create/update some part of configuration every-time url expires.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3658