Document how to add swap space on Azure resource disk

[Stephen A. Zarkos]

The process for creating a swap partition on Ubuntu VMs on Azure is not well understood or documented. Most of the documentation on Azure generally refers to using the walinuxagent to manage the swap space on the resource disk (ephemeral disk). However, Ubuntu VMs use cloud-init for managing the resource disk.

There may also be two workflows to document as the customer may need to add swap space during provisioning by using CustomData, or after provisioning (I presume using a local cloud-config).

[Cracco (daniele-cozzo)]

Hi,

these are the steps I've used to add a swap partition to my Ubuntu VM on Azure:

1) Add from Azure portal a new data disk (in my case I added 2 data disk, the first is a 100 GB disk, the second one is a 6GB size for swap partition)

“/dev/sda” is the Azure VM OS disk;
“/dev/sdb” is the Azure VM local temporary disk;
“/dev/sdc” is the first Azure VM added data disk (100GB);
“/dev/sdd” is the second Azure VM added data disk (6GB);

2) create a new swap partition on /dev/sdd. Use cfdisk as follows:
# cfdisk /dev/sdd

partition type: Primary
FS type: Linux swap / Solaris (id 82)

3) mount swap partition on boot.
# nano /etc/fstab

append the following line and save the file:
/dev/sdd1 none swap sw 0 0

4) reboot the VM

5) create the swap partition. Use the following commands:
# mkswap /dev/sdd1
# swapon /dev/sdd1

6) verify if partition has been successfully created:
# cat /proc/swaps

Hope this helps,
Cracco

[trialotto (trialotto)]

@Cracco,

The discussion has to do with creating (persistent) swap space, on the resource (ephemeral) disk, without adding additional disks.

In essence, you provide a solution, but the solution ignores the possibility to use some of the unused disk space on the resource disk.

Furthermore, your solution is rather inflexible, if swap space has to be resized.

Kind regards

[Dan Watkins (oddbloke)]

OK, so passing a file containing the following as custom-data gives us a swap partition and a mounted disk at /mnt:

#cloud-config
disk_setup:
    ephemeral0:
        table_type: mbr
        layout: [[66, 82], 33]
        overwrite: True
fs_setup:
    - device: ephemeral0.1
      filesystem: swap
    - device: ephemeral0.2
      filesystem: ext4
mounts:
    - ["ephemeral0.1", "none", "swap", "sw", "0", "0"]
    - ["ephemeral0.2", "/mnt"]

Like so:

$ mount
...
/dev/sdb2 on /mnt type ext4 (rw,_netdev)

$ swapon -s
Filename Type Size Used Priority
/dev/sdb1 partition 418696184 0 -1

[trialotto (trialotto)]

@Daniel,

I did all the testing before and came to the same result (as you are probably aware of) BUT with a different CONCLUSION.

I personally concluded that the use of swap partitions, if ever desirable, is (also) not desirable in the current case AND certainly not on an ephemeral disk, given that:

1 - a swap partition at the beginning of a device/disk can be efficient, but will lead to a waste of disk space, with this being static,
2 - a huge swap partition (if someone makes that strange mistake) will result in severe problems and performance issues,
3 - a swap partition at creation time considerably reduces "provisioning speed/performance",
4 - a swap partition can become huge (i.e. too large) if a VM is upgraded to a larger size (with a larger ephemeral disk),
5 - an OPTIMAL swap partition should be a function of memory (similar to logic introduced with cloud-config setting swap).

Moreover, the swap partition as such can become obsolete, given the swap function(s) in cloud-init versions 0.7.6 and 0.7.7.

In short, I have an issue with the procentual division of the disk, into a normal partition and a swap partition.

The percentage assigned to the swap partition should be able to allow for

- automated calculations of the percentage (via logic, to optimize swap space)
- manual overrides of the automated calculations

Kind regards....

[Dan Watkins (oddbloke)]

Steve, I've written up some documentation on how to create swap partitions on Azure at https://wiki.ubuntu.com/AzureSwapPartitions

Please do let me know if there are any improvements that you'd like me to make!

(Marking this as Fix Committed until I hear back from you)

[Stephen A. Zarkos (stevez)]

Hi Daniel,

Thanks for doing this, looks great so far! I do have a couple recommendations:

In case customers are not familiar with the term "Custom Data" or how to use it, I would recommend linking to our documentation:
http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-how-to-inject-custom-data/
http://azure.microsoft.com/blog/2014/04/21/custom-data-and-cloud-init-on-windows-azure/

While passing a cloud-config via CustomData will work great during the initial setup and provisioning of the machine, is there a workflow that could allow a customer to configure a local cloud-config to configure swap space post-provisioning? This is feasible for those distributions that just use the Azure Linux agent, so preferably there is a way to do this on Ubuntu as well.

Thanks!
Steve

[trialotto (trialotto)]

@Steve,

I am working on a solution to pass some local cloud-config settings, for swap space settings and other settings.

One idea is to change the AzureDataSource.py script, in order to allow it to read from waagent.conf and/or other config files.

The possibility exists, given the fact that a user data file and a cloud-config.txt file are existing on the created instance.

The advantage of using cloud-init (instead of CustomData and/or ovf-env.xml) is that the cloud-init syntax (and scripts) can be used.

In fact, the only challenge is to have cloud-init read an additional datasource (on the created VM) at shut-down and/or start-up.

I do not have much time the coming days, but I´ll keep you posted.

Kind regards.....

[trialotto (trialotto)]

@Daniel,

The documentation looks great.

However, can you add the following (important) remarks to the documentation page:

a) as from cloud-init 0.7.6, the option exists to create swap by means of the swap stanza

b) in good practice, swap partitions should be at the start of the disk (and a function of the size of memory available)

Also note that your documentation is very suitable, if adapted slightly, for general documentation about creating partitions.

As a personal (and internal) note, the following:

- the new swap stanza (in newer versions of cloud-init) are automatically determined with some code logic, this should also be applied to swap partitions (and that is possible, since the settings are only applied on provisiniong or a restart of the VM),

- in general, cloud-init should be able to detect existing swap files and/or swap partitions and leaving them as is, when restarting a VM (instead of creating them again, as happens at this moment, with swap files and swap partitions). In essence, a restart (a shut-down and start-up sequence) should not be required to CHANGE swap space.

Can you react on the above personal notes?

Kind regards.....

[Dan Watkins (oddbloke)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Steve,

On 19/01/15 20:18, Stephen A. Zarkos wrote:
> In case customers are not familiar with the term "Custom Data" or
> how to use it, I would recommend linking to our documentation:
> http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-how-to-inject-custom-data/
>
>
http://azure.microsoft.com/blog/2014/04/21/custom-data-and-cloud-init-on-windows-azure/

Done.

> While passing a cloud-config via CustomData will work great during
> the initial setup and provisioning of the machine, is there a
> workflow that could allow a customer to configure a local
> cloud-config to configure swap space post-provisioning? This is
> feasible for those distributions that just use the Azure Linux
> agent, so preferably there is a way to do this on Ubuntu as well.

I'm not sure about this; I will confer with Ben and Scott Moser.

Cheers,

Dan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJUvi3UAAoJEO7mh3btsT6gWyUQAJQg3KgbGa3ShzP2vCPYbOr4
f9hREe2Kt8+i24wlecQ6SYnoqbab7ct/Il18MM3kj1fSRzq9QdW5LMsjeiaQCdTa
3xoG33Kyh/N6GQ4LfWmfK302WLPuA85fOSyk+7NLdvw2sxKh0HXvRjUtXmJ7dJvj
vQFU5oiqqpei8Q0GUN4MUs4HcR/ybzlOXBpIi1yNR11POBUmFHh4ubM3ZgZQODKc
riAvKCm2VY6s3EdZLK2lI6+UoSkNudAmhSL3AyFj+tpFvuB/h6Aak9tSHFeqHTXq
nChciSiQt0a7IlzbPOBK0w4MZOG3Xmg1tkHXFaS/eQR15W8HLhWqqtSwO4NFV5Gx
uXuYh+RPS1jypX+f5Pz3oQrV2y3sENk+bfYt4hqV7RodkwLfpZHHiIgStDq6bjtr
FGrWma4SkqZKVcXqzRT9DvVE3I9qXvj/YOeGGEQudlg9zYkb1CIDhxk18sog+7Vq
WuN+GU0boDzLMgOvVcoh7zW9uZWcbtuceqMqcTUY3Vr8j+4G8hKeTQeB2vjmxhoY
3zkY/lBKs7EHcd89ud68M2Ec7WSXb5pOtiMZ517S0cXTrg7cllHhcixrmK7IXmEQ
ecvemZEQGvebRxx8cas0y5zOuTUpgiB9Ecp7jsU+rbkvyx4xRqJq6LRSaNTX42BV
mr7RPAJlpLogLOukXBiN
=g+mH
-----END PGP SIGNATURE-----

[Dan Watkins (oddbloke)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi trialotto,

On 20/01/15 02:15, trialotto wrote:
> The documentation looks great.
Thanks!

> However, can you add the following (important) remarks to the
> documentation page:
>
> a) as from cloud-init 0.7.6, the option exists to create swap by
> means of the swap stanza

Thanks for the pointer. The swap directive will only create swap
_files_, rather than swap partitions. I've added a note to the page
and a link to the cloud-init example of how to use the swap directive.

> b) in good practice, swap partitions should be at the start of the
> disk (and a function of the size of memory available)

Best practices for swap partitions are outside of the scope of this
documentation (particularly given that when we're talking about
storage in a cloud, it's very difficult to know what it's actually
backing on to).

> - the new swap stanza (in newer versions of cloud-init) are
> automatically determined with some code logic, this should also be
> applied to swap partitions (and that is possible, since the
> settings are only applied on provisiniong or a restart of the VM),

Feel free to open a new bug for this, explaining precisely the change
in behaviour you would want to see. :)

> - in general, cloud-init should be able to detect existing swap
> files and/or swap partitions and leaving them as is, when
> restarting a VM (instead of creating them again, as happens at this
> moment, with swap files and swap partitions). In essence, a restart
> (a shut-down and start-up sequence) should not be required to
> CHANGE swap space.

I'm not 100% sure what behaviour you are seeing that you don't expect
to. Could you file a new bug with details?

Thanks,

Dan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJUvi4EAAoJEO7mh3btsT6gtjQP/0HypN2gaBNwYOJ7iLpZsvuM
guG19/Cq+Cjz4V1cD56i3bj7isFHo7jYUp+ZfGZZq3jvU7CssjGeQIllAs7shrDb
G4xrmVj8Lsm5+oytmaiff3k6sPY1QHOlKZNRGmPBsjYqA5Ni3cp+mWMQppwd4aI1
NwHvLQRwVOD40dbJuzmP7+IKoGUx9fhnmN9FjpMDBfEV5Hgdyld6xLhI3iGUZICO
S6qg1Y3L8TWTeihx6Kvlh3my+gDbEQW3Z8y+/3OdT9hUKBZt2aZZtcDr9WHSlx2Z
5thycOaYCZjHqKzc4ih/zxmlS+9oIYqba8Gk4tu0KlCLl9wZQHdl/ByH7gzIW5P5
AfB2/mBmZqvEIJooykcqds3nNiE/yx/3IBd59LKUwHOtr3RegzFInf1jqIC8XHMi
CoxMoEfhud2d5MRF7YApJU/DsMAKXTBd/kXesZGu+nN9BbcFMu7+4y5bNQFixz0U
dsfA70YZMbh6fHMYeouGZ3M0aGouwW7lMfzSlHRUnr2Vb8p1NppIgv1c6KOwHgQR
mZ39YCx2+XIOHbZ/daB2UFg2lpOv30VFPNuws8Vr7bdYCRLA7jT9clbzcgUyoMbz
eUHepQSQTgNykXfJS3/hCvMivFv/pRWWNV5JVIaRTwLQU62TfUyE7v5edsoXVM/b
kYIKEv56MRnh7oKcN/k2
=SHJb
-----END PGP SIGNATURE-----

[trialotto (trialotto)]

@Dan(iel),

I will post two new "bugs" (that are not really bugs, but a wish for code improvement):

- size logic (for both swap space as swap partitions)
- swap stanza that allows changes of swap file size (not swap partitions) on a created/running VM to be persistent

By the way, if necessary, I am willing to contribute to the documentation and/or cloud-init source code.

Just let me know....

Ciao...

[Dan Watkins (oddbloke)]

Steve,

I think the conclusion we've reached is that cloud-init isn't an appropriate way to add swap to an instance after it has been spun up. cloud-init has been designed to do disk setup at first boot and so makes assumptions that wouldn't hold after that point.

If people do want to add swap after first boot, they should use the standard tooling they would use on any other Ubuntu instance. :)

Thanks,

Dan

[trialotto (trialotto)]

@Daniel,

The documentation is not okay: the swap partition (on the last part of a disk) will not persist through shutdown in Azure.

The strange thing is:

a) often a NTFS formatted ephemeral disk reoccurs when starting up after a shutdown,

b) a swap partition on the FIRST part of a disk IS persistent through shutdown in Azure.

Please shed a light on the above.

Kind regards....

[Ben Howard (darkmuggle-deactivatedaccount)]

I'm happy to report that this will be fixed in the next SRU of Cloud-init currently in -proposed. For Bug #1422919, we backported bits to support GPT partition tables. In testing the -proposed package, I confirmed that things worked.

@trialotto, can you please try the package in -proposed and report back? I think that we can finally put this issue to bed.

[trialotto (trialotto)]

@Ben,

As soon as time is available, I will test the -proposed package (if possible, on the old Azure VMs that have been used for testing).

I will add the following test: testing the 0.7.6 version on Ubuntu Trusty (14.04.1 LTS)

Please note that I just received the message, implying that I have not been able to verify what is changed in the 0.7.5 version (the standard version on Trusty) and/or whether swap file cloud-config options have been backported.

Also note that cloud-init documentation with respect to

- cloud-config examples, and/or
- Azure custom AND standard disk_setup and fs_setup settings

can be adjusted for the reasons that disk_setup and fs_setup settings for attached disks, in the form of

#cloud-config
disk_setup:
    [ .. ephemeral config lines .. ]
    /dev/sdc:
        table_type: mbr
        layout: True
        overwrite: True
fs_setup:
    [ .. ephemeral config lines .. ]
    - device: /dev/sdc1
      filesystem: ext4

suffice to setup and format attached disk at VM creation time (instead of manually partitioning and formatting afterwards).

Note that the "overwrite: True" at /dev/sdc disk setup does not have any impact, in the sense that data on the disk is not lost (!).

@Ben: for the sake of clear cloud-init standard settings, it would propably be desirable to remove the alias "ephemeral" in Azure standard settings, since this removal cannot do harm and makes more easy to reference the /dev/sda (ephemeral) disk. Is this a change to be considered?

@Ben: if I am not mistaken, it should be easy for cloud-init to automatically detect attached disks (Azure devices) and assign a "standard" disk_setup and fs_setup settings (as provided above). The standard settings can than be easily overruled by custom cloud-config settings. However, I am not completely sure whether this is desirable behavior for cloud-init, what do you think?

@Stephen: the walinuxagent (waagent script) should be able to create a custom cloud-config file for the automatic detection, setup and formatting of the attached disks (Azure devices), in order to have formatted disks (after creation of the VM) that are persistent across (normal) restarts AND shutdown/start sequences (the latter is normally a problem, not in this case). Can we expect some adjustment of the walinuxagen (waagent script)?

Kind regards....

[trialotto (trialotto)]

Ben,

The patch lp-1422919-azure-g5_ephemeral.patch does not help, when introducing this patch (alone) in cloud-init.

There seems to be some basic problem with the basic config settings, in the sense that /dev/sdb is always starting as NTFS.

The standard result of the tests are summarized as: checking disks with parted -l does indicate ext4 settings for all disks (even an attached /dev/sdc) after creation, checking disks with fdisk -l indicates that (only) /dev/sdb is a HPFS/NTFS/exFAT device.

Note that at creation time, cloud-init (0.7.5-0ubuntu1.3) is not patched.

A reboot and/or restart of the VM does not make any difference, the standard result remains.

A shut-down/start sequence for the VM, with cloud-init patched (0.7.5-0ubuntu1.3 with lp-1422919-azure-g5_ephemeral.patch), does yield different results:

a) a check with parted -l indicates /dev/sdb being ntfs, (and)
b) a check with fdisk -l indicates (again) that (only) /dev/sdb is a HPFS/NTFS/exFAT device,

and the following varying factors do not have effect:

- disk mount locations (persistently correct across all test situations)
- using aliases (ephemeral0 and devicename ephemeral0.x) for /dev/sdb or using no aliases (/dev/sdb and devicename /dev/sdb1)
- using /dev/sdb table_type: gpt at creation time (not supported by 0.7.5-0ubuntu1.3, but that does not seem to matter)
- the /dev/sdb overwrite at disk_setup (this can be False or True)

Again note that the /dev/sdc disk is a constant factor, passing "parted -l" and "fdisk -l" checks every time.

Also note that exceptions on the standard result are caused by defining a swap partition (fs type 82) on the first part of the disk.

Any definition in the form of layout: [[100,83]] does not help.

In short, cloud-config setting do not change the erratic behavior with respect to /dev/sdb AND the current patch does not either.

In fact, /dev/sdc is well-behaved (as one expects), with the only difference (in old and new cloud-init code) that /dev/sdc is not hindered by standard config settings (in the python scripts).

In fact (again), the erratic /dev/sdb does not change with settings, in a shutdown/start sequence issue still remains (i.e. /dev/sdb returns to ntfs state), while after VM creation a check with fdisk -l indicates that /dev/sdb is a HPFS/NTFS/exFAT device .

All the above suggests that there is one major issue with EITHER standard config settings for /dev/sdb (as present in python code) OR the disk_setup stanza for /dev/sdb.

It would be my guess that the disk_setup stanza (for dev/sdb) is the culprit (since overruling the disk_setup stanza for /dev/sdb does not have any effect AND specific, unsupported terms like "gpt" are ignored fully).

By the way, all the above is a preliminary conclusion, I will have a peek into the code, when I have time.

Your feedback is welcome!

Kind regards.....

[David Frank (bitinn)]

So I ended up on this page and found the conclusion is azure cloud-init doesn't support adding a swap through configuration, is that true?

Furthermore why isn't there a canonical doc or blog post that explain the correct way to add swap for Ubuntu VM on Azure?

I have seen examples that modify /etc/fstab, and examples that modify /etc/waagent.conf, as well as trying to use CustomData.

I have read both:

- https://wiki.ubuntu.com/AzureSwapPartitions
- https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-agent-user-guide/

Yet I am still not sure how could I edit my configuration to create a swap through cloud-init on next boot. If this is fixed and released, then please help point me and whoever comes next to the right doc.

[Dan Watkins (oddbloke)]

Hi David,

On 31/05/15 09:45, David Frank wrote:
> So I ended up on this page and found the conclusion is azure cloud-init
> doesn't support adding a swap through configuration, is that true?
>
> ...
>
> I have read both:
>
> - https://wiki.ubuntu.com/AzureSwapPartitions
> - https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-agent-user-guide/
>
> Yet I am still not sure how could I edit my configuration to create a
> swap through cloud-init on next boot. If this is fixed and released,
> then please help point me and whoever comes next to the right doc.
>

The wiki page has a pretty comprehensive description of how you can add
a swap partition using cloud-init configuration, including a full
configuration that will actually do it at the bottom of the page, so I'm
not really sure how you're drawing that conclusion.

That said, if that documentation doesn't apply to your case or is in
some way unclear, please let me know and I'll do my best to amend it. :)

Thanks,

Dan

[trialotto (trialotto)]

Daniel,

Defining swap with cloud-config will not result in a persistent swap space across shutdown and restart sequences on Azure.

In general, the cloud-config disk setup and/or filesystem stanza will not result in a persisting ext4 ephemeral disk in before mentioned sequences, in which the ephemeral disk often (but strangely, not always) returns to ntfs format.

The above suggests (on the one hand) an impossibility to use ephemeral disk space for swap purposes and (on the other hand) a bug in the Azure fabric, causing defined swap spaces to be lost when returning to the ntfs format.

Naturally, an added (non-ephemeral) disk can be used for swap purposes, but this is not adviceable.

Furthermore, configuring the waagent.conf variables "ResourceDisk.Format=y" would resolve the Azure fabric issue.

In conclusion, cloud-init and the Azure fabric are not really interacting properly, with respect to swap spaces.

Any advances on the cloud-init front?

Kind regards...

[Dan Watkins (oddbloke)]

Hi trialotto,

Could you confirm what version of cloud-init you are using? Recent versions detect when a reformatted ephemeral disk is presented and will always reformat it (I have just confirmed this with an Ubuntu trusty instance).

Also, "ResourceDisk.Format=y" only allows you to specify a single partition to create; cloud-init allows you to configure the disk how you want. :)

Dan

[trialotto (trialotto)]

Daniel,

You are very likely to be correct, I saw the specific code (for recent versions of cloud-init) and it should be able to reformat disks.

Questions are:

- is the swap created after or before the reformat? (Note: a reformat removes the characteristic "persistent" from persistent swap)
- is the Azure VM image running the most appropriate and/or recent cloud-init version?

It often has been the problem that cloud-init versions on Azure Ubuntu VMs are not really up-to-date.

However, I will try to test which issues occur (if any), after thursday though.

Kind regards....

[Dan Watkins (oddbloke)]

On 01/06/15 23:55, trialotto wrote:
> You are very likely to be correct, I saw the specific code (for recent
> versions of cloud-init) and it should be able to reformat disks.
>
> Questions are:
>
> - is the swap created after or before the reformat? (Note: a reformat removes the characteristic "persistent" from persistent swap)

swap is created as part of the reformat; before the reformat we have a
disk with a single NTFS partition. I'm not even sure what it would mean
to create it before the reformat.

Persistent swap is swap that continues to exist after a reboot; it
doesn't retain any data over reboots (any more than RAM does). Unless
you have an application that immediately requires swap on boot, this is
effectively persistent swap. (And if you have such an application, you
probably aren't using an appropriately sized VM for it)

> - is the Azure VM image running the most appropriate and/or recent cloud-init version?

This functionality is in all supported Ubuntu images on Azure; it was
introduced for trusty and backported to precise in April 2014 (i.e.
before trusty was even released).

> It often has been the problem that cloud-init versions on Azure Ubuntu
> VMs are not really up-to-date.

Each release of Ubuntu contains the latest version of cloud-init when it
is released. We then follow Ubuntu's Stable Release Update process[0]
to backport any important functionality (or security fixes) whilst
avoiding regression; the cloud-init versions on official Ubuntu images
in Azure _are_ up-to-date.

> However, I will try to test which issues occur (if any), after thursday
> though.

If you do find issues that aren't simply with the documentation at
https://wiki.ubuntu.com/AzureSwapPartitions, please open a new bug. :)

Thanks,

Dan

[0] https://wiki.ubuntu.com/StableReleaseUpdates