Comment 44 for bug 1668129

Ok, I figured out the problem.

You're using the yakkety kernel, 4.8. In the Xenial 4.4 kernel, memory hotplug auto-onlining is disabled; however in the 4.8 kernel, memory hotplug auto-onlining is enabled, so disabling the udev rule with the 4.8 kernel does nothing - the kernel's already onlined the balloon memory region.

Edit your /etc/default/grub.d/50-cloudimg-settings.cfg file to add a kernel boot param "memhp_default_state=offline", e.g.:

--- /etc/default/grub.d/50-cloudimg-settings.cfg.orig 2017-03-21 17:52:26.604389516 +0000
+++ /etc/default/grub.d/50-cloudimg-settings.cfg 2017-03-21 17:52:46.564462247 +0000
@@ -8,7 +8,7 @@
 GRUB_TIMEOUT=0

 # Set the default commandline
-GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"
+GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 memhp_default_state=offline"

 # Set the grub console type
 GRUB_TERMINAL=console

Then, run:

$ sudo update-grub

make sure you see the new boot param in your grub.cfg, e.g.:

$ grep memhp_default_state /boot/grub/grub.cfg
 linux /boot/vmlinuz-4.8.0-41-generic root=UUID=765f00af-531a-44bc-a083-66143320d408 ro console=tty1 console=ttyS0 memhp_default_state=offline

Then reboot, and when it comes back up, check to make sure memory auto-onlining is disabled now:

$ cat /sys/devices/system/memory/auto_online_blocks
offline

Note, you still need to disable the udev memory hotplug online rule, as mentioned in previous comments.