Narrowed this down to Cloud-init. Cloud-init is not selecting the right device, and is reconfiguring Grub to use /dev/sda:
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] __init__.py[DEBUG]: handling grub-dpkg with freq=None and args=[]
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] cc_grub_dpkg.py[DEBUG]: setting grub debconf-set-selections with '/dev/sda','false'
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] __init__.py[DEBUG]: handling apt-pipelining with freq=None and args=[]
And the mounts:
$ mount
/dev/xvda1 on / type ext4 (rw)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw,_netdev)
Narrowed this down to Cloud-init. Cloud-init is not selecting the right device, and is reconfiguring Grub to use /dev/sda: dpkg.py[ DEBUG]: setting grub debconf- set-selections with '/dev/sda','false'
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] __init__.py[DEBUG]: handling grub-dpkg with freq=None and args=[]
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] cc_grub_
Jul 2 16:53:10 ip-10-169-38-57 [CLOUDINIT] __init__.py[DEBUG]: handling apt-pipelining with freq=None and args=[]
And the mounts: nosuid, nodev) nosuid, nodev) fuse/connection s type fusectl (rw) security type securityfs (rw) nosuid, gid=5,mode= 0620) nosuid, size=10% ,mode=0755) nosuid, nodev,size= 5242880)
$ mount
/dev/xvda1 on / type ext4 (rw)
proc on /proc type proc (rw,noexec,
sysfs on /sys type sysfs (rw,noexec,
none on /sys/fs/
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,
tmpfs on /run type tmpfs (rw,noexec,
none on /run/lock type tmpfs (rw,noexec,
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/xvdb on /mnt type ext3 (rw,_netdev)