nova backup shutdown instances

[admgsic]

Hello good mornig for everyone,

For some time now I have been experiencing some problems when backing up instances with the instances on.

I have them programmed through cron daily with a script with a rotation of 7 copies to keep the last 7 images of that instance.

nova backup "instance" "instance" + "date" daily 7

All this self-authenticating previously as it would be logical.

Anyway there are times that backups are done without problem but recently I resized an instance with a custom flavor and sometimes the backup fails and I find the instance turned off.

I have to say that I am now with the Pike version.

With the Mitaka version it has happened to me in some instances that have not been resized but that do have a personalized flavor, although I have also had some that did not have personalized flavors.

Does anyone know what can happen?

Thank you very much in advance,

Regards,

[jichenjc (jichenjc)]

do you have more info about why it's failed such as api or compute log?

it's really hard to guess from what you explained..

[admgsic (j-3dmin-q)]

Hello it is true, if it happened again today in the Pike version and I was just at the moment in which I scheduled the backup of a resized instance in the compute node I found the following ..

2018-05-17 08:15:37.661 7677 ERROR nova.virt.libvirt.guest [req-e1743e3f-fdb3-48b4-8c0b-e3fc24937f31 190e67811cc043f19fe3dcc85273cf7a b7c7935421fa44e4912fc660037ab7ac - default default] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-0000110d</name>
  <uuid>35286194-3a24-47f3-b03b-f19f8c30eeae</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="16.0.3"/>
      <nova:name>NOMBRE INSTANCIA</nova:name>
      <nova:creationTime>2018-05-16 05:44:51</nova:creationTime>
      <nova:flavor name="SABOR PERSONALIZADO">
        <nova:memory>8192</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>4096</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>2</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="190e67811cc043f19fe3dcc85273cf7a">admin</nova:user>
        <nova:project uuid="b7c7935421fa44e4912fc660037ab7ac">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="6443c914-91b3-4bd9-bd9a-aa351afb9011"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <shares>2048</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>16.0.3</entry>
      <entry name='serial'>c87de31f-c71b-4d9b-b3ac-0981109f6d2b</entry>
      <entry name='uuid'>35286194-3a24-47f3-b03b-f19f8c30eeae</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-artful'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <topology sockets='2' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/35286194-3a24-47f3-b03b-f19f8c30eeae/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/35286194-3a24-47f3-b03b-f19f8c30eeae/disk.swap'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller t...

[jichenjc (jichenjc)]

looks like this is the error:

: libvirtError: Cannot access backing file '/var/lib/nova/instances/_base/swap_4096' of storage file '/var/lib/nova/instances/35286194-3a24-47f3-b03b-f19f8c30eeae/disk.swap' (as uid:64055, gid:117): No existe el archivo o el directorio

you are saying:

>>>Anyway there are times that backups are done without problem but recently I resized an instance with a custom flavor and sometimes the backup fails and I find the instance turned off.

so ,what's the previous flavor and what's the later flavor ? I guess some swap disk size difference?

[admgsic (j-3dmin-q)]

Hello jichenjc,

The last flavor that had that instance had the following characteristics.

m1.small: VCPU: 1 | RAM: 2GB | Disk: 10GB | Ephemeral Disk: 0GB | Swap: 0MB

And then I resized to another flavor with the following characteristics:

g1.newflavor: VCPU: 2 | RAM: 8GB | Disk: 20GB | Ephemeral Disk: 0GB | Swap: 4096MB

The interesting thing is that sometimes it fails and sometimes it does not, it starts to do the copy with nova backup but it is not able to store it or it fails and then that error appears in nova-compute.log

[jichenjc (jichenjc)]

not sure it's similar to https://bugs.launchpad.net/nova/+bug/1762423
but seems both related to swap disk... need further analysis

[admgsic (j-3dmin-q)]

Thank you very much for the reply. What I have done at the moment is to make a snapshot of the instance and create it again with a flavor without swap of 4GB and it seems that for now that failure has not been so frequent ...

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]