freezer-agent crashes as list index out of range

Bug #1874492 reported by hao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Backup/Restore and DR (Freezer)
Fix Released
Undecided
hao

Bug Description

When you create a lvm snapshot backup with freezer-agent, function get_lvm_info() of snapshot/lvm.py may produce a null string mount_point which will cause "Critical Error: list index out of range" in function lvm_guess().

- reproduce
# freezer-agent --debug --snapshot mysql_lv --lvm-srcvol /dev/cl/var --lvm-volgroup cl --path-to-backup /var/lib/mysql --mysql-conf /root/.freezer/freezer-mysql.conf --storage swift --container freezer_mysql-backup-prod --mode mysql --exclude "\*.lock" --backup-name mysql-test
Critical Error: list index out of range

- feezer-agent logs:
2020-04-23 12:35:58.646 7831 ERROR freezer.main [-] list index out of range: IndexError: list index out of range
2020-04-23 12:35:58.646 7831 ERROR freezer.main Traceback (most recent call last):
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/main.py", line 279, in main
2020-04-23 12:35:58.646 7831 ERROR freezer.main freezer_main(backup_args)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/main.py", line 144, in freezer_main
2020-04-23 12:35:58.646 7831 ERROR freezer.main return run_job(backup_args, storage)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/main.py", line 157, in run_job
2020-04-23 12:35:58.646 7831 ERROR freezer.main response = freezer_job.execute()
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/job.py", line 205, in execute
2020-04-23 12:35:58.646 7831 ERROR freezer.main backup_level = self.backup(app_mode)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/job.py", line 258, in backup
2020-04-23 12:35:58.646 7831 ERROR freezer.main snapshot_taken = snapshot.snapshot_create(self.conf)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/snapshot/snapshot.py", line 47, in snapshot_create
2020-04-23 12:35:58.646 7831 ERROR freezer.main return lvm.lvm_snap(backup_opt_dict)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/snapshot/lvm.py", line 200, in lvm_snap
2020-04-23 12:35:58.646 7831 ERROR freezer.main lvm_info = get_lvm_info(backup_opt_dict.path_to_backup)
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/snapshot/lvm.py", line 240, in get_lvm_info
2020-04-23 12:35:58.646 7831 ERROR freezer.main mount_point_path, mount_points, 'mount')
2020-04-23 12:35:58.646 7831 ERROR freezer.main File "/usr/lib/python2.7/site-packages/freezer/snapshot/lvm.py", line 281, in lvm_guess
2020-04-23 12:35:58.646 7831 ERROR freezer.main mount_path = mount_list[2]
2020-04-23 12:35:58.646 7831 ERROR freezer.main IndexError: list index out of range
2020-04-23 12:35:58.646 7831 ERROR freezer.main
2020-04-23 12:35:58.647 7831 CRITICAL freezer.main [-] Run freezer agent process unsuccessfully: IndexError: list index out of range
2020-04-23 12:35:58.647 7831 CRITICAL freezer.main [-] Critical Error: list index out of range

- analysis
When the mountpoint of freezer-agent option path-to-backup is invisible in /proc/mounts, function get_lvm_info() will try to use subprocess.Popen([mount]) to retieve mount-points.
However, simply use mount_points = mount_out.split('\n') may create a null string in the last of list mount_points. The null string eventually causes a list index out of range error.
e.g.
2020-04-23 12:35:58.645 7831 INFO freezer.snapshot.lvm [-] mount_points: ['sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)', 'proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)', 'devtmpfs on /dev type devtmpfs (rw,nosuid,size=3994280k,nr_inodes=998570,mode=755)', 'securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)', 'tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,size=4005196k,nr_inodes=1001299)', 'devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)', 'tmpfs on /run type tmpfs (rw,nosuid,nodev,size=4005196k,nr_inodes=1001299,mode=755)', 'tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4005196k,nr_inodes=1001299,mode=755)', 'cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)', 'pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)', 'cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)', 'cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)', 'cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)', 'cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)', 'cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)', 'cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)', 'cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)', 'cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)', 'cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)', 'cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)', 'configfs on /sys/kernel/config type configfs (rw,relatime)', '/dev/mapper/cl-root on / type xfs (rw,relatime,attr2,inode64,noquota)', 'systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)', 'hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)', 'mqueue on /dev/mqueue type mqueue (rw,relatime)', 'debugfs on /sys/kernel/debug type debugfs (rw,relatime)', '/dev/mapper/cl-home on /home type xfs (rw,relatime,attr2,inode64,noquota)', '/dev/mapper/cl-var on /var type xfs (rw,relatime,attr2,inode64,noquota)', '/dev/vda1 on /boot type xfs (rw,relatime,attr2,inode64,noquota)', 'sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)', 'tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=381612k,mode=700)', '/dev/mapper/cl-freezer_backup_snap_8d4235d457be404bb869ad7a7774625d on /var/lib/freezer/mount_8d4235d457be404bb869ad7a7774625d type xfs (ro,relatime,nouuid,norecovery,attr2,inode64,noquota)', '']

hao (dhs007)
Changed in freezer:
assignee: nobody → hao (dhs007)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to freezer (master)

Fix proposed to branch: master
Review: https://review.opendev.org/722404

Changed in freezer:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to freezer (master)

Reviewed: https://review.opendev.org/722404
Committed: https://git.openstack.org/cgit/openstack/freezer/commit/?id=d199ac67329fe25d1cd40c9b91a23962675bb664
Submitter: Zuul
Branch: master

commit d199ac67329fe25d1cd40c9b91a23962675bb664
Author: Haosheng He <email address hidden>
Date: Fri Apr 24 00:55:12 2020 +0800

    fixed an issue that null string exists in mountpoint list

    In function snapshot.lvm.get_lvm_info(), filter null string members
    before return the mountpoint list.

    Change-Id: Iae07dd081cafe92fa63c90a0d4bf967ec7206ea5
    Closes-Bug:1874492

Changed in freezer:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.