swift-container-info explodes when investigating quarantined containers

[Mark Kirkwood]

This analysis was performed on Swift 2.7.0.

Suppose a container is quarantined. One of the fist things you want to do is check where it came from initially. So reach for swift-container-info. It turns out this is a bad idea:

$ cd /srv/node/vdc/quarantined/containers/3a7b4bae41a17d0f54b247c727b4f0cd/
$ ls -l
total 20
-rw------- 1 swift swift 18432 Aug 15 2016 3a7b4bae41a17d0f54b247c727b4f0cd.db
$ sudo swift-container-info ./3a7b4bae41a17d0f54b247c727b4f0cd.db
Traceback (most recent call last):
  File "/usr/bin/swift-container-info", line 32, in <module>
    print_info('container', *args, **vars(options))
  File "/usr/lib/python2.7/dist-packages/swift/cli/info.py", line 327, in print_info
    info = broker.get_info()
  File "/usr/lib/python2.7/dist-packages/swift/container/backend.py", line 501, in get_info
    with self.get() as conn:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 366, in get
    self.possibly_quarantine(*sys.exc_info())
  File "/usr/lib/python2.7/dist-packages/swift/common/db.py", line 347, in possibly_quarantine
    renamer(self.db_dir, quar_path, fsync=False)
  File "/usr/lib/python2.7/dist-packages/swift/common/utils.py", line 1094, in renamer
    os.rename(old, new)
OSError: [Errno 16] Device or resource busy

ok that was bad - not only do we not get any useful info but it has tried to re-quarantine the container (and in the process mv the directory I am in...yikes, and hence the error):

$ ls -l
total 20
-rw------- 1 swift swift 18432 Aug 15 2016 3a7b4bae41a17d0f54b247c727b4f0cd.db
drwxr-xr-x 3 root root 24 Jun 16 06:55 quarantined

This comes about because swift-container-info:
- calls print_info which creates a ContainerBroker and...
- calls ContainerBroker.get_info which uses a self.get which resolves to DatabaseBroker.get
- DatabaseBroker.get calls self.possibly_quarantine

So the innocuous 'get_info' has dramatic side effects (possibly_quarantine)

We probably need to amend get_info e.g get_info(no_quarantine) to be side effect free for this type of use!

[Mark Kirkwood (mark-kirkwood)]

...probably applies to accounts too (as code looks similar)

[clayg (clay-gerrard)]

i can reproduce similarly on my development environment:

ubuntu@saio:~$ dd if=/dev/zero of=/srv/node2/sdb2/containers/767/1d7/bff782c1e5d0944511d817550b5841d7/bff782c1e5d0944511d817550b5841d7.db bs=1024 count=1 oflag=direct
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116758 s, 877 kB/s
ubuntu@saio:~$ swift-container-info /srv/node2/sdb2/containers/767/1d7/bff782c1e5d0944511d817550b5841d7/bff782c1e5d0944511d817550b5841d7.db
No handlers could be found for logger "root"
Traceback (most recent call last):
  File "/usr/local/bin/swift-container-info", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/vagrant/swift/bin/swift-container-info", line 47, in <module>
    run_print_info(args, vars(options))
  File "/vagrant/swift/bin/swift-container-info", line 23, in run_print_info
    print_info('container', *args, **opts)
  File "/vagrant/swift/swift/cli/info.py", line 382, in print_info
    info = broker.get_info()
  File "/vagrant/swift/swift/container/backend.py", line 501, in get_info
    with self.get() as conn:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/vagrant/swift/swift/common/db.py", line 371, in get
    self.possibly_quarantine(*sys.exc_info())
  File "/vagrant/swift/swift/common/db.py", line 361, in possibly_quarantine
    raise sqlite3.DatabaseError(detail)
sqlite3.DatabaseError: Quarantined /srv/node2/sdb2/containers/767/1d7/bff782c1e5d0944511d817550b5841d7 to /srv/node2/sdb2/quarantined/containers/bff782c1e5d0944511d817550b5841d7 due to corrupted database
ubuntu@saio:~$ ls /srv/node2/sdb2/quarantined/containers/bff782c1e5d0944511d817550b5841d7/
bff782c1e5d0944511d817550b5841d7.db
ubuntu@saio:~$ swift-container-info /srv/node2/sdb2/quarantined/containers/bff782c1e5d0944511d817550b5841d7/bff782c1e5d0944511d817550b5841d7.db
Traceback (most recent call last):
  File "/usr/local/bin/swift-container-info", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/vagrant/swift/bin/swift-container-info", line 47, in <module>
    run_print_info(args, vars(options))
  File "/vagrant/swift/bin/swift-container-info", line 23, in run_print_info
    print_info('container', *args, **opts)
  File "/vagrant/swift/swift/cli/info.py", line 382, in print_info
    info = broker.get_info()
  File "/vagrant/swift/swift/container/backend.py", line 501, in get_info
    with self.get() as conn:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/vagrant/swift/swift/common/db.py", line 371, in get
    self.possibly_quarantine(*sys.exc_info())
  File "/vagrant/swift/swift/common/db.py", line 350, in possibly_quarantine
    renamer(self.db_dir, quar_path, fsync=False)
  File "/vagrant/swift/swift/common/utils.py", line 1208, in renamer
    os.rename(old, new)
OSError: [Errno 18] Invalid cross-device link
ubuntu@saio:~$ find /srv/node2/sdb2/quarantined/
/srv/node2/sdb2/quarantined/
/srv/node2/sdb2/quarantined/containers
/srv/node2/sdb2/quarantined/containers/bff782c1e5d0944511d817550b5841d7
/srv/node2/sdb2/quarantined/containers/bff782c1e5d0944511d817550b5841d7/.l...