OpenStack Object Storage (swift)

swift-drive-audit broken on python 3

Bug #1847955 reported by Tim Burke on 2019-10-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	High	Unassigned

Bug Description

On python3, swift-drive-audit bombs out pretty quickly with something like

  Traceback (most recent call last):
    File ".../swift/common/utils.py", line 396, in backward
      f.seek(-blocksize, os.SEEK_CUR)
  io.UnsupportedOperation: can't do nonzero cur-relative seeks

During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File ".../bin/swift-drive-audit", line 7, in <module>
      exec(compile(f.read(), __file__, 'exec'))
    File ".../bin/swift-drive-audit", line 184, in <module>
      errors = get_errors(error_re, log_file_pattern, minutes, logger)
    File ".../bin/swift-drive-audit", line 95, in get_errors
      for line in backward(f):
    File ".../swift/common/utils.py", line 399, in backward
      f.seek(-blocksize, os.SEEK_CUR)
  io.UnsupportedOperation: can't do nonzero cur-relative seeks

Per the docs [1],

> In text files (those opened without a `b` in the mode string), only
> seeks relative to the beginning of the file are allowed ...

... so I guess we need to open /var/log/kern.log et al. in binary mode. Now to figure out what an appropriate encoding would be...

[1] https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-14: Fix merged to swift (master)

Reviewed: https://review.opendev.org/688326
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=405a2b2a55866b74f196ad81eaafedeee4615fa8
Submitter: Zuul
Branch: master

commit 405a2b2a55866b74f196ad81eaafedeee4615fa8
Author: Tim Burke <email address hidden>
Date: Sun Oct 13 21:55:54 2019 -0700

py3: Fix swift-drive-audit

    Walking through the kernel logs backwards requires that we open them
    in binary mode. Add a new option to allow users to specify which
    encoding should be used to interpret those logs; default to the same
    encoding that open() uses for its default.

Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6
Closes-Bug: #1847955

Changed in swift:
status:	Confirmed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-15: Fix proposed to swift (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/688766

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-15: Fix merged to swift (stable/train)

Reviewed: https://review.opendev.org/688766
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bcb2016673434e720e92fe4e75969c3b518a92c8
Submitter: Zuul
Branch: stable/train

commit bcb2016673434e720e92fe4e75969c3b518a92c8
Author: Tim Burke <email address hidden>
Date: Sun Oct 13 21:55:54 2019 -0700

py3: Fix swift-drive-audit

    Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6
    Closes-Bug: #1847955
    (cherry picked from commit 405a2b2a55866b74f196ad81eaafedeee4615fa8)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-22: Fix included in openstack/swift 2.23.1

This issue was fixed in the openstack/swift 2.23.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-07: Fix proposed to swift (feature/losf)

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/693283

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-07: Fix merged to swift (feature/losf)

Download full text (9.6 KiB)

Reviewed: https://review.opendev.org/693283
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=dbfb6a6fc90cb3367ed2248572089ee3cbafc15a
Submitter: Zuul
Branch: feature/losf

commit 7cb92810acfd3cd598860aefadff8fba47e3efdd
Author: SeongSoo Cho <email address hidden>
Date: Wed Nov 6 15:32:56 2019 +0800

Fix bug internal client's delete_object logs 499

    Internal Client's doest not read body for 404.
    So, When internal deleing object that not existed, proxy-server logs
    its response code to 499.

In this patch, add a code that drain the reponse body to prevent
unexpected disconnect in proxy-server

Change-Id: I6d170fead798d0d539c69d27a6fa8a2d0123ca99
Closes-Bug: #1835324

commit c512574e110c214396658743e75ae497ba76e2ff
Author: Tim Burke <email address hidden>
Date: Tue Nov 5 01:45:26 2019 +0000

Make our py2 func tests more explicitly py2

...because otherwise, I think they might be running under py3 right now.

Change-Id: I96ad9e5ed87a7e08d06e889233730cec476d3af7

commit a505d02b3762ecbe7ea80b968ddec90ca3446549
Author: Chris Smart <email address hidden>
Date: Sat Nov 2 20:48:47 2019 +1100

Tweak Swift AIO docs, update CentOS and Fedora

The Swift AIO documentation is out of date for CentOS and Fedora.

This patch updates the documentation to ensure that the instructions are
clear and accurate.

    It also cleans up a few other sections along the way and adds some new
    headings to make it easier to read. Some explanatory notes are added,
    such as the need for XFS storage and test directories. XFS tmp loopback
    device is moved out of the user's home directory to /srv to match
    storage.

Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
Signed-off-by: Chris Smart <email address hidden>

commit 281ffab2a52f348aa3f8ad39e800a43c6047e1cf
Author: Thiago da Silva <email address hidden>
Date: Fri Nov 1 09:57:34 2019 +0200

pin lower-constraints to run against python 2.7

Change-Id: I5c856012a04ef8f910c28b0248ea7bcda47e4816

commit 0c1b485ad671fc0abac15cd0bbec894e517e1dd9
Author: John Dickinson <email address hidden>
Date: Thu Oct 17 09:43:55 2019 -0700

exclude utf8 tests under py3

These are known to not work until https://bugs.python.org/issue37093
is addressed in CPython upstream.

Change-Id: I4a6877907d14b632a9a477c887913488427b62b7

commit dd235b6ac0757e34805b041d331511c957764f3e
Author: Clay Gerrard <email address hidden>
Date: Mon Oct 28 11:27:58 2019 -0500

use request_helpers import namespace for tests

Change-Id: I8e18ff2036312d3049a420f3332a6ea6110447c4

commit b75d5939352c405ab82ab52c3a9a1c3ad030aa26
Author: Clay Gerrard <email address hidden>
Date: Fri Oct 25 14:34:41 2019 -0500

Refactor timestamp iter in db backend tests

Change-Id: I61d044a6d778e158af73a3f4d9fc75d642af0beb

commit 739c3704107bd991493472c18bfe7ac5f933077c
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Oct 18 07:10:33 2019 +0000

Imported Translations from Zanata

For more information about this automat...

Reviewed:  https://review.opendev.org/693283
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=dbfb6a6fc90cb3367ed2248572089ee3cbafc15a
Submitter: Zuul
Branch:    feature/losf

commit 7cb92810acfd3cd598860aefadff8fba47e3efdd
Author: SeongSoo Cho <ppiyakk2@printf.kr>
Date:   Wed Nov 6 15:32:56 2019 +0800

Fix bug internal client's delete_object logs 499
    
    Internal Client's doest not read body for 404.
    So, When internal deleing object that not existed, proxy-server logs
    its response code to 499.
    
    In this patch, add a code that drain the reponse body to prevent
    unexpected disconnect in proxy-server
    
    Change-Id: I6d170fead798d0d539c69d27a6fa8a2d0123ca99
    Closes-Bug: #1835324

commit c512574e110c214396658743e75ae497ba76e2ff
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Nov 5 01:45:26 2019 +0000

Make our py2 func tests more explicitly py2
    
    ...because otherwise, I think they might be running under py3 right now.
    
    Change-Id: I96ad9e5ed87a7e08d06e889233730cec476d3af7

commit a505d02b3762ecbe7ea80b968ddec90ca3446549
Author: Chris Smart <chris.smart@humanservices.gov.au>
Date:   Sat Nov 2 20:48:47 2019 +1100

Tweak Swift AIO docs, update CentOS and Fedora
    
    The Swift AIO documentation is out of date for CentOS and Fedora.
    
    This patch updates the documentation to ensure that the instructions are
    clear and accurate.
    
    It also cleans up a few other sections along the way and adds some new
    headings to make it easier to read. Some explanatory notes are added,
    such as the need for XFS storage and test directories. XFS tmp loopback
    device is moved out of the user's home directory to /srv to match
    storage.
    
    Change-Id: Ieb0341536b7149c99139a1cf620828eba25f4bc6
    Signed-off-by: Chris Smart <chris.smart@humanservices.gov.au>

commit 281ffab2a52f348aa3f8ad39e800a43c6047e1cf
Author: Thiago da Silva <thiagodasilva@gmail.com>
Date:   Fri Nov 1 09:57:34 2019 +0200

pin lower-constraints to run against python 2.7
    
    Change-Id: I5c856012a04ef8f910c28b0248ea7bcda47e4816

commit 0c1b485ad671fc0abac15cd0bbec894e517e1dd9
Author: John Dickinson <me@not.mn>
Date:   Thu Oct 17 09:43:55 2019 -0700

exclude utf8 tests under py3
    
    These are known to not work until https://bugs.python.org/issue37093
    is addressed in CPython upstream.
    
    Change-Id: I4a6877907d14b632a9a477c887913488427b62b7

commit dd235b6ac0757e34805b041d331511c957764f3e
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Oct 28 11:27:58 2019 -0500

use request_helpers import namespace for tests
    
    Change-Id: I8e18ff2036312d3049a420f3332a6ea6110447c4

commit b75d5939352c405ab82ab52c3a9a1c3ad030aa26
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Oct 25 14:34:41 2019 -0500

Refactor timestamp iter in db backend tests
    
    Change-Id: I61d044a6d778e158af73a3f4d9fc75d642af0beb

commit 739c3704107bd991493472c18bfe7ac5f933077c
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Oct 18 07:10:33 2019 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I7a72be3568c2e8afd7211978cd917810c9af22ad

commit 2ac50079c1a9f2dfe2db4f9671d823086915e9d2
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Oct 17 09:52:50 2019 -0700

docs: Fix then/than grammar
    
    Change-Id: I948d65d0d3b58fcb13f8141fe095ccad4b8f1425
    Closes-Bug: #1848485

commit 86952dd3edf4ca45f8df7440cb8b6d31d9e415d7
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Oct 15 15:06:35 2019 -0700

tests: Stop invoking python just to get the real source directory
    
    While we're at it, stop checking for a seven-year-old version of nose --
    we already list a newer version in test-requirements.txt anyway.
    
    Change-Id: I97beace3094f11286730b99653dc1e8f699c5aec

commit d270596b6766bf59ea8eaa58a7d48dd2b368cf37
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 14:03:16 2019 -0700

Consistently use io.BytesIO
    
    Change-Id: Ic41b37ac75b5596a8307c4962be86f2a4b0d9731

commit 8efaa3556fdb21f741ff897dac4730c3c3654421
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 17:59:36 2019 -0700

py38: Provide readinto() interface for RingReader
    
    Otherwise pickle complains
    
       TypeError: file must have 'read', 'readinto' and 'readline' attributes
    
    Change-Id: I7ae0d7e47ae44286564f8ab7126e8ff131bc0de6

commit 3bd5653873bbb0ddc154481c3804ce6d801664b3
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 17:40:42 2019 -0700

Use `==` to compare against the empty string, not `is`
    
    py38 will start complaining like
    
       SyntaxWarning: "is" with a literal. Did you mean "=="?
    
    Change-Id: I799e288f66230a7847a13b7f303e4cd8e63eea59

commit 2edcd0b7b9dd02a041fd1452ea17efda5f27f21c
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 11:40:03 2019 -0700

py3: Fix swift-recon
    
    Avoid things like
    
        TypeError: '<' not supported between instances of 'NoneType' and 'int'
    
    Change-Id: I82edbf1ddcc28f86e97405d21db0b96249412eac

commit 4c66596e630ee63cf2cd54d216c8e443203e59fe
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 11:34:31 2019 -0700

py3: fix swift-dispersion-populate
    
    Change-Id: I1f140ae00cbd25b23c9a40ee91dccee8c7c15d81

commit 3e90ddb37d1d13977b69ae01117f50bab3495266
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 11:15:37 2019 -0700

swift-account-audit: clean up some error formatting
    
    "127.0.0.1s:sdas" is confusing at best.
    
    Change-Id: I37f78d5993082ac29b001e9563aa4b24fd009a27

commit 2f4fe56ca484c0efc11736aa323b2e63199e3513
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Oct 14 11:15:14 2019 -0700

py3: fix swift-account-audit
    
    Previously, we'd get a KeyError trying to read headers.
    
    Change-Id: I5d9f86784a3e39577ab010d29d8d03b26ffda357

commit 85d3658d6297efbe9a588d1b4364a2a68f9e5bb7
Author: Thomas Goirand <zigo@debian.org>
Date:   Fri Oct 11 14:12:01 2019 +0200

Fix kms_keymaster under Python 3
    
    Depending on how the key was stored in Barbican, it may come out of
    Castellan as a native string, which would not be suitable on Python 3.
    Now, check that the secret is a byte string, and if it isn't, encode as
    UTF-8 (to match Barbican's internal encoding).
    
    Change-Id: I6da047716c05e4f2a9e1e74ca19afb62e812d172
    Closes-Bug: #1847755

commit 405a2b2a55866b74f196ad81eaafedeee4615fa8
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sun Oct 13 21:55:54 2019 -0700

py3: Fix swift-drive-audit
    
    Walking through the kernel logs backwards requires that we open them
    in binary mode. Add a new option to allow users to specify which
    encoding should be used to interpret those logs; default to the same
    encoding that open() uses for its default.
    
    Change-Id: Iae332bb58388b5521445e75beba6ee2e9f06bfa6
    Closes-Bug: #1847955

commit 78a4070c90573fb8cf3f933329a27019d52aeea4
Author: Christian Schwede <cschwede@redhat.com>
Date:   Tue Jul 10 09:15:18 2018 +0000

Fix misleading error msg if swift.conf unreadable
    
    Several tools are returning a misleading error message if swift.conf is
    missing or not readable by the user, stating that the hash pre-/suffixes
    are missing. Let's fix this by catching the real issue down below.
    
    Change-Id: I7a47e6260ed51a3b7d9665b3a4510520429ae158

commit d667affb6f0b0b125901a0c002ddb00322a8e4f4
Author: Thiago da Silva <thiagodasilva@gmail.com>
Date:   Wed Oct 9 16:24:12 2019 +0200

Create segment container w/ same policy as primary
    
    When users upload an MPU object, s3api will automatically
    create a segment container if one doesn't already exist.
    
    Currently, s3api will create the segment bucket using the
    cluster's default storage policy. This patch changes that
    behavior to use the same storage policy as the primary bucket.
    
    Change-Id: Ib64a06868bd3670a1d4a1860ac29122e1ede7c39
    Closes-Bug: 1832390

commit 98f9781096f4f8f989e11f52fee0ec145fb5a173
Author: Christian Schwede <cschwede@redhat.com>
Date:   Sat Mar 12 06:36:46 2016 +0000

Add commands to ring-builder to change region / zone
    
    Currently one has to remove and re-add devices when the region or zone needs to
    be changed. Adding the subcommands set_region and set_zone simplifies this, now
    it is possible to change the region and/or zone easily. Note that there is no
    change to the required rebalancing; it is likely that data still needs to be
    moved within the cluster.
    
    This is mostly copy-n-paste of the existing set_weight subcommand and adopting
    tests accordingly. Some duplicated code in the tests has been aggregated as
    well.
    
    Change-Id: I37dd8e8ac24e2b0bb196758449a1d91a769b2e96

commit bb5fa0ea2e5be9d7537f31091015e816796499ba
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Fri May 3 14:51:47 2019 -0500

tests: bust md5 of object not footers
    
    The test test_PUT_ec_fragment_quorum_archive_etag_mismatch
    busts the md5 in server.py, so it ends damaging the md5 of
    footers instead of the fragment archive. It appears that the
    intention of the test was to check the integrity verification
    for fragment archive, so change the test to bust diskfile.py
    instead.
    
    Change-Id: I54a203bb637d5f5814e8df2b4297758b0b72adac

commit 3f7ffbc2ccc899cdf17ff1a338560ce3bc075e07
Author: Thiago da Silva <thiagodasilva@gmail.com>
Date:   Thu Oct 3 17:05:23 2019 +0200

Start building py3 docker images
    
    Change-Id: Id6f1075b2774536bb095de0904a1cdcf2b34b0fd

tags:

added: in-feature-losf

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-03: Fix included in openstack/swift 2.24.0

This issue was fixed in the openstack/swift 2.24.0 release.