Test failures do not cause build failures

Bug #1646488 reported by Lars Tangvald on 2016-12-01
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mysql-5.7 (Ubuntu)
Undecided
Lars Tangvald
Xenial
Undecided
Unassigned

Bug Description

Some time back (5.6) the packaging was changed so failing tests during the build did not cause the build to fail. It's not clear why, as the test suite should at least be expected to pass on platforms that are supported by upstream.

[Impact]
Failing tests might not be noticed (there are occasionally some differences between build test runs and dep8 test runs). Failing builds can take longer because they're only spotted during dep8 runs.
Passing builds pointlessly spend time on the test runs, but ignore the result.

There are multiple elements to the change:
* Test failures will cause build failures on i386 and amd64 (for now. We will expand this list as we verify green test runs on other platforms.)
* We run the mtr suite with --force, meaning it doesn't abort on the first failure. This will help us get a complete list of which tests fail on which platforms
* We add a simple system for skipping spurious tests until they can be fixed or removed upstream (dep8 already has this)
* The test datadir_permissions has been consistently failing at build-time since 5.7.13, so we add a fix for it.

[Test case]
* Build package (the test datadir_permission will fail, but the failure is ignored)

[Regression potential]
* Spurious tests can cause build failures and not just dep8 failures (though this is technically a good thing, since they should be reported upstream and fixed.), so we might see more random build failures.
* --force makes builds failing due to tests take longer

Changed in mysql-5.7 (Ubuntu):
assignee: nobody → Lars Tangvald (lars-tangvald)
status: New → Fix Committed
Changed in mysql-5.7 (Ubuntu Xenial):
status: New → In Progress
Robie Basak (racb) on 2017-01-06
Changed in mysql-5.7 (Ubuntu Xenial):
milestone: none → xenial-updates
Robie Basak (racb) wrote :

This is present in 5.7.17-0ubuntu1 (present since 5.7.16-0ubuntu2).

Changed in mysql-5.7 (Ubuntu):
status: Fix Committed → Fix Released

Hello Lars, or anyone else affected,

Accepted mysql-5.7 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.17-0ubuntu0.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.7 (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Lars Tangvald (lars-tangvald) wrote :

Verified that building the 5.7.17-0ubuntu0.16.04.2 fails if there are test failures (tested by deliberately breakinga test).

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.7 - 5.7.17-0ubuntu0.16.04.2

---------------
mysql-5.7 (5.7.17-0ubuntu0.16.04.2) xenial; urgency=medium

  * Add libjson-perl dependency for test suite (LP: #1631338)
    The last two python tests in the mtr suite were rewritten
    in perl, and require this package. Added for mysql-testsuite
    and as a build-dep.
  * Build with DWITH_LZ4=system (LP: #1631339)
    Package was previously built with bundled liblz4, though it
    was specified as a build-dep.
  * Add support for custom datadir to systemd service (LP: #1574782)
    The service was reporting an error if no database could be found in
    /var/lib/mysql. It now checks the location specified in the config.
    Note that user must still handle apparmor access for custom datadir.
  * Fix copy of soft-link datadir to /var/lib/mysql-upgrade (LP: #1474212)
    Upgrade would sometimes fail if mysql-upgrade already contained a
    link copy from a previous upgrade.
  * Escape special characters in password (LP: #1598992)
    Special characters in the root password would cause syntax errors
    and postinst failures.
  * Failing tests on platforms supported by upstream fail build (LP: #1646488)
    A passing test suite is now enforced on i386 and amd64 platforms at
    build-time.
  * d/copyright: Updated with information about new source files
  * Unstable test main.xa_prepared_binlog_off disabled pending upstream fix.
    Upstream bug report: http://bugs.mysql.com/bug.php?id=83340
  * d/lintian-overrides: Updated line numbers

 -- Lars Tangvald <email address hidden> Fri, 03 Feb 2017 08:53:04 +0100

Changed in mysql-5.7 (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for mysql-5.7 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.