Update for Gentoo testing routines

Bug #830545 reported by Christian Faulhammer
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
bzr-buildbot-net
In Progress
High
Unassigned
Gentoo Linux
New
Undecided
Unassigned

Bug Description

Follow up from bug 614713:

>>>>> Christian Faulhammer writes:

    > A word from the Bazaar Gentoo maintainer:

You're very warmly welcome !

    > I disabled the tests

All the tests or just these ones ? How did you disable them ?

    > in Gentoo, the rest is fine, so 2.4 is the first version in a long
     > time that can be emerged with tests.

\o/

    > Vincent, I would like to see the test runs done within the Portage
     > system because that is the most common case where Bazaar users
     > will see the test suite in action.

I read that as the same goal we're trying to achieve in Ubuntu where the
 full test suite is run as part of the package build for our stable
 series.

I'm all for achieving this goal on as many supported platforms as is
 humanly possible.

    > There is a live ebuild in the Gentoo Bazaar overlay which always
     > fetches the current code.

And by that you mean lp:bzr or lp:bzr/2.4 ? I.e. do you track the trunk
 or the latest stable series ?

    > Should I open a new issue for that?

Yes, please do ! And let's try to align what is done on your side with
 what is done on
http://babune.ladeuil.net:24842/view/%20High/job/selftest-gentoo/ where
 bzr.dev (lp:bzr) runs the full test suite daily with only sporadic
 failures.

I.e.: I'd really like to address the distortion between the gentoo
 install where *all* tests pass every *day* without any significant
 failure and your install (probably not the right word nor concept[1])
 where there seem to be far too much failures for you to enable all the
 tests.

Feel free to contact me privately if you want to discuss this or let's
 do that on a new bug, both are fine with me.

[1]: I'm only trying to do a minimal amount of administration on the
 babune's gentoo slave while maintaining a "standard" gentoo install. If
 several configurations needs to be tracked (openssl/nss/you name it),
 I'd like to understand which ones are needed and how we can maintain
 them.
-------------------------------------------------------------------------------------------

1. Tests are disabled through the -x switch of bzr selftest, it is a hand-made collection.
2. The live ebuild builds trunk lp:bzr but needs an overhaul.
3. Gentoo's Portage runs the test suite if FEATURES=test is set, this can be done locally on the command-line for an emerge run or globally in make.conf.

My proposal is to regularly pull from the Bazaar overlay (lp:bzr-gentoo-overlay) to a local directory, go to dev-vcs/bzr in that directory and
 $ FEATURES=test ebuild bzr-9999.ebuild test

Maybe one can run:
 $ USE="ssl -gnutls -nss" emerge curl -1
 $ FEATURES=test ebuild bzr-9999.ebuild test

 $ USE="ssl gnutls -nss" emerge curl -1
 $ FEATURES=test ebuild bzr-9999.ebuild test

 $ USE="ssl -gnutls nss" emerge curl -1
 $ FEATURES=test ebuild bzr-9999.ebuild test

To find possible problems with different SSL implementations.

Is anything still unclear?

Revision history for this message
Martin von Gagern (gagern) wrote :

Thinking about other permutations worth trying:

1. Different versions of Python.
# echo 'USE_PYTHON="2.4 2.5 2.6 2.7"' >> /etc/make.conf
# emerge python:2.{4..7}
Doing so once should cause bzr to be installed and tested for all four versions of python, using the commands above.

2. Stable and testing.
It would be nice to have two setups of Gentoo, one at stable (e.g. ACCEPT_KEYWORDS=amd64 in /etc/make.conf) and the other at testing (e.g. ACCEPT_KEYWORDS=~amd64). This would influence the version numbers of a large number of direct and indirect dependencies, and should reflect common sets of package versions installed by Gentoo users.

Thanks for making this effort of getting the selftests running on Gentoo! I guess that there still will be a large number of possible combinations of configuration details out in the wild, way too many to test up front. So there may still be occasional bug reports coming in from users, in which case we'd have to identify the involved config aspects and perhaps add them to the testing server in one way or another. Think of it as a community testing effort...

Revision history for this message
Vincent Ladeuil (vila) wrote :

I tried to setup a babune job for that: http://babune.ladeuil.net:24842/job/selftest-gentoo-ebuild/

It fails because:

    $ FEATURES=test ebuild bzr-9999.ebuild test

requires root access. The test farm can't allow that for security reasons (in its actual design but changing it to allow root access is not planned so far).

Anyway, I tried to run it manually and got:

 * Applying bzr-0.90-tests-fix_root.patch ...

 * Failed Patch: bzr-0.90-tests-fix_root.patch !
 * ( /home/babune/babune/slaves/gentoo.local/workspace/selftest-gentoo-ebuild/dev-vcs/bzr/files/bzr-0.90-tests-fix_root.patch )
 *
 * Include in your bugreport the contents of:
 *
 * /var/tmp/portage/dev-vcs/bzr-9999/temp/bzr-0.90-tests-fix_root.patch.out

 * ERROR: dev-vcs/bzr-9999 failed (prepare phase):
 * Failed Patch: bzr-0.90-tests-fix_root.patch!
 *
 * Call stack:
 * ebuild.sh, line 56: Called src_prepare
 * environment, line 5760: Called epatch '/home/babune/babune/slaves/gentoo.local/workspace/selftest-gentoo-ebuild/dev-vcs/bzr/files/bzr-0.90-tests-fix_root.patch'
 * environment, line 2544: Called die
 * The specific snippet of code:
 * die "Failed Patch: ${patchname}!";
 *
 * If you need support, post the output of 'emerge --info =dev-vcs/bzr-9999',
 * the complete build log and the output of 'emerge -pqv =dev-vcs/bzr-9999'.
 * This ebuild is from an overlay named 'bazaar': '/home/babune/babune/slaves/gentoo.local/workspace/selftest-gentoo-ebuild/'
 * The complete build log is located at '/var/tmp/portage/dev-vcs/bzr-9999/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-vcs/bzr-9999/temp/environment'.
 * S: '/var/tmp/portage/dev-vcs/bzr-9999/work/bzr-9999'

This patch seems unnecessary as bzr.dev already calls self.requireFeature(features.not_running_as_root) for these tests.

But more importantly: is there a way to run this command without being root ?

Also note that the test farm expect selftest to be run with '--subunit' and further filter the results roughly:

   ./bzr selftest --subunit | subunit2junitxml -o tests.xml -f | subunit2pyunit

Is there a way we could converge ?

affects: bzr → bzr-buildbot-net
Changed in bzr-buildbot-net:
importance: Undecided → High
status: New → Incomplete
Revision history for this message
Vincent Ladeuil (vila) wrote :

@Martin: jenkins is good at tracking jobs as long as each job has a well-defined focus. In this specific case, we are talking about one job for each combination of a python version and a pycurl build variation), not a single job running all combinations.

Splitting the jobs makes it possible to track specific regressions instead of trying to make a single job run successfully (even once).

> Think of it as a community testing effort...

I do :)

Are you saying you'll be open to setup a babune slave on your side ?

Changed in bzr-buildbot-net:
status: Incomplete → In Progress
Revision history for this message
Martin von Gagern (gagern) wrote :

ebuild should not require root access. However, it does require access to /var/tmp/portage, which is usually restricted to the portage group iirc. So you could add the user to that group, change access permissions on that dir, or export PORTAGE_TMPDIR to point somewhere the user has write privileges for.

I'll have to look at how that babune thing works. Will do so in a week or so, am on holidays now.

Revision history for this message
Martin von Gagern (gagern) wrote :

Wrt bzr-0.90-tests-fix_root.patch: I'm just reviewing a mrege proposal by fauli which gets rid of that: https://code.launchpad.net/~fauli/bzr-gentoo-overlay/bzr-live-overhaul/+merge/72351

Revision history for this message
Vincent Ladeuil (vila) wrote :

Great ! adding babune to the portage group did the trick !

Currently running the job with lp:~fauli/bzr-gentoo-overlay/bzr-live-overhaul (instead of the default lp:bzr-gentoo-overlay) at http://babune.ladeuil.net:24842/job/selftest-gentoo-ebuild/6/console

Revision history for this message
Vincent Ladeuil (vila) wrote :

The run completed with two failures both seemingly caused by not being able to load a'.pyc' file just after compiling it... does that ring a gentoo bell somehow ?

Revision history for this message
Christian Faulhammer (fauli) wrote :

No, nothing starts ringing. I am on holiday, too, so no real time to look into it. The other issues (like pyftpdlib instead of medusa) will be looked in after my holiday.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.