canonical-certification-server needs the iperf tests added and needs to honor settings in /etc/xdg/canonical-certification.conf

[Kent Baxley]

canonical-certification-server doesn't seem to have the iperf tests included in it, nor does it seem to honor any of the settings listed in /etc/xdg/canonical-certification.conf for the iperf server.

Steps to reproduce:

Install system with 14.04 server.

Install checkbox-certification-server from the hardware-certification public ppa or the checkbox-dev ppa (checkbox-ng and plainbox-provider-certification-server in this case).

In the /etc/xdg/certification.conf file, specify the iperf server that you want the SUT to point to.

Fire up c-c-s

Actual results: Only the network-detect tests are run. The iperf test isn't listed as an option in the testing menu, either. This of course means that any settings you configured on the SUT for iperf aren't going to be of any value since the test is never run.

[Daniel Manrique (roadmr)]

Looks like either the iperf package itself isn't installed (unlikely, since in this case you'd see the test in the selection list but it wouldn't run due to a missing requirement), or it's not in the whitelist (this more closely matches the behavior you saw).

I'll have a look with your reproduction instructions and see what happened here.

[Rod Smith (rodsmith)]

I noticed on one failed test that the returned text included the message that '"package.name == 'nmap'" evaluates to false'. I therefore tried installing the nmap package, and it worked; I got:

DEBUG:root:Passed benchmark
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.51 port 44880 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.7 sec 1.00 GBytes 804 Mbits/sec

Transfer speed: 804 Mb/s
80.40% of theoretical max 1000Mb/s

If the test really needs nmap, then a dependency should be included.

One other point: I wasn't aware that the iperf test was supposed to be reading data from /etc/xdg/certification.conf; I set the location of the iperf server in /etc/checkbox.d/network.cfg. The iperf server was running on 10.0.0.1, and that's what I set in that file, so I assume that's the file that this test was reading. As I'm writing up the self-testing guide for this software, it's important that somebody tell me these things.

[Daniel Manrique (roadmr)]

Hi Rod, so to be clear, if you install nmap and put the configuration in /etc/xdg/certification.conf, do things work as expected?

We can certainly add the nmap dependency to the provider package that supplies these jobs and whitelists. If it's not there already, it's an omission for which I apologize, as it caused you guys to waste time figuring out.

As for adding things in /etc/checkbox.d/network.cfg, where did you find instructions specifying to do this in that file? if you saw this in the existing server testing guide, then of course as you have noticed, things have changed, and had to, to accomodate the engine change from checkbox to plainbox.

I realize there may have been some miscommunication there, but I'm unsure what we could have done to better convey these changes. When we implemented those features (using certification.conf instead of checkbox.d/network.cfg) we discussed it extensively with Jeff, so I assumed, perhaps too overarchingly, that "people were aware" of the change.

One way that comes to mind is notifying people about this change using a relevant mailing list (which one would you say is relevant? perhaps there's one I'm not subscribed to, and I should be as we have a lot of interlock as we work together on getting these tools ready). And of course, when in doubt, you're always welcome to ask us, either by e-mail or by filing bug reports.

So actions:

- Could you please confirm that installing nmap and adding the configuration in certification.conf enabled the test to run?
- Could you confirm where you got the information about the old network.cfg mechanism, so we can decide whether we need to update it?

THanks!

[Rod Smith (rodsmith)]

To clarify: I have not tested with nmap installed and configuration settings in /etc/xdg/certification.conf, just with nmap installed and configuration settings in /etc/checkbox.d/network.cfg. FWIW, the way I've been installing everything does leave an /etc/checkbox.d/network.cfg file on the system. Perhaps I'm installing some old packages that are no longer necessary, or perhaps something's continuing to create those old files. Either way, the presence of the old file contributed to my using it. I'm currently running a test that's restored /etc/checkbox.d/network.cfg to its original state and that places the server address in /etc/xdg/certification.conf. The information on using /etc/checkbox.d/network.cfg, BTW, was from the old 12.04 self-test guide, which I've used as the basis for the updated 14.04 self-test guide (current draft here: https://docs.google.com/a/canonical.com/document/d/1G_8MbdegChuPlXyKLbHxfr7Q0X9zTvCGAIzW9NBv8CA/).

I just reviewed my e-mail, and I see that I WAS informed of the new filename in an e-mail from Jeff, but that data obviously didn't "stick" in my mind. What would be more helpful than a mailing list post, IMHO, would be a document (a Google document or wiki page) maintained by the developers that outlines the current process. This document could (indeed, MUST) be updated as things change. It needn't be very detailed or "neat," just highlight the major points and changes from the previous version.

[Rod Smith (rodsmith)]

An update: When run with the iperf information in /etc/xdg/certification.conf but not in /etc/checkbox.d/network.cfg, the following error occurs:

ERROR:root:Please supply target via: /etc/checkbox.d/network.cfg

[Kent Baxley (kentb)]

Rod,

I've also got a bug opened up on the test's missing references, etc. to the /etc/xdg file. You can track that here:

https://bugs.launchpad.net/checkbox/+bug/1296980

[Jeff Lane  (bladernr)]

I'm bumping this to critical... it will very quickly be gating certification efforts for 14.04 zero day certification. As in, we start testing next week :/

[Jeff Lane  (bladernr)]

Also:

I've been poking at this... there are several issues:

1: It does indeed appear that the file /etc/xdg/canonical-certification.conf is being completely ignored.

I uncommented the test_target_iperf line and gave it an IP, and then printed os.environ inside the network script itself and the env var that SHOULD be created isn't there.

I also changed the welcome_text and discovered that it too is not honored, so it appears the entire file is skipped.

2: the script (network) also parses the old network.cfg file in the interface test section, so this bit will also dump a traceback unless it's configured like the iperf test is.

[Jeff Lane  (bladernr)]

<zyga> bladernr: what needs to happen there? is it a packaging issue (misisng iperf) or config issue (missing config for one of the tests)
<bladernr> Oh there are a couple things... packaging, we've discovered, is missing iperf and nmap
<zyga> bladernr: ok, I can fix packaging, anything else?
<bladernr> also, the file that roadmr created to hold the test target data (/etc/xdg/canonical-certification.conf) is not being read by plainbox
<bladernr> or canonical-certification-server specifically
<bladernr> so the network test never gets things like iperf target, and the virtualization test will never see any overrides for image location
<bladernr> (added a comment to the bug just now)
<bladernr> and I can't for the life of me, find where in the checkbox trunk the base copy of canonical-certification.conf lives...
<bladernr> :/
<zyga> bladernr: we read '/etc/xdg/canonical-certification.conf' (amongst others)
<zyga> bladernr: can you run checkbox check-config and see if you have that file listed as present?
<zyga> bladernr: plainbox does *not* read '/etc/xdg/canonical-certification.conf' though
<zyga> bladernr: only checkbox-ng and friends do
<bladernr> hrmm....
<zyga> bladernr: oh, maybe that is the problem
<zyga> bladernr: does that job run as root?
<bladernr> yeah, no checkbox
<zyga> bladernr: ok, no worries, just use /etc/xdg/plainbox.conf instead
<zyga> bladernr: so apart from misconfigurartion, all that is needed is iperf and nmap in dependencies, right?
<bladernr> I think so.
<zyga> bladernr: on the server provider
<bladernr> right
<zyga> bladernr: ok, I'll get that fix for you now
<zyga> bladernr: would you mind adding this irc log to the bug report please
<bladernr> ok...
<bladernr> so I don't have plainbox.conf, and renaming the canonical-certification.conf file doesn't work either

[Jeff Lane  (bladernr)]

<zyga> bladernr: iperf is already a dependency of plainbox-provider-certification-server
<bladernr> ahh, ok, so roadmr fixed that one, it's just nmap then
<zyga> bladernr: do you generate the .conf file in some way?
<bladernr> I do not...
<bladernr> :/
<zyga> bladernr: so where do you get it from?
* zyga is sorry for not getting the bigger picture
<bladernr> that's part of the problem... it doesn't appear to exist in the trunk, yet it's created on install
<bladernr> and no worries, I don't know either... it's a roadmrism
<zyga> bladernr: where are you seeing this? in checkbox-satellite workflow?
<bladernr> I tried finding it in the trunk, and it's only referenced here: checkbox-ng/checkbox_ng/config.py
<zyga> bladernr: right
<bladernr> no actual file exists in trunk :/
<zyga> bladernr: I mean where do you see the bug
<bladernr> on, on a 14.04 system
<zyga> bladernr: how do you test that the bug occurs (who sets up the environment there)
<bladernr> the packaging does... e.g manually running apt-get install canonical-certification-server
<zyga> bladernr: https://code.launchpad.net/~zkrynicki/checkbox/fix-1296965/+merge/212661 <- one part of that bug
<zyga> ok
<zyga> bladernr: so let's recap
<zyga> bladernr: missing nmap (check, fixed by that proposal above)
<zyga> bladernr: missing iperf (check, fixed earlier)
<zyga> bladernr: missing plainbox.conf (when using plainbox for running tests), (check, something you need to provide on-site for proper test run)
<zyga> bladernr: is there anything else to this bug?
<bladernr> not that I'm aware of... I think once those bits are in place, anything else will be bugs in the network script and that can be handled separately
<bladernr> ^^ and possibly the virtualization script
<bladernr> or rather, should be handled separately
<zyga> bladernr: ok, just open new bugs and ping me if you need support

[Rod Smith (rodsmith)]

To clarify: nmap must be installed on the system being tested, iperf must be installed on some other system (such as the MAAS server).

[Jeff Lane  (bladernr)]

So doing some testing also and I've discovered that neither the [common] nor [environment] sections of canonical-certification.conf are honored.

HOWEVER, if I add the [sru] section to set a Secure ID prior to testing, that IS honored. On a fresh 14.04 system I installed c-c-s from the dev ppa and edited the file /etc/xdg/canonical-certification.conf to look like this:

root@CCS-SUT:~# cat /etc/xdg/canonical-certification.conf
[common]
welcome_text = THIS IS JEFFS HOUSE OF PAIN!!!

 Abandon all hope, ye who enter here...

[environment]
# For virtualization testing.
# Uncomment the following 2 lines and set the proper parameters:
#kvm_timeout:
#kvm_image:
# For network testing.
# Uncomment the following 4 lines and set the proper parameters:
#test_target_ftp: your-ftp-server.example.com
#test_user: anonymous
#test_pass:
test_target_iperf: 10.128.242.171

[sru]
secure_id=a00D000000M60SE

as you can see, I changed the welcome text completely, and added a test_target_iperf variable to point to my iperf server, and added the secure_id variable.

Then I ran through it and deselected everything but the ethernet tests.

As expected the ethernet tests failed

[Zygmunt Krynicki (zyga)]

Does the job advertise environment variables using the environ field in the job definition?

Sent from my BlackBerry 10 smartphone.
  Original Message
From: Jeff Lane
Sent: środa, 26 marca 2014 23:35
To: <email address hidden>
Reply To: Bug 1296965
Subject: [Bug 1296965] Re: canonical-certification-server needs the iperf tests added and needs to honor settings in /etc/xdg/canonical-certification.conf

So doing some testing also and I've discovered that neither the [common]
nor [environment] sections of canonical-certification.conf are honored.

HOWEVER, if I add the [sru] section to set a Secure ID prior to testing,
that IS honored. On a fresh 14.04 system I installed c-c-s from the dev
ppa and edited the file /etc/xdg/canonical-certification.conf to look
like this:

root@CCS-SUT:~# cat /etc/xdg/canonical-certification.conf
[common]
welcome_text = THIS IS JEFFS HOUSE OF PAIN!!!

Abandon all hope, ye who enter here...

[environment]
# For virtualization testing.
# Uncomment the following 2 lines and set the proper parameters:
#kvm_timeout:
#kvm_image:
# For network testing.
# Uncomment the following 4 lines and set the proper parameters:
#test_target_ftp: your-ftp-server.example.com
#test_user: anonymous
#test_pass:
test_target_iperf: 10.128.242.171

[sru]
secure_id=a00D000000M60SE

as you can see, I changed the welcome text completely, and added a
test_target_iperf variable to point to my iperf server, and added the
secure_id variable.

Then I ran through it and deselected everything but the ethernet tests.

As expected the ethernet tests failed

--
You received this bug notification because you are a member of Checkbox
Bug Wranglers, which is subscribed to checkbox.
https://bugs.launchpad.net/bugs/1296965

Title:
canonical-certification-server needs the iperf tests added and needs
to honor settings in /etc/xdg/canonical-certification.conf

To manage notifications about this bug go to:
https://bugs.launchpad.net/checkbox/+bug/1296965/+subscriptions

[Zygmunt Krynicki (zyga)]

Could someone please verify that this is fixed with the new packaging scheme in the dev ppa (trusty only)?

[Daniel Manrique (roadmr)]

OK, with the new set of packages in the PPA (notice that canonical-certification-server is now gone), I did:

$ sudo apt-get install checkbox-ng plainbox-provider-certification-server

- nmap is installed by plainbox-provider-certification-server
- Said provider package also installs a sample /etc/xdg/canonical-certification.conf file
- the canonical-certification-server *binary* is present as part of checkbox-ng
- canonical-certification-server --check-config shows me it's reading the environment variables correctly.

Three problems remain: first, the config object is not being used when running tests (a fix is committed but not yet released to the PPA: bug 1298166.

Second, the template file needs an update; I assumed values were uppercased automatically but they are not; thus, the lowercase config variables as seen in the example config file will not be taken by the network script. A fix is here:

https://code.launchpad.net/~roadmr/checkbox/ppa-packaging-plainbox-provider-canonical-certification-update-config-file-environment-case/+merge/213271

Third, we still need to provide a *single* package to install this, as remembering to install two packages is confusing. This will be upcoming in the next few days.

Once the first two fixes are merged, things should work. Already from the summary I posted above, things seem to be mostly in good shape now.

[Zygmunt Krynicki (zyga)]

Nice summary. Thanks Daniel.

ZK

On Fri, Mar 28, 2014 at 3:22 PM, Daniel Manrique <
<email address hidden>> wrote:

> OK, with the new set of packages in the PPA (notice that canonical-
> certification-server is now gone), I did:
>
> $ sudo apt-get install checkbox-ng plainbox-provider-certification-
> server
>
> - nmap is installed by plainbox-provider-certification-server
> - Said provider package also installs a sample
> /etc/xdg/canonical-certification.conf file
> - the canonical-certification-server *binary* is present as part of
> checkbox-ng
> - canonical-certification-server --check-config shows me it's reading the
> environment variables correctly.
>
> Three problems remain: first, the config object is not being used when
> running tests (a fix is committed but not yet released to the PPA: bug
> 1298166.
>
> Second, the template file needs an update; I assumed values were
> uppercased automatically but they are not; thus, the lowercase config
> variables as seen in the example config file will not be taken by the
> network script. A fix is here:
>
> https://code.launchpad.net/~roadmr/checkbox/ppa-packaging-plainbox-
> provider-canonical-certification-update-config-file-environment-
> case/+merge/213271
>
> Third, we still need to provide a *single* package to install this, as
> remembering to install two packages is confusing. This will be upcoming
> in the next few days.
>
> Once the first two fixes are merged, things should work. Already from
> the summary I posted above, things seem to be mostly in good shape now.
>
> --
> You received this bug notification because you are a member of Checkbox
> Bug Wranglers, which is subscribed to checkbox.
> https://bugs.launchpad.net/bugs/1296965
>
> Title:
> canonical-certification-server needs the iperf tests added and needs
> to honor settings in /etc/xdg/canonical-certification.conf
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/checkbox/+bug/1296965/+subscriptions
>

[Daniel Manrique (roadmr)]

The only item remaining from comment #15 is providing a single seed package for this. The bug itself is fixed, I'll mark as such.