I'm pasting in this email text that I originally sent so that others can also see it (with one slight addition *).

I've been working on and exploring this bug #1198277 [0]. I don't have a
solution yet, but I've noticed a couple of odd things during my pondering.
I fear this email is a long one but contains my brain dump of the situation.

I've been poking around the device itself (I put a sleep 30m into the jenkins
job so the device was still marked as in use while I did so).
What I found is that I was unable to do any of the things that I could do
manually on my local desktop/device.

For instance what I was doing in a terminal was:

  1. Starting `gallery-app -testability`
  2. using a simple python script to determine the bus name (i.e. :1.209)
  3. using qdus and dbus-send to query it manually (i.e. qdbus :1.209 will
     return the available object paths. Specifically interested in
     /com/canonical/Autopilot/Introspection for instance.)

When I first logged onto the device step 3 didn't work as expected, I got an
error (examples to come).
After rebooting the device however my manual tests worked, it wasn't until after
I ran the memory usage did it get into the state were my tests didn't work.

So I'm not entirely certain at this stage that the issue is specifically an
Autopilot testcase bug.

Description and steps for testing and reproducing:.

I'm using this script here [1]
What it does is launch an introspectable application (i.e. gallery-app
-testability) and queries it using qdbus/dbus-send. It outputs the object paths
and PIDs so we can ensure we're looking at the correct objects.
*[I tested this script with the shell both enabled and disabled and it worked multiple times in a row with both]

The process to my madness is this:

  1. Run the pid test script `./test_pids.sh` -> output will show success,
     example here [2]
  2. Run the memory_usage_measurement tests -> will fail as expected
     (utah-autopilot run -v ubuntu_test_cases.memory_usage_measurement.tests.test_memory_usage.MemoryUsageTests.test_scenario)
  3. Re-run the pid test script -> The output will show failure [3]

Also, during this I realised this:

  1. Run the memory_usage_measurement tests -> Will fail at the gallery app step
  2. Run the memory_usage_measurement tests again -> Will fail at the browser
     app step (i.e. the first app launch)

So after the first run a second run will fail with 'raise RuntimeError("Search
criteria returned no results")' at browser launch (where as the first run does
this at the gallery-app launch step).

So there we go, that's my findings so far. To be honest I'm not sure what the
issue is at the moment so I'm wondering if anyone else might be able to shed
some light for me or point me in a new direction?

Is there something happening in the test code that is causing the system to do
something weird? Perhaps something on the device is crashing in an odd way,
either dbus or libautopilot-qt or <something>.

I'll ponder this tonight and try again in the morning.
Please let me know if I'm missing any part of my explanation or you would like
a clarification of any points.

-- Links --
[0:bug] https://bugs.launchpad.net/autopilot/+bug/1198277

[1:test_pids.sh] http://pastebin.ubuntu.com/5864109/

[2:passing example] http://pastebin.ubuntu.com/5864112/
  Note: The points of success are: lines 52, 53-60, 62-70
  Note: The test script will succeed multiple times in a row before running the tests

[3:failing example] http://pastebin.ubuntu.com/5864118/
  Note: The output of failure is lines 23-27 should look more like 53-60 or 62-70 from [2]
  Note: Also worth noting now that I'm looking at it, the run after the memory
        usage test; gallery-app doesn't output as much stuff as in the successful

Regards,
Chris