perf failure on panda (omap4)

[Avik Sil]

Though perf lists the PMU events but fails to gather performance counter statistics on panda

# perf list

List of pre-defined events (to be used in -e):

  cpu-cycles OR cycles [Hardware event]
  instructions [Hardware event]
  cache-references [Hardware event]
  cache-misses [Hardware event]
  branch-instructions OR branches [Hardware event]
  branch-misses [Hardware event]
  bus-cycles [Hardware event]

  cpu-clock [Software event]
  task-clock [Software event]
  page-faults OR faults [Software event]
[...]

# perf stat sleep 1
  Error: cache-misses event is not supported.
  Fatal: Not all events could be opened.

[Ricardo Salveti (rsalveti)]

Which kernel version are you using? Can you also reproduce this with the TI LT kernel (ones available with the lt-panda-hwpacks)?

[Fathi Boudra (fboudra)]

For the record, see also the related threads:
http://lists.linaro.org/pipermail/linaro-dev/2011-August/006901.html
http://lists.linaro.org/pipermail/linaro-dev/2011-September/007211.html

and https://launchpad.net/bugs/702999

[Fathi Boudra (fboudra)]

From Dave L. (http://lists.linaro.org/pipermail/linaro-dev/2011-September/007560.html) :

I have been unable to reproduce the boot hang problem after unreverting the interrupt patch. Can those experiencing it please verify they are using the latest released lmc tools to create their SD card? And can you please send me your config file and Pandaboard revision?

It has been suggested that the limited sample count in the given example is consistent with the fact the ARM oprofile code uses HZ as the sampling frequency (even though it uses a separate timer). I'm not yet familiar with the x86 code for setting the profiling interval on the fly but Frederic Turgis's suggesting of doing the same thing on ARM makes a whole lot of sense. We should make this a new requirement.

Once the interrupt issue is resolved I might suggest sampling cpu-cycles as a workaround to real-time sampling granularity, except that there apparently is an issue with reliably getting interrupts from the PMU. Does anyone know if this is still a problem in the A9 (I've only seen it discussed regarding the A8)? If it's still an issue I think it simply kills using PMU event counters with oprofile.

We need to do a little work to make configuring hardware events counters into the kernel easier. A recent change means that you need to set at least a couple independent config options for this. This should be simple to fix.

[Vinod Krishnamoni (vinod-krishnamoni)]

Does the fact that this is not attached to any milestone for linux-linaro-omap (Ubuntu) mean the fix will not be delivered into a linux-linaro-omap (Ubuntu) release?

[hehemouse (hehemouse)]

I can't get any PMU interrupt on pandaboard, although I am not using perf or oprofile. I am just write my own code to operate the PMU. The overflow register shows that overflow happens, but the PMU interrupt is not triggered. Is this a bug of hardware or a bug of BSP?

I am using source code in http://git.linaro.org/gitweb?p=kernel/linux-linaro-2.6.39.git;a=summary, but with the following patch:
http://lists.linaro.org/pipermail/linaro-dev/2011-April/004019.html

[Al Grant (al-grant)]

Why was the milestone removed?

[Fathi Boudra (fboudra)]

On 17 November 2011 19:25, Al Grant <email address hidden> wrote:
> Why was the milestone removed?

It has been removed for Linaro Ubuntu project. Developer Platform
isn't working actively to resolve this bug. We rely on TI LT to get it
fixed. Dave Long was looking initially at the issue but we don't have
received any ETA since a long time.

[David Long (dave-long)]

The plan is to revert the revert that broke this very shortly. That change will require soak time in case the boot failure problem that caused the revert in the first place resurfaces. Therefore we are targeting December for release of this change.

[warmcat (andy-warmcat)]

Guys on tilt-tracking we have a patch from Dave Long

http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git;a=patch;h=32a92107a7137ea32a2209f83dd4bae42ad4ec7c

that re-enables these interrupts for 4430 Panda. We don't get any boot crashes so far.

It doesn't hurt 4460 Panda, but it doesn't succeed get any results on that yet.

tilt-tracking -->

http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git;a=shortlog;h=refs/heads/tilt-tracking

[Ricardo Salveti (rsalveti)]

Seems perf is completely broken with tilt-linux-linaro-3.1 this time.

This is what I'm getting with kernel 3.1.0-1402-linaro-lt-omap #5rsalveti1, based on today's tilt-linux-linaro-3.1 head: http://paste.ubuntu.com/770821/

[Paweł Moll (pawel-moll)]

It may be the same issue as https://bugs.launchpad.net/linaro-landing-team-freescale/+bug/893653/comments/6

[Will Deacon (will-deacon)]

Hi guys,

This looks like the same problem the imx guys were seeing in:

https://bugs.launchpad.net/linaro-landing-team-freescale/+bug/893653

I'll mail Nico on linaro-dev about including the fix from mainline. Note that OMAP isn't providing PMU platform_device so perf isn't going to work with these fixes, you'll need to pick some more out of linux-next and off the list for that.

Will

[warmcat (andy-warmcat)]

About the platform_device, In tracking, for 4430 it's enabled and believed workable. We can't get the unit to start up on 4460 yet.

[warmcat (andy-warmcat)]

Will, I just updated tilt-linux-linaro-3.1 with latest linux-linaro-3.1 from Nicolas tonight.

[warmcat (andy-warmcat)]

... and tilt-linux-linaro-3.1 now has the platform device patch in via Dave.

[Ricardo Salveti (rsalveti)]

Avik, can you verify this bug with the 11.12 RC image and post back the results? Thanks.

[Avik Sil (aviksil)]

With 11.12 RC image, perf seems to work fine:

# perf stat sleep 1

 Performance counter stats for 'sleep 1':

         11.810303 task-clock # 0.011 CPUs utilized
                 1 context-switches # 0.000 M/sec
                 0 CPU-migrations # 0.000 M/sec
               148 page-faults # 0.013 M/sec
           3514014 cycles # 0.298 GHz
                 0 stalled-cycles-frontend # 0.00% frontend cycles idle
                 0 stalled-cycles-backend # 0.00% backend cycles idle
           1777780 instructions # 0.51 insns per cycle
            197443 branches # 16.718 M/sec
             71889 branch-misses # 36.41% of all branches

       1.037902836 seconds time elapsed

[warmcat (andy-warmcat)]

Dave's also found the remaining 4460-specific PMU init issue, and his fix

http://git.linaro.org/gitweb?p=landing-teams/working/ti/kernel.git;a=patch;h=fab7a4a5a0cb4c8fc5692970be6e62aece99599c

is on tilt-3.1, tilt-linux-linaro-3.1 (linaro ubuntu basis) and tilt-tracking.