NullPointerException with docs maven build

[Tom Fifield]

Hi,

In the last week or so, we've been getting these errors intermittently.

2013-07-13 02:54:17.743 | Parsing POMs
2013-07-13 02:54:18.854 | ERROR: Processing failed due to a bug in the code. Please report this to <email address hidden>
2013-07-13 02:54:18.854 | java.lang.NullPointerException
2013-07-13 02:54:18.854 | project=hudson.maven.MavenModuleSet@43c804bf[gate-openstack-manuals-cli-guide]
2013-07-13 02:54:18.854 | project.getModules()=[hudson.maven.MavenModule@561734ad[gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][relativePath:doc/src/docbkx/cli-guide]]
2013-07-13 02:54:18.854 | project.getRootModule()=hudson.maven.MavenModule@561734ad[gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][relativePath:doc/src/docbkx/cli-guide]
2013-07-13 02:54:18.854 | FATAL: null
2013-07-13 02:54:18.855 | java.lang.NullPointerException

(from http://logs.openstack.org/36928/1/check/gate-openstack-manuals-cli-guide/605/console.html.gz)

a recheck makes them go away.

Not sure what the cause is

[David Cramer (david-thingbag)]

Hi there,
I'm not sure what "a recheck makes them go away" means. Could you clarify?

Thanks,
David

[Lorin Hochstein (lorinh)]

David:

It means that the bug is intermittent.

"recheck" is a way you can kick Jenkins to run the gate tests again.

If you add a comment "recheck bug <bugnumber>" to Gerrit (where "<bugnumber>" corresponds to the infrastructure bug that is breaking things, in this case it's 1201035), then Jenkins will re-run the gate tests.

You can also do "recheck no bug", but in this case there's a specific bug for tracking this issue.

[Anne Gentle (annegentle)]

With our CI Jenkins/Gerrit system, you can click a Recheck button on the
Gerrit interface on the patch at review.openstack.org, which re-runs the
build.

My hunch is that it's a memory problem, but I guess that because the Gerrit
gate has had a lot of jobs going through it lately.

On Mon, Jul 15, 2013 at 9:15 AM, David Cramer <email address hidden>wrote:

> Hi there,
> I'm not sure what "a recheck makes them go away" means. Could you clarify?
>
> Thanks,
> David
>
> --
> You received this bug notification because you are subscribed to
> OpenStack.
> https://bugs.launchpad.net/bugs/1201035
>
> Title:
> NullPointerException with docs maven build
>
> Status in OpenStack Core Infrastructure:
> New
> Status in OpenStack Manuals:
> Confirmed
>
> Bug description:
> Hi,
>
> In the last week or so, we've been getting these errors
> intermittently.
>
> 2013-07-13 02:54:17.743 | Parsing POMs
> 2013-07-13 02:54:18.854 | ERROR: Processing failed due to a bug in the
> code. Please report this to <email address hidden>
> 2013-07-13 02:54:18.854 | java.lang.NullPointerException
> 2013-07-13 02:54:18.854 | project=hudson.maven.MavenModuleSet@43c804bf
> [gate-openstack-manuals-cli-guide]
> 2013-07-13 02:54:18.854 |
> project.getModules()=[hudson.maven.MavenModule@561734ad
> [gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][relativePath:doc/src/docbkx/cli-guide]]
> 2013-07-13 02:54:18.854 |
> project.getRootModule()=hudson.maven.MavenModule@561734ad
> [gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][gate-openstack-manuals-cli-guide/org.openstack.docs:openstack-cli-guide][relativePath:doc/src/docbkx/cli-guide]
> 2013-07-13 02:54:18.854 | FATAL: null
> 2013-07-13 02:54:18.855 | java.lang.NullPointerException
>
>
> (from
> http://logs.openstack.org/36928/1/check/gate-openstack-manuals-cli-guide/605/console.html.gz
> )
>
>
> a recheck makes them go away.
>
> Not sure what the cause is
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openstack-ci/+bug/1201035/+subscriptions
>

[Tom Fifield (fifieldt)]

Hi David,

Based on what happens in the Gerrit review system, certain jobs in Zuul (http://ci.openstack.org/zuul.html) are run.

For docs, we run a job that tries to build the guide both on upload of a patch set, and after the merge has been approved to go ahead.

Sometimes this job can fail , and zuul/jenkins/gerrit is trained to re-perform the tests, if you leave a code review that contains the words "recheck no bug" or "recheck bug ####".

Essentially, if it works after a 'recheck', and is happening on several reviews, it means it's a intermittent issue that should be investigated ...

This is covered in https://wiki.openstack.org/wiki/GerritJenkinsGithub#Test_Failures

[David Cramer (david-thingbag)]

Ok, thanks. Giving Java more heap space is worth a try (though usually I see OutOfMemory errors rather than an NPE). You can set the environment variable MAVEN_OPTS from the Jenkins job. In the Build section click Advanced and you should see a MAVEN_OPTS field. Set it to something like:

-Xms1024m -XX:MaxPermSize=1024m -Xmx1024m -Xrs

David

[James E. Blair (corvus)]

It is possible that this error is related to a bug in the gearman-plugin. At least, in one other job, we got an NPE with a stacktrace that pointed squarely at the gearman-plugin. Unfortunately this instance doesn't have a stacktrace, so we can't confirm it's the same, but it seems reasonable that it may be related.

Khai is going to work on fixing the gearman-plugin issue, so let's see if this happens again after we get that into production. Or, if you see a clear stacktrace with this error in the future, please note here.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/37108
Committed: http://github.com/openstack-infra/gearman-plugin/commit/13b4b52968218290fc03282b2d095abb315c32b0
Submitter: Jenkins
Branch: master

commit 13b4b52968218290fc03282b2d095abb315c32b0
Author: zaro0508 <email address hidden>
Date: Mon Jul 15 11:27:12 2013 -0700

    ignore non-deterministic build failure and log it.

    maven builds intermittently fail due to something in the code.
    essentially its the same issue as in bug 1201035.

    This patch makes it so the build ignores the error and just logs
    it so that we can have a record of the error. Hopefully we
    can use logstash to query for a pattern to determine the
    root cause of this problem.

    The error on this build:
      java.lang.NullPointerException
          at hudson.plugins.gearman.ExecutorWorkerThread.registerJobs(ExecutorWorkerThread.java:150)
          at hudson.plugins.gearman.SaveableListenerImpl.onChange(SaveableListenerImpl.java:68)
          at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:78)
          at hudson.model.AbstractItem.save(AbstractItem.java:474)
          at hudson.model.Job.save(Job.java:154)
          at hudson.model.AbstractProject.save(AbstractProject.java:273)
          at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:915)
          at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:622)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
          at hudson.model.Run.execute(Run.java:1575)
          at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:477)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:241)

    Change-Id: I16fb0261312fe0aef55d2f2b6c83652333aed05a

[Khai Do (zaro0508)]

I haven't seen the NPE again after the above fix.

[Anne Gentle (annegentle)]

Me neither, thanks for fixing!