FontTriplet errors when building on Jenkins when heavy load

[Anne Gentle]

This has happened on the operations-guide and openstack-training on both 1.8.0 and 1.9.2.

http://logs.openstack.org/19/45119/3/check/gate-openstack-manuals-validate/535fd23/console.html

[Tom Fifield (fifieldt)]

We should collect some more data perhaps?

Does David Cramer have any ideas?

Should we talk to infra>

[Anne Gentle (annegentle)]

I just showed it to David Cramer and he doesn't see a root cause.

I think we should ask -infra about the load as it seems to come up when lots of activity is going on.

[Anne Gentle (annegentle)]

Also, builds for the failed book works locally, so I suspect the Jenkins slave is at fault.

[Tom Fifield (fifieldt)]

load sounds like a good working theory for now

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45475
Committed: http://github.com/openstack/openstack-manuals/commit/f5bba0834b27f387e54443f26d6efc18732eb4c4
Submitter: Jenkins
Branch: master

commit f5bba0834b27f387e54443f26d6efc18732eb4c4
Author: Andreas Jaeger <email address hidden>
Date: Fri Sep 6 21:19:49 2013 +0200

    Handle removed files in tools/validate.py

    Previously if a file was removed, no checks were done that the file
    was not referenced anywhere.

    Now you get e.g.:
    Checking for removed files
      Removed file: doc/src/docbkx/common/section_keystone_cli_users_tenants_roles.xml
      Removed file: doc/src/docbkx/openstack-user-admin/src/section_cli_set_block_storage_quotas.xml
      Removed file: doc/src/docbkx/openstack-user-admin/src/section_cli_set_compute_quotas.xml
      Removed file: doc/src/docbkx/openstack-user-admin/src/section_keystone_cli_manage_projects_users.xml
      Removed file: doc/src/docbkx/openstack-user/src/section_cli_help.xml
      Removed file: doc/src/docbkx/openstack-user/src/section_cli_install.xml
      Removed file: doc/src/docbkx/openstack-user/src/section_cli_openrc.xml
      Removed file: doc/src/docbkx/openstack-user/src/section_cli_overview.xml
      Removed file: doc/src/docbkx/openstack-user/src/section_cli_version.xml
      File section_compute-configure-quotas.xml has an xi:include on deleted file ../openstack-user-admin/src/section_cli_set_compute_quotas.xml

    And validate.py exists with error code to mark this as failure.

    Also, limit number of parallel build jobs to 4 to reduce Jenkins load,
    hope we see less org.apache.fop.fonts.FontTriplet.
    Related-Bug: # 1221721

    Change-Id: I3e47217958840f61347f71b159414928f266d6d5

[Davanum Srinivas (DIMS) (dims-v)]

Here's a theory: Looks like the cache file is shared between Fop running in different jenkins jobs.
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.xmlgraphics/fop/1.1/org/apache/fop/fonts/FontCache.java#FontCache.getDefaultCacheFile%28boolean%29

So it's possible that the cache file is being written by one process is being picked by another...

[Tom Fifield (fifieldt)]

That is a good theory indeed!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47509

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47509
Committed: http://github.com/openstack/openstack-manuals/commit/dc7e0b27b1c94616b1671153668cd87844bbbab1
Submitter: Jenkins
Branch: master

commit dc7e0b27b1c94616b1671153668cd87844bbbab1
Author: Davanum Srinivas <email address hidden>
Date: Thu Sep 19 22:17:17 2013 -0400

    Possible fix for FontTriplet errors during heavy load

    Testing a theory about bad fop cache directory

    Fixes LP# 1221721

    Change-Id: Ic355f0d34b817ba52eb96b04b1ec0d2d1e7fe953