Oozie isn't working

[Vadim Rovachev]

Step 1.
Create node group templates:
   1. namenode, jobtracker, oozie
   2. tasktracker, datanode

Create cluster template

Create cluster

Step 2.
Wait for 'Active' status for cluster

Step 3.
Go to oozie link

expected result:
open oozie page

actual result:
can't open page

INFO in /opt/oozie/logs/catalina.out:

ERROR: Oozie could not be started

REASON: org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot load JDBC driver class 'com.mysql.jdbc.Driver'

Stacktrace:
-----------------------------------------------------------------
org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Cannot load JDBC driver class 'com.mysql.jdbc.Driver'
        at org.apache.oozie.service.Services.loadServices(Services.java:291)
        at org.apache.oozie.service.Services.init(Services.java:208)
        at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
        at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
        at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
        at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
        at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
        at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
        at org.apache.catalina.core.StandardService.start(StandardService.java:525)
        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: <openjpa-2.2.2-r422266:1468616 fatal general error> org.apache.openjpa.persistence.PersistenceException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver'
        at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:102)
        at org.apache.openjpa.jdbc.conf.JDBCConfigurationImpl.getDBDictionaryInstance(JDBCConfigurationImpl.java:603)
        at org.apache.openjpa.jdbc.meta.MappingRepository.endConfiguration(MappingRepository.java:1518)
        at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:531)
        at org.apache.openjpa.lib.conf.Configurations.configureInstance(Configurations.java:456)
        at org.apache.openjpa.lib.conf.PluginValue.instantiate(PluginValue.java:120)
        at org.apache.openjpa.conf.MetaDataRepositoryValue.instantiate(MetaDataRepositoryValue.java:68)
        at org.apache.openjpa.lib.conf.ObjectValue.instantiate(ObjectValue.java:83)
        at org.apache.openjpa.conf.OpenJPAConfigurationImpl.newMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:967)
        at org.apache.openjpa.conf.OpenJPAConfigurationImpl.getMetaDataRepositoryInstance(OpenJPAConfigurationImpl.java:958)
        at org.apache.openjpa.kernel.AbstractBrokerFactory.makeReadOnly(AbstractBrokerFactory.java:644)
        at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:203)
        at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:156)
        at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:227)
        at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:154)
        at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:60)
        at org.apache.oozie.service.JPAService.getEntityManager(JPAService.java:266)
        at org.apache.oozie.service.JPAService.init(JPAService.java:162)
        at org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
        at org.apache.oozie.service.Services.setService(Services.java:354)
        at org.apache.oozie.service.Services.loadServices(Services.java:287)
        ... 26 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver'
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1429)
        at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
        at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
        at org.apache.openjpa.jdbc.sql.DBDictionaryFactory.newDBDictionary(DBDictionaryFactory.java:91)
        ... 46 more
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1420)
        ... 51 more

[Alexander Ignatov (aignatov)]

Actual issue is not in the Dashboard. It occurs because savanna-core sends some configs for Oozie with '\n' chars. UI removes it and then applies wrong configs during cluster creation. These configs are applied in the Oozie node as is follows:
  <name>oozie.services</name>
    <value> org.apache.oozie.service.SchedulerService, org.apache.oozie.service.InstrumentationService, org.apache.oozie.service.CallableQueueService, org.apache.oozie.service.UUIDService, org.apache.oozie.service.ELService, org.apache.oozie.service.AuthorizationService, org.apache.oozie.service.HadoopAccessorService, org.apache.oozie.service.MemoryLocksService, org.apache.oozie.service.DagXLogInfoService, org.apache.oozie.service.SchemaService, org.apache.oozie.service.LiteWorkflowAppService, org.apache.oozie.service.JPAService, org.apache.oozie.service.StoreService, org.apache.oozie.service.CoordinatorStoreService, org.apache.oozie.service.SLAStoreService, org.apache.oozie.service.DBLiteWorkflowStoreService, org.apache.oozie.service.CallbackService, org.apache.oozie.service.ActionService, org.apache.oozie.service.ActionCheckerService, org.apache.oozie.service.RecoveryService, org.apache.oozie.service.PurgeService, org.apache.oozie.service.CoordinatorEngineService, org.apache.oozie.service.BundleEngineService, org.apache.oozie.service.DagEngineService, org.apache.oozie.service.CoordMaterializeTriggerService, org.apache.oozie.service.StatusTransitService, org.apache.oozie.service.PauseTransitService, org.apache.oozie.service.GroupsService, org.apache.oozie.service.ProxyUserService </value>
  </property>

[Alexander Ignatov (aignatov)]

Such configs lead to the FATAL errors mentioned in bug description during cluster provisioning when starting Oozie service

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/49065

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/49065
Committed: http://github.com/stackforge/savanna/commit/78567b593493a3e205c4c4ea442b03b60f7a7ff2
Submitter: Jenkins
Branch: master

commit 78567b593493a3e205c4c4ea442b03b60f7a7ff2
Author: Alexander Ignatov <email address hidden>
Date: Tue Oct 1 01:38:00 2013 +0400

    Removing line breaks from default configs

    Fixes bug #1233192

    Change-Id: I14a74199fbbd3ba1acf479886ea01da6931bfeca