"[ClassLoader] java.lang.NoClassDefFoundError"

mansoorbahrian
mansoorbahrian New Altair Community Member
edited November 5 in Community Q&A
Dear group members,

I am developing a plug-in which uses linear algebra Fortran packages (like LAPACK, blas etc.) (from f2j). The problem is blas.jar (namespace: org.netlib.blas) is also referenced in RapidMiner's main project. The problem is the same namespace also exist in my plug-in and I am using the class "org.netlib.blass.DGEMM" and this class internally calls "org.netlib.util.MatConv", both of the classes (DGEMM, MatConv) exist in my plug-in.jar.

Most likely the problem is because of class loader, because the CurrentThread is referring to "org.netlib.blass.DGEMM" in blas.jar included in the main RapidMiner project and not the "org.netlib.blass.DGEMM" in my plug-in. I even try to set the CurrentThread to the current class loader e.g.  processor.class in my plug-in but it did not work out.

Please have a look to the exception output, looking forward to get some feedback.

Kind Regards
Mansoor



2011-02-09 19:05:51 SEVERE: Process failed: org/netlib/util/MatConv
(ProcessThread.run())
  java.lang.NoClassDefFoundError: org/netlib/util/MatConv
      org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:281)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:99)
      de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
      de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
      com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
      com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
      com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
      com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.Process.run(Process.java:863)
      com.rapidminer.Process.run(Process.java:770)
      com.rapidminer.Process.run(Process.java:765)
      com.rapidminer.Process.run(Process.java:755)
      com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
Caused by:
  java.lang.ClassNotFoundException: org.netlib.util.MatConv
      java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      java.security.AccessController.doPrivileged(Native Method)
      java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:281)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:99)
      de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
      de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
      com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
      com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
      com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
      com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.Process.run(Process.java:863)
      com.rapidminer.Process.run(Process.java:770)
      com.rapidminer.Process.run(Process.java:765)
      com.rapidminer.Process.run(Process.java:755)
      com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
2011-02-09 19:05:51 SEVERE: Here:          Process[1] (Process)
          subprocess 'Main Process'
            +- Read RDF[0] (Read RDF)
            +- Set Role[0] (Set Role)
            +- Filter Examples[0] (Filter Examples)
            +- Retrieve[1] (Retrieve)
            +- Validation[1] (X-Validation)
          subprocess 'Training'
      ==>      |  +- Train Correspondence Analysis[1] (Train
Correspondence Analysis)
                |  +- Multiply[0] (Multiply)
                |  +- Distance Matrix[0] (Distance Matrix)
                |  +- Naive Bayes[0] (Naive Bayes)
          subprocess 'Testing'
                  +- Test Correspondence Analysis[0] (Test Correspondence
Analysis)
                  +- Apply Model[0] (Apply Model)
                  +- Performance[0] (Performance) (ProcessThread.run())
2011-02-09 19:05:51 SEVERE: org/netlib/util/MatConv
(SwingTools.showFinalErrorMessage())
  java.lang.NoClassDefFoundError: org/netlib/util/MatConv
      org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:281)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:99)
      de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
      de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
      com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
      com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
      com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
      com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.Process.run(Process.java:863)
      com.rapidminer.Process.run(Process.java:770)
      com.rapidminer.Process.run(Process.java:765)
      com.rapidminer.Process.run(Process.java:755)
      com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
Caused by:
  java.lang.ClassNotFoundException: org.netlib.util.MatConv
      java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      java.security.AccessController.doPrivileged(Native Method)
      java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:281)
      de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:99)
      de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
      de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
      de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
      com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
      com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
      com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
      com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
      com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
      com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
      com.rapidminer.operator.Operator.execute(Operator.java:768)
      com.rapidminer.Process.run(Process.java:863)
      com.rapidminer.Process.run(Process.java:770)
      com.rapidminer.Process.run(Process.java:765)
      com.rapidminer.Process.run(Process.java:755)
      com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
Tagged:

Answers

  • fischer
    fischer New Altair Community Member
    Hi Mansoor,

    class loading issues are always tricky. Can't you drop the duplicate classes from your jar?

    Best,
    Simon
  • mansoorbahrian
    mansoorbahrian New Altair Community Member
    Thank you Simon for the reply.

    Actually earlier the "blas" package .class files was not added to my plug-in. But when I got the ClassNotFound exceptions then I thought might be its a good idea to have blas.jar within the plug-in so that it can find the org.netlib.util.MatConv class. Even after that the problem exist there.

    Also after your suggestion of removing duplicated files (i.e. org.netlib.blas package) from my plug-in, I once again remove the "blas" package from my plug-in, but I got the same exception.

    2011-02-14 18:12:35 SEVERE: Process failed: org/netlib/util/MatConv (ProcessThread.run())
      java.lang.NoClassDefFoundError: org/netlib/util/MatConv
          org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:313)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:101)
          de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
          de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
          com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
          com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
          com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
          com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.Process.run(Process.java:863)
          com.rapidminer.Process.run(Process.java:770)
          com.rapidminer.Process.run(Process.java:765)
          com.rapidminer.Process.run(Process.java:755)
          com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
    Caused by:
      java.lang.ClassNotFoundException: org.netlib.util.MatConv
          java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          java.security.AccessController.doPrivileged(Native Method)
          java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:313)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:101)
          de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
          de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
          com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
          com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
          com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
          com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.Process.run(Process.java:863)
          com.rapidminer.Process.run(Process.java:770)
          com.rapidminer.Process.run(Process.java:765)
          com.rapidminer.Process.run(Process.java:755)
          com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
    2011-02-14 18:12:35 SEVERE: Here:          Process[1] (Process)
              subprocess 'Main Process'
                +- Retrieve[1] (Retrieve)
                +- Validation[1] (X-Validation)
              subprocess 'Training'
          ==>      |  +- Train Correspondence Analysis[1] (Train Correspondence Analysis)
                    |  +- Multiply[0] (Multiply)
                    |  +- Distance Matrix[0] (Distance Matrix)
                    |  +- Naive Bayes[0] (Naive Bayes)
              subprocess 'Testing'
                      +- Test Correspondence Analysis[0] (Test Correspondence Analysis)
                      +- Apply Model[0] (Apply Model)
                      +- Performance[0] (Performance) (ProcessThread.run())
    2011-02-14 18:12:35 SEVERE: org/netlib/util/MatConv (SwingTools.showFinalErrorMessage())
      java.lang.NoClassDefFoundError: org/netlib/util/MatConv
          org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:313)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:101)
          de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
          de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
          com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
          com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
          com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
          com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.Process.run(Process.java:863)
          com.rapidminer.Process.run(Process.java:770)
          com.rapidminer.Process.run(Process.java:765)
          com.rapidminer.Process.run(Process.java:755)
          com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
    Caused by:
      java.lang.ClassNotFoundException: org.netlib.util.MatConv
          java.net.URLClassLoader$1.run(URLClassLoader.java:202)
          java.security.AccessController.doPrivileged(Native Method)
          java.net.URLClassLoader.findClass(URLClassLoader.java:190)
          java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          java.lang.ClassLoader.loadClass(ClassLoader.java:248)
          org.netlib.blas.DGEMM.DGEMM(DGEMM.java:170)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixMult(CorrespondenceAnalysis.java:313)
          de.dfki.km.semweb.CA.CorrespondenceAnalysis.matrixDecomposition_train(CorrespondenceAnalysis.java:101)
          de.dfki.km.semweb.CA.PreprocessingCA.FCAProcessing(PreprocessingCA.java:242)
          de.dfki.km.semweb.CA.PreprocessingCA.startFCAProcessing(PreprocessingCA.java:220)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.processDimReduction_train(CorrespondenceAnalysisProcessor.java:1107)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.addRowPrincipalMetrics_train(CorrespondenceAnalysisProcessor.java:679)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.doWork(CorrespondenceAnalysisProcessor.java:345)
          de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor_train.doWork(CorrespondenceAnalysisProcessor_train.java:21)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.validation.ValidationChain.executeLearner(ValidationChain.java:202)
          com.rapidminer.operator.validation.ValidationChain.learn(ValidationChain.java:288)
          com.rapidminer.operator.validation.XValidation.performIteration(XValidation.java:140)
          com.rapidminer.operator.validation.XValidation.estimatePerformance(XValidation.java:133)
          com.rapidminer.operator.validation.ValidationChain.doWork(ValidationChain.java:261)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
          com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
          com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:368)
          com.rapidminer.operator.Operator.execute(Operator.java:768)
          com.rapidminer.Process.run(Process.java:863)
          com.rapidminer.Process.run(Process.java:770)
          com.rapidminer.Process.run(Process.java:765)
          com.rapidminer.Process.run(Process.java:755)
          com.rapidminer.gui.ProcessThread.run(ProcessThread.java:65)
    2011-02-14 18:15:17 INFO: Running shutdown sequence. (RapidMinerGUI$ShutdownHook.run())
    2011-02-14 18:15:17 CONFIG: Saving perspectives. (ApplicationPerspectives.saveAll())
    2011-02-14 18:15:18 CONFIG: Saving operator usage. (UsageStatistics.save())
  • fischer
    fischer New Altair Community Member
    Hi,

    this looks weird. For debugging purposes, could you dump the classpath and make sure all jars are on the system class path?

    Best,
    Simon
  • steffen
    steffen New Altair Community Member
    Hello together

    Here is small code snippet (parts sent my mansoor, who ignored my request to include it into this thread) which does not even run as part of an eclipse project, because the file f2util.jar is missing (which contains the MatConv-class). 

    package de.mathmagic;

    import java.util.Arrays;

    import org.netlib.blas.DGEMM;

    public class MathMagic {

    	private double[][] matrixMult(double[][] a, double[][] b) {

    		double[][] c = new double[a.length][b.length];

    		int m = a.length;
    		int n = b[0].length;
    		int k = a[0].length;
    		double alpha = 1.0;
    		double beta = 1.0;

    		DGEMM.DGEMM("N", "N", m, n, k, alpha, a, b, beta, c);

    		return c;
    	}

    	public static void main(String[] args) {
    		MathMagic mm = new MathMagic();

    		double[][] matrix = new double[2][2];
    		matrix[0] = new double[]{1,2};
    		matrix[1] = new double[]{3,4};

    		System.out.println("input:\n" + Arrays.toString(matrix[0]) + "\n" + Arrays.toString(matrix[1]));

    		double[][] res = mm.matrixMult(matrix,matrix);
    		System.out.println("output:\n" + Arrays.toString(res[0]) + "\n" + Arrays.toString(res[1]));
    	}
    }
    I admit I do not know how this f2 - thingi works exactly, but from my point of view it seems that rapidminer is missing the mentioned f2utils.jar (which is part of the jar-pack-0.1 available at f2j - sourceforge-page).

    @Mansoor: I think it is better to add the mentioned jar to rapidminer/lib and buildpath and rebuild the whole thing. The problem should go away then. Although it seems like an interesting riddle to make the classloaders "talk to each other", I prefer to keep this pain joy for another day.

    regards,

    steffen
  • mansoorbahrian
    mansoorbahrian New Altair Community Member
    Hello,

    Apology for Steffen: I am sorry for not posting the code snippets as you mentioned. This has nothing to do with any kind of secrecy, it's only because I was severely stuck with some other issues and did not find time to reply.

    I do agree with steffen that if we add f2utils.jar to rapidminer/lib then the problem will be resolved.

    As per my discussion with Mr. Simon Fisher, we agreed to publish this plug-in but my question is if we opt for the above choice then what about Thousands of RapidMiner 5.0 installations world wide. Will they be able to use my plug-in or should I have to wait for the new release of RapidMiner.

    Here is the code and relevant output regarding the classloader issue: 

    private double[][] matrixMult(double[][] a, double[][] b){

    		double[][]c = new double [a.length][b.length];
            
    		int m = a.length;
    		int n = b[0].length;
    		int k= a[0].length;
    		double alpha = 1.0;
    		double beta = 1.0;


    		System.out.println("First class loader call");
    		System.out.println("Thread.currentThread().getContextClassLoader() \n" + Thread.currentThread().getContextClassLoader());		
    		// output : 
    		// sun.misc.Launcher$AppClassLoader@93dee9
    		ClassLoader currThLoader = Thread.currentThread().getContextClassLoader();

    		System.out.println("Sencond class loader call");
    		System.out.println("de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.class.getClassLoader() \n " +de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.class.getClassLoader());
    		// output :
    		// PluginClassLoader ([file:C:\eclipse-galileo\workspace\ThesisWorkspace6\rapidminer_vega\lib\plugins\rapidminer-SemWeb-Extension-1.0.000.jar])

    		Thread.currentThread().setContextClassLoader(de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.class.getClassLoader());

    		System.out.println("Third class loader call");
    		System.out.println("Thread.currentThread().getContextClassLoader() \n" + Thread.currentThread().getContextClassLoader());
                    // output :
                   // PluginClassLoader([file:C:\eclipse-galileo\workspace\ThesisWorkspace6\rapidminer_vega\lib\plugins\rapidminer-SemWeb-Extension-1.0.000.jar])


    		// Even I try to set the class loader explicitly
    		try {
    			ClassLoader classLoader = de.dfki.km.semweb.Operator.CA.CorrespondenceAnalysisProcessor.class.getClassLoader();
    			System.out.println("classloader " + classLoader);

    			classLoader.loadClass("org.netlib.blas.DGEMM");
    			Thread.currentThread().setContextClassLoader( classLoader );
    		} catch (ClassNotFoundException e) {
    			e.printStackTrace();
    		}

                  // output:
                 // Even after trying to set the class loader explicitly, I got the same exception which I have already posted.

    		DGEMM.DGEMM("N", "N", m, n, k, alpha, a, b, beta, c);

    		return c;
    	}


    Looking forward to have some conclusive discussion/ solution.

    Kind Regards
    Mansoor



  • steffen
    steffen New Altair Community Member
    Hello again

    This is not a complete solution, rather a collection of thoughts and gathered information. Maybe I will elaborate the details later this week in form of code, but I know myself, so I won't promise anything. Maybe someone else can work it out.

    What is the problem ?
    The problem (as far as I understand all the classloader magic) is the following:
    The classloader of the plugin is a child of the "rapidminer-core"-classloader. What does this imply ?
    - DGEMM is loaded first, the associated classloader is the core-classloader
    - If the plugin tries to load DGEMM, too, the parent classloader (from the core) tells that this class already has been loaded, so nothing is done here.
    - The plugin loads the code from f2utils (MathConv.class etc.)

    Now, when DGEMM is called for execution, the associated core-classloader is used to find MathConv. But this class is only available in the child(plugin)-classloader, which is not visible to the parent(core)-classloader => ClassNotFoundException.

    What can we do ?
    1. Overwrite DGEMM in the core - classloader
    Not possible, as already told

    2. Set the parent - classloader of the plugin to the Bootstrap-Classloader
    Not really an option. Not only the core-code has to be changed, additionally all required rapidminer classes have to be loaded again.

    3. Use Thread.setContextClassLoader() to "tunnel" the classloader-hierarchy
    Yes, this is an option, although it is more  complicated than the attempt in mansoors code. In detail the following steps have to be done (constructed in mind, not yet tested via code):
    - create a new thread where you want to execute blas-code. The code which uses blas has to be independent of rapidminer-core-classes.
    - set (after creation) the classloader to a self implemented subclass of classloader, which has the bootstrap - classloader as parent
    - This Self-Implemented Classloader loads the required blas- and f2utils-classes plus the the classes which use them "manually" by overriding the findClass-Method. By using the plugin-Classloader one should be capable of retrieving the *.class-files, load the byte-code from the files and then define the classes on its own.
    - Now, by executing this thread (with the code which uses blas as runnable inside), the thread loads everything via this classloader and hence uses the correct class-instance of DGEMM etc.

    What was wrong with Mansoors code ?
    - One cannot change of the context-classloader of a running/alive thread (as far as I know)
    - The set plugin-classloader is still a child of the core-classloader, hence the "wrong" instance of DGEMM is used.

    I recommend the following resources to read ...
    - Internals of Java Classloading
    - When should I use Thread.getContextClassLoader()?

    I am looking forward to your replies ... where you point out the awful mistakes in my trail of thoughts ;)

    kind regards,

    steffen



  • steffen
    steffen New Altair Community Member
    Hm ...

    I have written a proof of concept of my idea. The plugin (including source) can be downloaded from here.
    All in all it was more fun than pain to solve this, but nevertheless it was work.

    Note that the idea behind this plugin can also be used to "override" rm-dependencies in plugins to substitute older with newer versions of a library.

    regards,

    steffen

Categories