Problem with Radoop on Spark failed error
Dear All,
I am using Radoop and connected to a single node Hadoop cluster. I have also Hive running, and Spark is also on my machine. I run the Quick test of Radoop connection and is successful.
Next I have started to follow the Titanic data tutorial to understand more about Radoop and Rapidminer functions. When I run an example which is retrieving data from Hive and performing Radoop Nest validation, in which an SPARK job Decision Tree Binomial MLib is being used, I recieve an error as following, a full listinng of the rest of the errors I have also included, can you guide me why the SPARK job fails, have not installed SPARK correctly, is the problem with my process, or is this a connection error ?
********************************************************************************************8
Dec 12, 2016 11:16:04 PM FINE: Hive query: CREATE VIEW radoop__tmp_maziar_1481571964383_mf8n48v AS SELECT age,passenger_class,sex,no_of_siblings_or_spouses_on_board,no_of_parents_or_children_on_board,passenger_fare,survived FROM radoop__tmp_maziar_1481571909693_kee9ej8 WHERE rndp=0
Dec 12, 2016 11:16:05 PM FINE: Hive query: CREATE VIEW radoop__tmp_maziar_1481571965033_rtiirp4 AS SELECT age,passenger_class,sex,no_of_siblings_or_spouses_on_board,no_of_parents_or_children_on_board,passenger_fare,survived FROM radoop__tmp_maziar_1481571909693_kee9ej8 WHERE rndp=1
Dec 12, 2016 11:16:05 PM FINE: Hive query: DESCRIBE radoop__tmp_maziar_1481571964383_mf8n48v
Dec 12, 2016 11:16:06 PM FINE: Hive query: DESCRIBE radoop__tmp_maziar_1481571965033_rtiirp4
Dec 12, 2016 11:16:06 PM INFO: Decision Tree (MLlib binominal): Materializing input HadoopExampleSet as Parquet.
Dec 12, 2016 11:16:06 PM FINE: Hive query: DROP VIEW IF EXISTS radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:16:06 PM FINE: Hive query: DROP TABLE IF EXISTS radoop__tmp_maziar_1481571966498a_fwxtjmd PURGE
Dec 12, 2016 11:16:07 PM FINE: (Decision Tree (MLlib binominal)) Hive query: CREATE TABLE radoop__tmp_maziar_1481571966498a_fwxtjmd STORED AS PARQUET LOCATION '/tmp/radoop/maziar/tmp_1481571966498_tbf6xaw/tmp_1481571966919_higs6s4' AS SELECT * FROM radoop__tmp_maziar_1481571964383_mf8n48v
Dec 12, 2016 11:16:43 PM INFO: Decision Tree (MLlib binominal): Materialized.
Dec 12, 2016 11:16:43 PM FINE: Hive query: DESCRIBE FORMATTED radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:16:43 PM FINE: Hive query: DESCRIBE FORMATTED radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:16:43 PM FINE: Hive query: DESCRIBE FORMATTED radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:16:43 PM FINE: Hive query: DESCRIBE radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:16:44 PM INFO: Getting radoop-common.jar file from plugin jar...
Dec 12, 2016 11:16:44 PM INFO: Remote radoop-common-7.3.0.jar is up to date.
Dec 12, 2016 11:16:44 PM INFO: Getting radoop-spark20.jar file from plugin jar...
Dec 12, 2016 11:16:44 PM INFO: Remote radoop-spark20-7.3.0.jar is up to date.
Dec 12, 2016 11:16:46 PM FINE: Decision Tree (MLlib binominal): Setting lower driver memory (512 Mb) than requested (2048 Mb), because the request exceeded the cluster's resources.
Dec 12, 2016 11:16:47 PM INFO: Decision Tree (MLlib binominal): Spark application submitted.
Dec 12, 2016 11:16:52 PM FINE: Decision Tree (MLlib binominal): Yarn application state of application_1481571703578_0004: ACCEPTED
Dec 12, 2016 11:16:57 PM FINE: Decision Tree (MLlib binominal): Yarn application state of application_1481571703578_0004: ACCEPTED
Dec 12, 2016 11:17:02 PM FINE: Decision Tree (MLlib binominal): Yarn application state of application_1481571703578_0004: FAILED
Dec 12, 2016 11:17:02 PM INFO: Decision Tree (MLlib binominal): Spark application finished.
Dec 12, 2016 11:17:02 PM INFO: Decision Tree (MLlib binominal): Distributed final state: FAILED
Dec 12, 2016 11:17:02 PM WARNING: Radoop Nest: com.rapidminer.operator.UserError: The Spark job failed.
Dec 12, 2016 11:17:02 PM FINE: Hive query: DROP VIEW IF EXISTS radoop__tmp_maziar_1481571909693_kee9ej8
Dec 12, 2016 11:17:02 PM FINE: THREADS AT CLOSE:
Dec 12, 2016 11:17:02 PM FINE: Deleting temporary directory or file: /tmp/radoop/maziar/tmp_1481571966495_kmwaf9l/
Dec 12, 2016 11:17:02 PM FINE: Deleting temporary directory or file: /tmp/radoop/maziar/tmp_1481571966498_tbf6xaw/
Dec 12, 2016 11:17:02 PM SEVERE: Process failed: The Spark job failed.
Dec 12, 2016 11:17:02 PM SEVERE: Here:
Dec 12, 2016 11:17:02 PM SEVERE: Process[1] (Process)
Dec 12, 2016 11:17:02 PM SEVERE: subprocess 'Main Process'
Dec 12, 2016 11:17:02 PM SEVERE: +- Radoop Nest[1] (Radoop Nest)
Dec 12, 2016 11:17:02 PM SEVERE: subprocess 'Radoop Nest'
Dec 12, 2016 11:17:02 PM SEVERE: +- Retrieve[1] (Retrieve from Hive)
Dec 12, 2016 11:17:02 PM SEVERE: +- Validation[1] (Split Validation)
Dec 12, 2016 11:17:02 PM SEVERE: subprocess 'Training'
Dec 12, 2016 11:17:02 PM SEVERE: ==> | | +- Decision Tree (MLlib binominal)[1] (Decision Tree (MLlib binominal))
Dec 12, 2016 11:17:02 PM SEVERE: subprocess 'Testing'
Dec 12, 2016 11:17:02 PM SEVERE: | +- Apply Model[0] (Apply Model)
Dec 12, 2016 11:17:02 PM SEVERE: | +- Performance[0] (Performance (Classification))
Dec 12, 2016 11:17:02 PM SEVERE: +- Retrieve (2)[0] (Retrieve from Hive)
Dec 12, 2016 11:17:02 PM SEVERE: +- Apply Model (2)[0] (Apply Model)
Dec 12, 2016 11:17:02 PM FINE: Hive query: DROP TABLE IF EXISTS radoop__tmp_maziar_1481571909693_kee9ej8 PURGE
Dec 12, 2016 11:17:03 PM FINE: Hive query: DROP VIEW IF EXISTS radoop__tmp_maziar_1481571964383_mf8n48v
Dec 12, 2016 11:17:04 PM FINE: Hive query: DROP TABLE IF EXISTS radoop__tmp_maziar_1481571964383_mf8n48v PURGE
Dec 12, 2016 11:17:04 PM FINE: Hive query: DROP VIEW IF EXISTS radoop__tmp_maziar_1481571965033_rtiirp4
Dec 12, 2016 11:17:04 PM FINE: Hive query: DROP TABLE IF EXISTS radoop__tmp_maziar_1481571965033_rtiirp4 PURGE
Dec 12, 2016 11:17:04 PM FINE: Hive query: DROP VIEW IF EXISTS radoop__tmp_maziar_1481571966498a_fwxtjmd
Dec 12, 2016 11:17:05 PM FINE: Hive query: DROP TABLE IF EXISTS radoop__tmp_maziar_1481571966498a_fwxtjmd PURGE
Dec 12, 2016 11:18:58 PM FINE: Mnemonic key q not found for action hive.write_query (Execute Query...).
Dec 12, 2016 11:18:59 PM FINE: Hive query: SHOW TABLES