Process Pushdown error

asmahassani08

I'm working on radoop. I'm using the sparkRM to extract association rules. for that, I apply the fpgrowth in the sparkRM operator subprocess.
I have these error : SparkRM: The driver resource request has not yet been granted to the Spark job.

joined the process and the error.

asmahassani08

I changed the pushdown process to sparkRM operator and have problems: 

what does it mean: The driver resource request has not yet been granted to the Spark job.
In rapidminer's logs, I have this  SparkRM (2): Yarn application state of application_1544000400659_0013: ACCEPTED 

in the yarn :

ContainersMonitorImpl: Memory usage of ProcessTree container_1544000400659_0013_01_000001 for container-id container_1544000400659_0013_01_000001: 278.2 MB of 2 GB physical memory used; 455.0 MB of 4.2 GB virtual memory used

the job is accepted and not yet running, what can be the source of the problem?

sgenzer

cc @asimon

toth_zsolt_bme

Hi,

this can mean multiple things, my guess is that your cluster doesn't have enough free resources to start the job. You can try to play a bit with the Spark Resource Allocation Policy settings in the Radoop Connection, and the cluster resources, driver memory, executor memory settings in the operator params, e.g. try to decrease the cluster resources%.

If you have access to the yarn CLI, you can do "yarn logs -applicationId application_1544000400659_0013" to see the full log of the application.

asmahassani08

thank you for your response, the problem is that I'm new with Hadoop and spark and I don't know what and how should I configure things.
follow yarn-site.xml.
this is spark-default.conf : 

<div class="Quote"># spark.master&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;yarn<br># spark.eventLog.enabled&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;true<br># spark.eventLog.dir&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;hdfs://namenode:8021/directory<br># spark.serializer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;org.apache.spark.serializer.KryoSerializer<br># spark.driver.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4G<br># spark.driver.cores					1<br># spark.yarn.driver.memoryOverhead	384MB<br># spark.yarn.am.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1G<br># spark.yarn.am.cores				1<br># spark.yarn.am.memoryOverhead		384MB<br># spark.yarn.nodemanager.resource.cpu-vcores&nbsp; &nbsp;1<br># spark.yarn.nodemanager.resource.memory-mb 2048MB<br># spark.executor.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2G<br># spark.executor.instances			1<br># spark.executor.cores				1<br># spark.yarn.executor.memoryOverhead	384MB<br># spark.executor.extraJavaOptions&nbsp; -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</div>

Spark Resource Allocation policy = static, default configuration 
driver memory = 1024MB
in the diagnostic of the application i have as follow: 
hope that I have a solution and get the things working.