🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Process Pushdown error

User: "asmahassani08"
asmahassani08
New Altair Community Member
Updated by Jocelyn
I'm working on radoop. I'm using the sparkRM to extract association rules. for that, I apply the fpgrowth in the sparkRM operator subprocess.
I have these error : SparkRM: The driver resource request has not yet been granted to the Spark job.
joined the process and the error.

Find more posts tagged with

AI Studio
AI Studio Radoop
Errors
SparkRM
Sort by:
1 - 4 of 41
    User: "asmahassani08"
    asmahassani08
    New Altair Community Member
    OP
    I changed the pushdown process to sparkRM operator and have problems: 

    what does it mean: The driver resource request has not yet been granted to the Spark job.
    In rapidminer's logs, I have this  SparkRM (2): Yarn application state of application_1544000400659_0013: ACCEPTED 
    in the yarn :
    ContainersMonitorImpl: Memory usage of ProcessTree container_1544000400659_0013_01_000001 for container-id container_1544000400659_0013_01_000001: 278.2 MB of 2 GB physical memory used; 455.0 MB of 4.2 GB virtual memory used
    the job is accepted and not yet running, what can be the source of the problem?

    User: "sgenzer"
    sgenzer
    Altair Employee
    cc @asimon
    User: "toth_zsolt_bme"
    toth_zsolt_bme
    New Altair Community Member
    Hi,

    this can mean multiple things, my guess is that your cluster doesn't have enough free resources to start the job. You can try to play a bit with the Spark Resource Allocation Policy settings in the Radoop Connection, and the cluster resources, driver memory, executor memory settings in the operator params, e.g. try to decrease the cluster resources%.
    If you have access to the yarn CLI, you can do "yarn logs -applicationId application_1544000400659_0013" to see the full log of the application.


    User: "asmahassani08"
    asmahassani08
    New Altair Community Member
    OP
    Updated by asmahassani08
    thank you for your response, the problem is that I'm new with Hadoop and spark and I don't know what and how should I configure things.
    follow yarn-site.xml.
    this is spark-default.conf : 
    <div class="Quote"># spark.master&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;yarn<br># spark.eventLog.enabled&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;true<br># spark.eventLog.dir&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;hdfs://namenode:8021/directory<br># spark.serializer&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;org.apache.spark.serializer.KryoSerializer<br># spark.driver.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4G<br># spark.driver.cores					1<br># spark.yarn.driver.memoryOverhead	384MB<br># spark.yarn.am.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1G<br># spark.yarn.am.cores				1<br># spark.yarn.am.memoryOverhead		384MB<br># spark.yarn.nodemanager.resource.cpu-vcores&nbsp; &nbsp;1<br># spark.yarn.nodemanager.resource.memory-mb 2048MB<br># spark.executor.memory&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2G<br># spark.executor.instances			1<br># spark.executor.cores				1<br># spark.yarn.executor.memoryOverhead	384MB<br># spark.executor.extraJavaOptions&nbsp; -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"</div>
    Spark Resource Allocation policy = static, default configuration 
    driver memory = 1024MB
    in the diagnostic of the application i have as follow: 
    hope that I have a solution and get the things working. 
    yarn-site.xml
Quick Links