Rapidminer to HPC

icsdm16069
icsdm16069 New Altair Community Member
edited November 5 in Community Q&A
Hello,

i would like to ask if i could run the rapidminer (using linux distribution) in an HPC with slurm. I am really confused because i am trying to run it and have memory problems. My rapidminer process is working well and is inside a loop. Is anyone that have the same problem?
Tagged:

Answers

  • MartinLiebig
    MartinLiebig
    Altair Employee
    how much ram did you give it?
    BR,
    Martin
  • icsdm16069
    icsdm16069 New Altair Community Member
    my slurm program is the following:

    #!/bin/bash -l

####################################
#      slurm script template   #
#                                  #
# Submit script: sbatch filename   #
#                                  #
####################################

#SBATCH --job-name=rapidminer    # Job name
#SBATCH --output=rapidminer.%j.out # Stdout (%j expands to jobId)
#SBATCH --error=rapidminer.%j.err # Stderr (%j expands to jobId)
#SBATCH --ntasks=3     # Number of tasks(processes)
#SBATCH --nodes=1     # Number of nodes requested
#SBATCH --ntasks-per-node=3     # Tasks per node
#SBATCH --cpus-per-task=1     # Threads per task
#SBATCH --time=10:10:30   # walltime
#SBATCH --mem=64G   # memory per NODE
#SBATCH --partition=task    # Partition
#SBATCH --account=pa    # Replace with your system project

if [ x$SLURM_CPUS_PER_TASK == x ]; then
  export OMP_NUM_THREADS=1
else
  export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
fi


## LOAD MODULES ##
module purge		# clean up loaded modules 

# load necessary modules
module load gnu/4.9.2
module load intel/15.0.3
module load intelmpi/5.0.3
module load cuda/8.0.61
module load java/1.8.0
module load python/3.5.0
## RUN YOUR PROGRAM ##
srun ./rapidminer-studio/scripts/rapidminer-batch.sh -f /rapidminercodes/xml.rmp

  • icsdm16069
    icsdm16069 New Altair Community Member
    ok i fixed it. The first thing that you need to modify is the max memory that can be used from rapidminer and the second is the making an mpi program in order to split your input data and run the rapidminer in parallel.

Categories