Validating and Grading Process File - Teaching Context

btibert

Admittedly this is not a a fully formed idea just yet, but I use RM for one of my graduate courses in a B-school, and based on the design, it's intended to really be a class that forces the students to work through analytics projects (data cleaning, EDA, modeling, etc.).  I say that because the students will have to work through problems in RapidMiner.  

My question/thought/idea:  Has anyone explored a path to have students submit their processes (.rmp file) and have a utility review the submission?  The parallel here would be nbgrader in python, where you can grade python notebooks, or even higher level, simply writing tests against input/work.  This is seen in various tools like Gradescope, Mimir, or even Github Classroom.

My need:  I would love to be able to have students submit their work, and this "autograder" would review the XML of the process to look for checks, proper settings, etc.  Ideally it could be executed to see if there are errors and such, but I am trying to keep this idea high level for now.  Basically, instead of reviewing a process 1x1 for every student, having a pathway to programmatically review submitted .rmp files.

In the past, I have either manually reviewed these files, or completely stayed away from using these submission as assignments, but I am curious as to what you all think.

MartinLiebig

HI @btibert ,

I think there is not much around and especially not for the way we would use it here. As mentioned this is a creative missuse of tools.

Usually you create processes to test, that either you process or your operator does what its supposed to do. Each operator in the operator toolbox has multiple test cases we define [and the core operators are tested way more religiously].

This is for example a test we run for the read office operator to read a word file:

The word file we read is put as a file object next to it. After defining the process, you can run it and define the output as the "solution" by using Testing -> Run and Store As Expected result.

After doing so you can either run individual processes or folders and test if they are still delivering the results which is expected by right clicking on a process or folder and then say "Run as Test". One can also use Execute Process Test operator in a process to do this.

I am not sure exactly what you want, but I think this functionality may be a corner stone.

Cheers,

Martin

MartinLiebig

Hi,

i think the key idea is to use the process testing extension. This is usually designed to verify that the process is doing what its supposed to do. I assume one can use it maybe in conjunction with loop repo to do this?

Best,

Martin

btibert

Interesting.  Can you point me to some documentation where I can read more about the process testing extension?  I am unaware of this tool.  Thanks so much!

MartinLiebig

HI @btibert ,

I think there is not much around and especially not for the way we would use it here. As mentioned this is a creative missuse of tools.

Usually you create processes to test, that either you process or your operator does what its supposed to do. Each operator in the operator toolbox has multiple test cases we define [and the core operators are tested way more religiously].

This is for example a test we run for the read office operator to read a word file:

The word file we read is put as a file object next to it. After defining the process, you can run it and define the output as the "solution" by using Testing -> Run and Store As Expected result.

After doing so you can either run individual processes or folders and test if they are still delivering the results which is expected by right clicking on a process or folder and then say "Run as Test". One can also use Execute Process Test operator in a process to do this.

I am not sure exactly what you want, but I think this functionality may be a corner stone.

Cheers,

Martin

btibert

Thanks for your comment.  I will poke around.  As noted above, the idea is not fully formed yet, but I am trying to think of equivalent ways to automate the review and grading of assignments that are based out of this tool.  As noted earlier, I wasn't aware of the Execute Process operator, and perhaps there are things to do by looping over the submissions.  Regardless, I truthfully was expecting that there wouldn't be any possible options, so this is great.