Run Rapidminer Server 8+ process from command line

cesar_ortiz

I have up and running a Rapidminer Server 8.1. Created a process that connects to AWS EMR through Radoop. Everything works fine, in Studio and in the Server. I've already succesfully run the Studio process through the command line with "rapidminer-batch.sh" but I want to know if it is possible to run a process stored in the server repository through the command line, in this case Ubuntu.

 

Any ideas?

Thanks in advance

Edin_Klapic

Hi @cesar_ortiz,

 

I think there is a misunderstanding here. In the REST description on swagger hub (https://app.swaggerhub.com/apis/RapidMiner/JobService/0.1.0#/jobs/createUsingPOST) you can click on Model (see screenshot).

There you can see that the string for the parameter process should be the "base64 encoded process XML".

 

You can i.e. use an online available base64 encoder for that.

 

Example process just generating data:

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.1.003" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data" compatibility="8.1.003" expanded="true" height="68" name="Generate Data" width="90" x="112" y="34"/>
      <connect from_op="Generate Data" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Base64 encoded:

PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAgIDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg0K

Happy Mining,

Edin

 

 

 

 

MartinLiebig

Hi,

 

there is no CLI for the server repo. But you can use the JobService API to create a process execution: https://app.swaggerhub.com/apis/RapidMiner/JobService/0.1.0#/jobs/createUsingPOST

 

Best,

Martin

 

cesar_ortiz

Thank you for your replay,

 

I followed your adviced and checked out the REST API. I can create a JWT as the documentation suggests with:

curl -u username:password http://localhost:8080/internal/jaxrest/tokenservice

but when i wanna summit a job I always get code 401. (being aa.bb.cc the 100+-character long idToken i've got from the first step)

curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json" -H "Authorization: aa.bb.cc" -H "Content-Type: application/json" -d "{ \"queueName\": \"default\", \"process\": \"test\", \"location\": \"/test\"}"

{"timestamp":1522187911798,"status":401,"error":"Unauthorized","message":"Unauthorized","path":"/executions/jobs"}

 

Is the location relative to the location in the server repository?

Do I need to make any changes in the server?

Edin_Klapic

Hi @cesar_ortiz,

 

Some things to check:

1. Is the the JWT token still valid?
2. Has the user for which you created the JWT token permissions to the Process location & the Job Agent?

Happy Mining,

Edin

cesar_ortiz

Well, I try to connect immediately after I create the token, unless those are valid only for a few seconds.

The user for which I created the token is the admin, which have access to everything

 

Here is a caption of the permissions of my process:

 

Thank you for your response 

 

 

Marco_Boeck

Hi,

 

the "Authorization" header needs to be prefixed with "bearer ". After that, append the entire JWT token. For example, for my local test server it currenty looks like that:

Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJyYXBpZG1pbmVyLmNvbSIsInN1YiI6ImFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pc3RyYXRvciIsImFuYWx5c3QiLCJleGVjdXRlIiwicmVwb3J0IGVkaXRvciIsInJlcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyMzM0ODk5OCwiaWF0IjoxNTIzMzQ4Njk4fQ.dxqOqsvgwRXIEkscYb_k9q-mrcfkpln6tgTtQXeDtek

Regards,

Marco

cesar_ortiz

Thanks Marco,

 

Now with "bearer" as a prefix the processes are sent to the server and I get this as response:

 

{"id":"824ea00b-aaa1-4d12-a521-8e8e4a66","state":"PENDING","location":"/test","ownerId":"admin","createdAt":1525735389338,"queueName":"DEFAULT"}

 

But now I get a new error:

 

com.rapidminer.tools.XMLException: Cannot parse document: Content is not allowed in prolog.

Apparently is an encoding issue with the xml. Tried changing my code to this

 

curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json;charset=utf-8" -H "Authorization: bearer eyJhbGciOiJIUzI16IkpXVCJ9.eyJpc3MiOiJyYXFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pc3RyYXRvciIsImFuYWx5c3QiLcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyNTczMTk0OCwiaWF0IjoxNTI1NzMxNjQ4fQ.43g_939QDu-TA-CdbxmQHV2Ow_63fw" -H "Content-Type: application/json;charset=utf-8" -d "{ \"queueName\": \"default\", \"process\": \"test\", \"location\": \"/test\"}"

And other combinations with iso-8859-1 and utf-16 with the same luck. Also tried to change the encoding in the Rapidminer Studio.

 

Any ideas?

Nils_Woehler

Hi @cesar_ortiz,

 

the REST API expects the process XML to be provided within the "process" parameter. Currently, you only provide "test" which is not a valid process XML

The location is only needed to resolve relative paths (in case your process uses those). If you do not interact with the repository at all you can omit the location.

 

Best,

Nils

cesar_ortiz

But my process is named "test" and is stored in the root directory of the server repository

 

 

By the way, this is my process. It's really simple, just a proof of concept

 

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.1.003" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="radoop:radoop_nest" compatibility="8.1.001" expanded="true" height="68" name="Radoop Nest" width="90" x="45" y="34">
        <parameter key="connection" value="Radoop"/>
        <parameter key="sample_size" value="200000"/>
        <enumeration key="tables_to_reload"/>
        <process expanded="true">
          <operator activated="true" class="radoop:retrieve" compatibility="8.1.001" expanded="true" height="68" name="Retrieve" width="90" x="45" y="34">
            <parameter key="use_default_database" value="false"/>
            <parameter key="database" value="bigdata"/>
            <parameter key="table" value="table1"/>
            <parameter key="filter_clause" value="WHERE 1 = 1;"/>
          </operator>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
        </process>
      </operator>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
    </process>
  </operator>
</process>

Edin_Klapic

Hi @cesar_ortiz,

 

I think there is a misunderstanding here. In the REST description on swagger hub (https://app.swaggerhub.com/apis/RapidMiner/JobService/0.1.0#/jobs/createUsingPOST) you can click on Model (see screenshot).

There you can see that the string for the parameter process should be the "base64 encoded process XML".

 

You can i.e. use an online available base64 encoder for that.

 

Example process just generating data:

<?xml version="1.0" encoding="UTF-8"?><process version="8.1.003">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.1.003" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data" compatibility="8.1.003" expanded="true" height="68" name="Generate Data" width="90" x="112" y="34"/>
      <connect from_op="Generate Data" from_port="output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Base64 encoded:

PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAgIDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg0K

Happy Mining,

Edin

 

 

 

 

cesar_ortiz

It's alive!!!

 

Thank you all for your help :smileyhappy:

MartinLiebig

Hi @cesar_ortiz,

this is great news! 

 

Would you mind to write a few lines on how you did it? I would love to have this documented somewhere. There is so much knowledge to be shared in the community. We would, of course, help you wherever needed and push this via our social media channels.

 

Best,

Martin

cesar_ortiz

Well, to launch a server process from command line first you need to create a JWT with:

 

curl -u username:password http://localhost:8080/internal/jaxrest/tokenservice

Then, you encode the process xml file to base64 with some online tool or other solution, and you'll get something like this:

 

 

JK94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAhawDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCigICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg

And the line to launch the process will look like this:

curl -X POST "http://localhost:8080/executions/jobs" -H "accept: application/json;charset=utf-8" -H "Authorization: bearer eyJhbGciOiJzI16IkpXVCJ9.eyJpcMiOiJyYXFkbWluIiwiZ3JwIjpbImFkbWluIiwiYWRtaW5pcYXRvciIsImFuYWx5c3QiLcG9ydCBtYW5hZ2VyIiwicmVwb3J0IHZpZXdlciIsInNjaGVkdWxlciIsInNlcnZpY2UiLCJ1c2VycyJdLCJhZG1pbiI6dHJ1ZSwiYXVkIjoicmFwaWRtaW5lciIsImV4cCI6MTUyNTczMTk0OCwiaWF0IjoxNTI1NzMQ4fQ.43g_939QDu-TA-CdbxmQHV2Ow_63fw" -H "Content-Type: application/json;charset=utf-8" -d "{ \"queueName\": \"default\", \"process\": \"JK94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48cHJvY2VzcyB2ZXJzaW9uPSI4LjEuMDAzIj4NCiAgPGNvbnRleHQ+DQogICAgPGlucHV0Lz4NCiAgICA8b3V0cHV0Lz4NCiAgICA8bWFjcm9zLz4NCiAgPC9jb250ZXh0Pg0KICA8b3BlcmF0b3IgYWN0aXZhdGVkPSJ0cnVlIiBjbGFzcz0icHJvY2VzcyIgY29tcGF0aWJpbGl0eT0iOC4xLjAwMyIgZXhwYW5kZWQ9InRydWUiIG5hbWU9IlByb2Nlc3MiPg0KICAhawDxwcm9jZXNzIGV4cGFuZGVkPSJ0cnVlIj4NCiAgICAgIDxvcGVyYXRvciBhY3RpdmF0ZWQ9InRydWUiIGNsYXNzPSJnZW5lcmF0ZV9kYXRhIiBjb21wYXRpYmlsaXR5PSI4LjEuMDAzIiBleHBhbmRlZD0idHJ1ZSIgaGVpZ2h0PSI2OCIgbmFtZT0iR2VuZXJhdGUgRGF0YSIgd2lkdGg9IjkwIiB4PSIxMTIiIHk9IjM0Ii8+DQogICAgICA8Y29ubmVjdCBmcm9tX29wPSJHZW5lcmF0ZSBEYXRhIiBmcm9tX3BvcnQ9Im91dHB1dCIgdG9fcG9ydD0icmVzdWx0IDEiLz4NCigICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzb3VyY2VfaW5wdXQgMSIgc3BhY2luZz0iMCIvPg0KICAgICAgPHBvcnRTcGFjaW5nIHBvcnQ9InNpbmtfcmVzdWx0IDEiIHNwYWNpbmc9IjAiLz4NCiAgICAgIDxwb3J0U3BhY2luZyBwb3J0PSJzaW5rX3Jlc3VsdCAyIiBzcGFjaW5nPSIwIi8+DQogICAgPC9wcm9jZXNzPg0KICA8L29wZXJhdG9yPg0KPC9wcm9jZXNzPg\", \"location\": \"/test\"}"

 Authorization: bearer is the JWT and \"process\" is the base64 xml