problems with the Execute Program operator

kayman

Going nuts over something that seems pretty trivial.

 

I have a shell script that I want to run from within RM at the end of a workflow. the command I use is as simple as this : /mypath/myshell.sh

Running this shell script directly on my server (Ubuntu 16) works fine, but running it from RM server fails with error message : Process '/mypath/myshell.sh' exited with error code 255.

 

My shell script itself is as shown below, using sftp with a pem file

 

sftp -o ProxyCommand='nc -X connect -x myProxy:myPort %h %p' -i /mypath/mykey.pem myaccount@myDomain <<upload
put /myPath/myFile.csv
quit
upload

Running this using ssh / console works fine, so the syntax used is probably correct.

The only thing I can think about are access rights, my pem file needs to be set to owner read only (600) so if I login as admin and my pem file is assigned to my admin group it works fine, but how does rapidminer communicate with my server? Which group / user account is this using? I assume these are different, and I modified a few times the settings of the pem files (assigning them to root for instance) but no difference.

 

Or is there anything else I might be missing here ?

Or any other advice to get this sh running?

kayman

Managed to fix it by 'mixing' the user and group setting, so instead of root:root or user:user I tried user:root, which allowed me to run and add the domain to safe sites when using console. After that it worked fine in RM also.

 

The joy of Linux :-)

 

Thanks for the tips !

sgenzer

hi @kayman - yep my money is on access rights as well. Happens to me all the time. Try chmod 700 (I think 600 is R/W only, not execute). 

 

Scott

 

kayman

nope, the problem is that the API I use requires the pem file to be this restrictive. Anything above 600 /different as owner and it just says no. 
So I would need to know which account RM server is using, if that's the right terminology to use. 

 

I still have the option to create cron jobs for my shell scripts as an alternative, these do work fine. I would loose the option to start my upload ad hoc but it's not a big problem. 

Any idea if I can force a saved cron job to start ad hoc from within rapidminer?

MartinLiebig

Hey @kayman,

operator toolbox has a new operator for sftp connection. Maybe this solves your issue?

 

Best,

Martin

kayman

Hi Martin, I noticed this indeed (and some other nice updates to the toolbox btw)

 

Unfortunatly it does not seem to be able to work with .pem files. I was anyway able to get rid of some of my other sftp scripts that I handled with python by using the new operator, so usefull for sure. 

 

Now, I'm getting closer to the root cause, my whole ssh flow will probably need to be tailored to root (as this appears the default setting for RM) but I also need to be able to add my sftp site to known hosts while running as root, but since I cannot access my server as root but only as sudo etc etc...

 

So I'm probably stuck to use cron jobs instead. There are most likely workarounds but my Linux knowledge needs some upgrades also...

sgenzer

calling in reinforcements – tagging the RM Server "power team" @jpuente @hhomburg @JessForbesRM @Nils_Woehler

Scott

 

 

jpuente

Hi kayman,

RapidMiner Server runs as the user that was chosen during installation. It might be root but, for what you're saying it's probably not in your case. If you can, try logging in as that user and try the script in the command line. That will help to determine the issue.  As Scott and you said, it's probably about permissions. What error appears in Rapidminer?

 

Jesús

kayman

Managed to fix it by 'mixing' the user and group setting, so instead of root:root or user:user I tried user:root, which allowed me to run and add the domain to safe sites when using console. After that it worked fine in RM also.

 

The joy of Linux :-)

 

Thanks for the tips !