Trouble starting RMServer after upgrading to 8.10

adamf
adamf New Altair Community Member
edited November 5 in Community Q&A

I "upgraded" from RM Server 8.00 (free edition) to 8.10.  The server service starts, and then quickly quits.  I tried a full installation next, deleting the previous server directory, files, and service.  Same result.  The server service starts, the CPU utilization is pegged at 100% for several minutes, and then the service quits.  Next I tried creating a new schema and another full installation.  Same result.  I killed the service after several minutes of 100% CPU utilization, and that appeared to fail as well.  Service shows as Stopped in Windows Service list, but CPU still pegged at 100%.  I had to restart to clear the condition.  Any help would be greatly appreciated!!

Tagged:

Answers

  • adamf
    adamf New Altair Community Member

    Attached is the Log file if that is helpful.

  • sgenzer
    sgenzer
    Altair Employee

    tagging @Edin_Klapic

     

     

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi @adamf,

     

    Thanks for sending over the logfile. Interestingly enough, it seems not to be the full file.

    At the bottom it states ... starting deployment of "rapidminer-server-8.1.0.ear"... - so it seems your Server was just about to start.

     

    Can you please try the following:

    1. Make sure RapidMiner Server is not running
    2. Start Rapidminer Server using the standalone.bat file located in <Server installation folder>\bin\

    Some console windows should be popping up - they are basically displaying the content of the server.log file. 1 for the Server and 1 for the default JobAgent - if you installed one.

    The startup is finished when you see something along the lines of

     

    11:55:20,731 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://10.20.10.27:9990/management 
    11:55:20,731 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://10.20.10.27:9990 
    11:55:20,733 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss EAP 6.4.0.Alpha1 (AS 7.5.0.Final-redhat-15) started (with errors) in 97189ms -

     

    If you cannot access the Server via the WebInterface or the startup breaks beforehand, please post the new server.log file then.

     

    Happy Mining,

    Edin

  • adamf
    adamf New Altair Community Member

    Hi Edin,

     

    I will retry starting the server.  However, I am not convinced the server was just about to start.  The server service seems to stop/abort itself, as when I check the list of Windows Services, it has stopped after several minutes.  I think it has more likely crashed and not cleaned up as the CPU performance remains pegged at 100% until I restart Windows and disable the RMS810SVC service.

     

    I'm unclear why you suggested starting the server using the BAT file vs. auto-starting the RMS810SVC service.  But, I shall try your approach.

     

    Best Regards,

     

    Adam

     

  • adamf
    adamf New Altair Community Member

    Hello again,

     

    I have run the standalone.bat file to attempt to start the RMserver process.  It appears that the Job Agent is unable to make a connection, not sure to what or why it is failing.  Attached are the log file and a screen shot of the Job Agent console window with failure messages.

     

    Thanks for any suggestions/assistance.

     

    - Adam

  • adamf
    adamf New Altair Community Member

    Screen shot of the Job Agent console window showing connection error is attached . . .

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi @adamf,

     

    Thank you for the full log file. It definitely helps.

    Unfortunately, I could not identify how to solve your problem so I forwarded it to our Dev team.

     

    I'll keep you posted.

     

    Happy Mining,

    Edin

  • sgenzer
    sgenzer
    Altair Employee

    thanks @Edin_Klapic - let me know if this is a bug so I can push to Product Feedback + Jira.


    Scott

     

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi @adamf,

    Dev team just came back with the information that the log file indicates that the startup runs into a timeout. This might be because the hardware requirements are not met.
    If you are sure that this is not the case you might try to start the JobAgent after the Server has successfully started (which is done in parallel by default).

    Since you have already installed RapidMiner Server you can change the corresponding parameter named JOB_AGENT_ENABLED to false.
    This can be done at the end of the file standalone.conf (UNIX) or standalone.conf.bat (Windows) located in
    <RapidMiner-Server-installation-folder>\bin\
     
    After the successful Server startup you can start the JobAgent manually by executing the startup scripts rapidminer-jobagent (UNIX) or rapidminer-jobagent.bat (Windows) located in
    <RapidMiner-Server-installation-folder>\job-agent\bin\

    The JobAgent Stop commands are also located in this folder.

    Hope this helps.

    Happy Mining,
    Edin

  • adamf
    adamf New Altair Community Member

    Hello Edin,

     

    You mentioned that one possibility is that hardware requirements are not being met.  What specific requirements are relevant (CPU speed, RAM, other)?  Hardware requirements/performance have not been an issue with previous versions and installations of RMServer, including 8.01, on the same hardware platform.

     

    Also, I notice that at the end of the RMServer console window, the server process appears to fail/abort.  You are suggesting that it fails because of a timeout?

     

    I noticed also in the Job Agent console window, it is waiting/trying to establish a connection, but in reality there is no process listening on the port that it is trying to connect too.  Thus, it does not appear to be an issue with the job agent process.  Correct?

     

    Best Regards,

    Adam

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi @adamf,

     

    Yes, I startup fails because of a timeout. In your log file you can see the multiple timeout exception(s) similar to the following one

     

    09:33:24,532 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
        ("core-service" => "management"),
        ("management-interface" => "native-interface")
    ]'

    Timeout exceptions most likely happen when the underlying machine is too slow so the first guess would be that the system requirements (listed on https://docs.rapidminer.com/latest/server/installation/system-requirements.html) are not met. Since you didn't have the issue with v8.0.1. this is probably not the case in your environment.

    By disabling the automatic startup of the JobAgent we want to reduce the load on your machine, hopefully enabling RapidMiner Server to startup since there will be more resources available.

     

    Regarding the JobAgent:

    I agree, it is most likely not an issue with the JobAgent itself, especially because RapidMiner Server can totally run without any JobAgent. But the Agent communicates with RapidMiner Server on this port (5672) and just waits for a successful startup message from Server.

     

    I really hope we can solve the problem with this approach.

     

    Happy Mining,

    Edin

  • adamf
    adamf New Altair Community Member

    Hello again,

     

    I am increasingly confident that this issue is NOT related to the hardware/performance requirements.  My VM exceeds the HW requirements and is also unchanged from previous versions of RMServer that I have successfully installed and have had running for months.  The problems began with the introduction of RMServer v8.01 and v8.10. 

    I would appreciate any additional troubleshooting suggestions as I have been without the RMServer and a bunch of automated tasks that were run every day!

    Thank you, Adam.

     

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi @adamf,

     

    I totally agree that your Hardware meets the requirements if you have been able to run version 8.0.

    But as I mentioned in my last post, something extremely slows down the server at startup resulting in a 5 min timeout when it comes to port-binding. Maybe something else running on that port?

    Have you been able to test starting up the Server without the JobAgent? Which results did you get?

     

    Happy Mining,

    Edin

  • adamf
    adamf New Altair Community Member

    Perhaps something else (not HW performance) is causing the timeout.  For example, could any local firewall or Java restrictions be causing the port issues?

    I really need to resolve this soon or RMServer will be of no use.

    Thanks for any assistance!

    - Adam

     

  • adamf
    adamf New Altair Community Member

    I installed RMServer on a new Windows VM, and so far it seems to be working.  I also created a new MySQL schema (not sure if that was part of the problem or not).  But, I have two questions:

     

    1) Does RMServer use the database schema during the server startup (even before configuring the server from its console)?

     

    2) In RM Studio, the server's Repository does not show up in the list of Repositories (and I confirmed that the server is running).  I see only the Local Repository.  How do I add the Server Repository?

     

    Thanks,

     

    Adam

  • Pavithra_Rao
    Pavithra_Rao New Altair Community Member

    Hi @adamf,

     

    1) Does RMServer use the database schema during the server startup (even before configuring the server from its console)?

     

    Yes, RapidMiner server uses database schema during the server startup.

     

    2) Please follow the steps described in the section "Connecting RapidMiner Server to RapidMiner Studio" in the following link to setup access Server.

    https://docs.rapidminer.com/latest/server/installation/

     

     

    Hope this helps.

     

    Cheers,

     

     

     

     

Categories