Why is my fetch taking so long?

DU_ca
DU_ca New Altair Community Member
edited November 5 in Community Q&A
I am able to connect to my Oracle database system, but the "Fetching Database Tables" process takes 45 minutes to complete.  Why is this happening?
Tagged:

Answers

  • rfuentealba
    rfuentealba New Altair Community Member
    Hello @DU_ca:

    It appears to be something not fully compliant in the JDBC driver for Oracle that requires an extra step. Let's call @mschmitz and ask him if he knows someone who can help us, because I tried to clone the RapidMiner Studio source code and found out that the open source version of it doesn't include the JDBC connectors, so I just used Navicat and copy/pasted :-(

    All the best,

    Rodrigo.
  • DU_ca
    DU_ca New Altair Community Member
    Thanks for trying, Rodrigo.  Unfortunately, I'm still having this problem and so I guess RapidMiner is not a good option for me.
  • jczogalla
    jczogalla New Altair Community Member
    Hi @DU_ca!
    Can you maybe share your process and/or explain what exactly you want to do?
    If you are trying to load all your data from your database, this might be quite some effort to do if you have a huge database. RapidMiner has to convert each table into its own memory format, which might take a lot of RAM and time.
    If you want to manipulate a lot of your database data, especially preprocessing, it maybe would help you to use the in-DB extension from the marketplace, as discussesd here: https://community.rapidminer.com/discussion/55129/challenge-with-rm-server-running-out-of-memory#latest
    I hope this helps!
    Jan

  • sgenzer
    sgenzer
    Altair Employee
    using the caching tools in the Jackhammer extension can also be helpful here...

    https://oldworldcomputing.com/en/tutorial-introduction-to-caching-functions-of-the-jackhammer-extension-by-old-world-computing/

    Scott
  • rfuentealba
    rfuentealba New Altair Community Member
    Hey @jczogalla and @sgenzer,

    Since I've found this problem too, I can provide a few details:

    This is not about the amount of data. In fact, the error comes from trying to retrieve the list of tables from a database in the Parameters list. However, this was solved in RapidMiner 9.3. It's still a little slow (40 seconds on my MacBook i9-9900) but manageable.



    @DU_ca, you may want to upgrade to 9.3 to see if this works.

    All the best,

    Rodrigo.

  • DU_ca
    DU_ca New Altair Community Member
    Upgrading to 9.3 didn't help.  It's still taking a long time.  I also use SQL Developer for some of my work and it loads everything within seconds.  Why the difference?
  • sgenzer
    sgenzer
    Altair Employee
    hi @DU_ca so as was explained above, RapidMiner is a completely different architecture to SQL. If your db has a mountain of tables, RM has to go through the schema, grab them all, and then display them in its own architecture. 

    I guess if it were me I would take a small step back and ask what are you trying to do? Most of the time if you have a large db it is far more efficient to simply push pre-written SQL queries (e.g. write them in SQL Developer) using the Read Database operator and then grab the results for future work in RapidMiner. I am certain that SQL Developer is going to be a lot easier & faster for writing SQL than the very basic editor in RapidMiner. It's not what RM is built for. Most data scientists have a variety of tools that they use in tandem...no one tool is going to be the most efficient in all circumstances.

    Does that help?

    Scott

  • sgenzer
    sgenzer
    Altair Employee
    hi @Blodgett20 what kind of accurate solution are you looking for that has not been addressed here? We're happy to help but I'd like to know more about what exactly your struggles are.

    Thanks.

    Scott

Categories