Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Why is my fetch taking so long?

I am able to connect to my Oracle database system, but the "Fetching Database Tables" process takes 45 minutes to complete. Why is this happening?

Find more posts tagged with

AI Studio

Accepted answers

All comments

rfuentealba

Hello @DU_ca:

It appears to be something not fully compliant in the JDBC driver for Oracle that requires an extra step. Let's call @mschmitz and ask him if he knows someone who can help us, because I tried to clone the RapidMiner Studio source code and found out that the open source version of it doesn't include the JDBC connectors, so I just used Navicat and copy/pasted :-(

All the best,

Rodrigo.

DU_ca

Thanks for trying, Rodrigo. Unfortunately, I'm still having this problem and so I guess RapidMiner is not a good option for me.

jczogalla

Hi @DU_ca!

Can you maybe share your process and/or explain what exactly you want to do?

If you are trying to load all your data from your database, this might be quite some effort to do if you have a huge database. RapidMiner has to convert each table into its own memory format, which might take a lot of RAM and time.

If you want to manipulate a lot of your database data, especially preprocessing, it maybe would help you to use the in-DB extension from the marketplace, as discussesd here: https://community.rapidminer.com/discussion/55129/challenge-with-rm-server-running-out-of-memory#latest

I hope this helps!

Jan

sgenzer

using the caching tools in the Jackhammer extension can also be helpful here...

https://oldworldcomputing.com/en/tutorial-introduction-to-caching-functions-of-the-jackhammer-extension-by-old-world-computing/

Scott

rfuentealba

Hey @jczogalla and @sgenzer,

Since I've found this problem too, I can provide a few details:

This is not about the amount of data. In fact, the error comes from trying to retrieve the list of tables from a database in the Parameters list. However, this was solved in RapidMiner 9.3. It's still a little slow (40 seconds on my MacBook i9-9900) but manageable.

Image: https://us.v-cdn.net/6030995/uploads/editor/a3/447k2mrbv4ru.png

@DU_ca, you may want to upgrade to 9.3 to see if this works.

All the best,

Rodrigo.

DU_ca

Upgrading to 9.3 didn't help. It's still taking a long time. I also use SQL Developer for some of my work and it loads everything within seconds. Why the difference?

sgenzer

hi @DU_ca so as was explained above, RapidMiner is a completely different architecture to SQL. If your db has a mountain of tables, RM has to go through the schema, grab them all, and then display them in its own architecture.

I guess if it were me I would take a small step back and ask what are you trying to do? Most of the time if you have a large db it is far more efficient to simply push pre-written SQL queries (e.g. write them in SQL Developer) using the Read Database operator and then grab the results for future work in RapidMiner. I am certain that SQL Developer is going to be a lot easier & faster for writing SQL than the very basic editor in RapidMiner. It's not what RM is built for. Most data scientists have a variety of tools that they use in tandem...no one tool is going to be the most efficient in all circumstances.

Does that help?

Scott

sgenzer

hi @Blodgett20 what kind of accurate solution are you looking for that has not been addressed here? We're happy to help but I'd like to know more about what exactly your struggles are.

Thanks.

Scott