🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

[SOLVED] Remove Duplicates Weird Problem !

User: "aryan_hosseinza"
aryan_hosseinza
New Altair Community Member
Updated by Jocelyn
Hi  ,

I had used the remove duplicate operator on a dataset with 5 million records on my laptop and on a powerful linux system , it'd run in 7 seconds on both systems,

Now I have a dataset with 8 million records (same number of attributes) , it runs on my laptop in 10 seconds ! but it doesn't finish running on the same remote server ! it took 8 hours but it didn't finish !

It's very weird !!! It's a very simple operation !

How could something like this happen ? Is there anyway to solve it ?

Thanks,
Arian

Find more posts tagged with

AI Studio
Sort by:
1 - 2 of 21
    User: "MariusHelf"
    MariusHelf
    New Altair Community Member
    Hi Arian,

    maybe the server ran out of memory? Do you use the same memory settings on both machines?

    Best regards,
    Marius
    User: "aryan_hosseinza"
    aryan_hosseinza
    New Altair Community Member
    OP
    Actually It was very wired , I was doing some missing value replacing and now I changed the way I was doing it , it now works (the result of both missing value replacements seems to be the same ! but remove duplication doesn't work on one of them)

    Thanks by the way ,

    -Arian
Quick Links