🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Using 'external' word2vec models

User: "kayman"
kayman
New Altair Community Member
Updated by Jocelyn
Hi there, might be overlooking something but when I tried to use some word2vec models generated using colab I got an error message as below :

Message: Expected a space in the first line of file '/data/models/word2vec/tryout': 'ﾀcgensim.models.word2vec'

Does this mean we cannot really use word2vec models generated outside of rapidminer, even if they are generated 'according to the rules'?
Or is there a way to bypass these errors?

I could in the end regenerate the model in rapidminer also, but it's a bit less friendly on server resources and limits sharing across different apps so I'd prefer to be able to create once and share many.

Find more posts tagged with

AI Studio
Word2Vec
Text Mining + NLP
Sort by:
1 - 7 of 71
    User: "sgenzer"
    sgenzer
    Altair Employee
    cc @mschmitz
    User: "MartinLiebig"
    MartinLiebig
    Altair Employee
    Hi @kayman ,

    did you try the read word2vec model operator on the file?

    BR,
    Martin
    User: "kayman"
    kayman
    New Altair Community Member
    OP

    It is the read word2vec model operator that gives me the error.
    It doesn't really work the other way around either. Creating a word2vec model using Rapidminer works fine within RM, but the model fails if I want to use it in a python workflow using Gensim. Then I get a UnpicklingError stating invalid load key.

    Guess they are not fully compatible indeed?




    User: "MartinLiebig"
    MartinLiebig
    Altair Employee
    Accepted Answer
    Hi @kayman,
    the current implementation only supports binary files. This is the format used by  google: https://code.google.com/archive/p/word2vec/ .
    I am not sure which format gensim uses, but maybe you can change it? 

    BR,
    Martin
    User: "kayman"
    kayman
    New Altair Community Member
    OP
    Hi @mschmitz,

    Gensim is also using binaries, but I went through some of the documentation and they state as follows : 

    The training algorithms were originally ported from the C package https://code.google.com/p/word2vec/ and extended with additional functionality and optimizations over the years.

    So probably the changes are what causes the incompatibility problem. 

    Thanks anyway, I'll dig deeper in the documentation to understand if there is a way to get a more 'core' format output , and otherwise it's just working with duplicates.
    User: "MartinLiebig"
    MartinLiebig
    Altair Employee
    Hi @kayman,
    we will probably touch word2vec "soon". I'll add gensim compatibility to the todo list.
    BR,
    Martin
    User: "kayman"
    kayman
    New Altair Community Member
    OP
    Thanks @mschmitz

    Anyway, some further diving into the gensim code learned me that with using alternative load and save methods using KeyedVectors the model can be interchangeable after all.

    For the ones interested, loading a rapidminer generated model with gensim word2vec can be done like this : 
    myModel = gensim.models.KeyedVectors.load_word2vec_format(path_to_model_from_RapidMiner, binary=True)
    Saving a model from gensim to be used with rapidminer requires the save_word2vec_format, rather than using the 'gensim optimised' save procedures.
Quick Links