Replace Token Solution for abbreviates that are part of other words

TRSisme05
TRSisme05 New Altair Community Member
edited November 5 in Community Q&A
I am working on a text analysis. There are abbreviations in my original text, such as cust or cust. for customer. I can put the replace token operator before the tokenize operator and enter multiple replacements such as replace cust space with customer and cust. with customer, but I am curious if there is a way to do it after the tokenization because it has "grouped" the cust abbreviations together. I did try placing the replace operator after tokenization but it replaced all occurrences of cust with customer, including the full word customer. Any thoughts/ideas?  thank you for your help.

 
Tagged:

Best Answer

Answers

  • kayman
    kayman New Altair Community Member
    One option is to create your own stemming library, as in the end that's your goal here, to group similar words. The stem dictionary would be able to do that but it requires setting up a library yourself adding all of the words and can be tricky if characters are shared.

    So adding cust* would stem customer, customers etc to cust (or whatever you choose) , but it would do the same with customs or custody so be careful.

    Replacing using regex might be more secure, just ensure you use (word) boundaries in that case, so \bcust\b would replace only cust to customer, and leave all other words containing cust untouched. 
  • lionelderkrikor
    lionelderkrikor New Altair Community Member
    Hi @TRSisme05,

    To perform your task use the following regex in the Replace operator :



    Regards,

    Lionel 


  • TRSisme05
    TRSisme05 New Altair Community Member
    lionelderkrikor (cust)+$ does not replace cust within other words, but it only replaces cust space I still need to create multiple replacements for cust. etc