How do I tokenize an example set?

darwin_h_poritz
darwin_h_poritz New Altair Community Member
edited November 5 in Community Q&A
I have an example set of 451 comments (rows).  I want to generate mult-word tokens (n-Grams) from this example set.  What is the sequence of operators?
Tagged:

Answers

  • Telcontar120
    Telcontar120 New Altair Community Member
    You'll first need to set your attribute to text data type (it is probably nominal by default), and then use the Process Documents from Data operator.  Within that you will put the tokenize operator followed by the N-Grams operator.  You may also wish to conduct other types of text preprocessing as well. 
  • greg_lorincz79
    greg_lorincz79 New Altair Community Member
    edited March 2021
     <3 

Categories