🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

[SOLVED] How to count 0 values in large set of attributes?

User: "mikeb1029"
New Altair Community Member
Updated by Jocelyn
I am working with TF-IDF data, and I want to count the number of terms (attributes) for each document (example) that have a TF-IDF value of 0.  At this time we have approximately 8000 examples and 4000 attributes.  It seems there is no way to count numeirc values, so I've tried several types of conversions.  I have tried converting the 0 values to missing attributes and counting those.  I have tried converting the numeric TF-IDF scores to polynomials and counting those.  I've also tried to discretize the TF-IDF values into 2 bins (0, and >0).  After the conversions I mentioned, I use an Extract Macro operator to count with parameters of macro type: statistics, statistics: count, attribute name: %{loop_attribute}, and the appropriate attribute value for the type of conversion I did previously in the process (an adaptation of Haddock's Missing Value Count workflow (http://www.myexperiment.org/workflows/1292.html).  All of these different conversions eventually provide the results I want, but the performance on a sample set of 850 examples and 165 attributes takes 5 minutes or more, so I know the performance with the full set of data would be way too slow for our needs.  

What am I doing wrong?  It seems that counting values like I want to do here should be something that RapidMiner could do in the blink of an eye?

Find more posts tagged with

Sort by:
1 - 3 of 31
    User: "MariusHelf"
    New Altair Community Member
    Hi,

    what you could do is to convert non-zeros to missings (as you did before), and then use the Aggregate operator with count (no missings) as default aggregation.

    Best regards,
    Marius
    User: "mikeb1029"
    New Altair Community Member
    OP
    Thanks for your help, Marius.  Your suggestion made me realize that I could do this without resorting to the Loop Attributes operator, and that ended up speeding things up tremendously.
    User: "MariusHelf"
    New Altair Community Member
    :-) Nice to hear that!