🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

[SOLVED] How to count 0 values in large set of attributes?

User: "mikeb1029"
New Altair Community Member
Updated by Jocelyn
I am working with TF-IDF data, and I want to count the number of terms (attributes) for each document (example) that have a TF-IDF value of 0.  At this time we have approximately 8000 examples and 4000 attributes.  It seems there is no way to count numeirc values, so I've tried several types of conversions.  I have tried converting the 0 values to missing attributes and counting those.  I have tried converting the numeric TF-IDF scores to polynomials and counting those.  I've also tried to discretize the TF-IDF values into 2 bins (0, and >0).  After the conversions I mentioned, I use an Extract Macro operator to count with parameters of macro type: statistics, statistics: count, attribute name: %{loop_attribute}, and the appropriate attribute value for the type of conversion I did previously in the process (an adaptation of Haddock's Missing Value Count workflow (http://www.myexperiment.org/workflows/1292.html).  All of these different conversions eventually provide the results I want, but the performance on a sample set of 850 examples and 165 attributes takes 5 minutes or more, so I know the performance with the full set of data would be way too slow for our needs.  

What am I doing wrong?  It seems that counting values like I want to do here should be something that RapidMiner could do in the blink of an eye?

Find more posts tagged with