An Introduction to Filtering in 10.9
Altair EmployeeHi All,
I've been asked a few times the best way to do filtering within Knowledge Studio and up until this release my answer was always either a work around using a Variable Transform node and a Stratified Partition, or by using a code node.....until now!!
The most recent version has brought a much nicer solution, replacing the old filter node we had a couple of years ago. Not only do you have a new 'Filter' node, but you can also do filtering within the data node itself!
Those of you who haven't updated to the latest version (of both Studio and Seeker), I think this feature alone makes it worthwhile!
Here's a quick walk-through of the new capabilities.
Data Node Filtering:
This allows you to filter all of the data that you see within the data nodes. You will now see a small yellow arrow on the left side of the tab which will slide out to show the filter definition tab.
The definition will allow you to select:
- any field,
- a selection of conditions (equals, not equals, greater than, in a list, not in a list, between, etc.),
- and values (which can be typed or selected from a list).
The data displayed on the right side here will not be filtered until you click the 'Apply' button.
For pages which require user interaction, such as the 'Overview report' shown here or the 'Segment Viewer', will need to be recalculated like when you first opened the node. Others, such as the 'Data' tab will update automatically. You can see that 'Records' and 'Cardinality' fields have been updated however.
Here you can see the new minimum age is 40. The filter works!
All tabs within the data node are affected by the filtering, as can be previewed here. This includes 'Correlations' and the 'Segment Viewer'. Pretty cool, right?
Now that the filter has been applied and all relevant tabs updated, you are able to very quickly enable and disable the filter by toggling the funnel symbol at the top (it kind of looks like a torch). The filtered and unfiltered pages have now been cached, so no need to refresh anything, it's instant. Very useful when combined with the outlier analysis flags.
You can add as many conditions as you need, the statements are joined with an 'AND'. For example we're looking at Age >= 40 AND sex='Female' below.
If you are happy with the filtering you can easily create a new dataset with just the filtered data by selecting the 'Add to Workflow' button on the filter tab.
This will add both a filter node and dataset node (with a name of your choice) to the workflow as shown here.
As you can see the Filter node hasn't been run yet. Right clicking will enable you to select from 'Run' without entering the node, or you can double click to modify it.
Filter Node Filtering:
A filter node can also be added directly to the workflow from the palette if you prefer and you will see the same wizard as when you 'modify' from the flow above. The filter node itself allows for much more flexibility/complexity with the filters. Here you can select from 'AND' and 'OR' operators.
You can also preview the results by selecting the 'Show Data' button.
And with the 'Show Data' visible...
If you want to do more complex filtering such as including calculations, then you can program them in the Expression Editor.
This works in the same way as with the 'Variable Transformations' node – provided it's a Boolean expression (essentially results in a 1 or 0).
Below we've added an aggregate calculation of Average(age) and the preview shows we have narrowed the selection down to just 21 results.
I hope that makes your life easier, I know it helped me a lot! I'll follow this up with a short video soon too.
Please let us know if anything that wasn't clear, any filtering feedback, and, as always, if there's anything else you would like to see in the tool.
Happy Filtering!
------------------------------
Alex Gobolos
Sales Engineer
Datawatch Corporation
Toronto ON
------------------------------