"Learning Algorithm that can handle complex input relationships?"

Ghostrider
Ghostrider New Altair Community Member
edited November 5 in Community Q&A
Is there a learning algorithm that can handle complicated input relationships?  For example, say I feed in 7 signals and it turns out that the difference between two of the signals perfectly explains the output signal.  Or what if the output is equal to 2 times one of the inputs, but only if another input is equal to 3, otherwise, the output is equal to some kind of function of the other inputs.  Is there a learning algorithm that can learn those types of inputs?
Tagged:

Answers

  • haddock
    haddock New Altair Community Member
    Hi there!

    Indeed there is, check out the 'Generate Function Set' operator, and there is an example of how to use it here...

    http://www.myexperiment.org/workflows/1321.html

    You'll need to provide the data as indicated..

    Have fun.

  • Ghostrider
    Ghostrider New Altair Community Member
    So that's basically a pre-processing step based on a genetic algorithm for combining operators.  There's no off-the-shelf learning algorithm which can analyze inputs as I described and come up with some relation.  I did read about one open-source project called Eureqa which can discover such relationships, but it also uses a pre-defined list of operations.  It's not fully generic so you have to have some idea of what you are searching for.
  • haddock
    haddock New Altair Community Member
    Hi there,

    When you say..
    So that's basically a pre-processing step based on a genetic algorithm for combining operators.  
    What are you referring to, the operator ' Generate Function Set ', or the example I pointed to? The reason I ask is that the source of the operator is like this ... 
    /*
     *  RapidMiner
     *
     *  Copyright (C) 2001-2010 by Rapid-I and the contributors
     *
     *  Complete list of developers available at our web site:
     *
     *       http://rapid-i.com
     *
     *  This program is free software: you can redistribute it and/or modify
     *  it under the terms of the GNU Affero General Public License as published by
     *  the Free Software Foundation, either version 3 of the License, or
     *  (at your option) any later version.
     *
     *  This program is distributed in the hope that it will be useful,
     *  but WITHOUT ANY WARRANTY; without even the implied warranty of
     *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     *  GNU Affero General Public License for more details.
     *
     *  You should have received a copy of the GNU Affero General Public License
     *  along with this program.  If not, see http://www.gnu.org/licenses/.
     */
    package com.rapidminer.operator.features.construction;

    import java.util.ArrayList;
    import java.util.Iterator;
    import java.util.LinkedList;
    import java.util.List;

    import com.rapidminer.example.Attribute;
    import com.rapidminer.example.ExampleSet;
    import com.rapidminer.generator.AbsoluteValueGenerator;
    import com.rapidminer.generator.BasicArithmeticOperationGenerator;
    import com.rapidminer.generator.ExponentialFunctionGenerator;
    import com.rapidminer.generator.FeatureGenerator;
    import com.rapidminer.generator.FloorCeilGenerator;
    import com.rapidminer.generator.MinMaxGenerator;
    import com.rapidminer.generator.PowerGenerator;
    import com.rapidminer.generator.ReciprocalValueGenerator;
    import com.rapidminer.generator.SquareRootGenerator;
    import com.rapidminer.generator.TrigonometricFunctionGenerator;
    import com.rapidminer.operator.OperatorDescription;
    import com.rapidminer.operator.OperatorException;
    import com.rapidminer.operator.ports.metadata.AttributeMetaData;
    import com.rapidminer.operator.ports.metadata.ExampleSetMetaData;
    import com.rapidminer.operator.ports.metadata.MetaData;
    import com.rapidminer.parameter.ParameterType;
    import com.rapidminer.parameter.ParameterTypeBoolean;
    import com.rapidminer.parameter.UndefinedParameterError;
    import com.rapidminer.tools.Ontology;
    import com.rapidminer.tools.math.MathFunctions;

    /**
     * This operator applies a set of functions on all features of the input example set. Applicable functions include +, -,
     * *, /, norm, sin, cos, tan, atan, exp, log, min, max, floor, ceil, round, sqrt, abs, and pow. Features with two
     * arguments will be applied on all pairs. Non commutative functions will also be applied on all permutations.
     * 
     * @see com.rapidminer.generator.FeatureGenerator
     * @author Ingo Mierswa
     */
    public class CompleteFeatureGenerationOperator extends AbstractFeatureConstruction {

    	/**
    	 * The parameter name for "If set to true, all the original attributes are kept, otherwise they are removed
    	 * from the example set."
    	 */
    	public static final String PARAMETER_KEEP_ALL = "keep_all";

    	/** The parameter name for "Generate sums." */
    	public static final String PARAMETER_USE_PLUS = "use_plus";

    	/** The parameter name for "Generate differences." */
    	public static final String PARAMETER_USE_DIFF = "use_diff";

    	/** The parameter name for "Generate products." */
    	public static final String PARAMETER_USE_MULT = "use_mult";

    	/** The parameter name for "Generate quotients." */
    	public static final String PARAMETER_USE_DIV = "use_div";

    	/** The parameter name for "Generate reciprocal values." */
    	public static final String PARAMETER_USE_RECIPROCALS = "use_reciprocals";

    	/** The parameter name for "Generate square root values." */
    	public static final String PARAMETER_USE_SQUARE_ROOTS = "use_square_roots";

    	/** The parameter name for "Generate the power of one attribute and another." */
    	public static final String PARAMETER_USE_POWER_FUNCTIONS = "use_power_functions";

    	/** The parameter name for "Generate sinus." */
    	public static final String PARAMETER_USE_SIN = "use_sin";

    	/** The parameter name for "Generate cosinus." */
    	public static final String PARAMETER_USE_COS = "use_cos";

    	/** The parameter name for "Generate tangens." */
    	public static final String PARAMETER_USE_TAN = "use_tan";

    	/** The parameter name for "Generate arc tangens." */
    	public static final String PARAMETER_USE_ATAN = "use_atan";

    	/** The parameter name for "Generate exponential functions." */
    	public static final String PARAMETER_USE_EXP = "use_exp";

    	/** The parameter name for "Generate logarithmic functions." */
    	public static final String PARAMETER_USE_LOG = "use_log";

    	/** The parameter name for "Generate absolute values." */
    	public static final String PARAMETER_USE_ABSOLUTE_VALUES = "use_absolute_values";

    	/** The parameter name for "Generate minimum values." */
    	public static final String PARAMETER_USE_MIN = "use_min";

    	/** The parameter name for "Generate maximum values." */
    	public static final String PARAMETER_USE_MAX = "use_max";

    	/** The parameter name for "Generate ceil values." */
    	public static final String PARAMETER_USE_CEIL = "use_ceil";

    	/** The parameter name for "Generate floor values." */
    	public static final String PARAMETER_USE_FLOOR = "use_floor";

    	/** The parameter name for "Generate rounded values." */
    	public static final String PARAMETER_USE_ROUNDED = "use_rounded";

    	public CompleteFeatureGenerationOperator(OperatorDescription description) {
    		super(description);
    	}

    @Override
    	protected MetaData modifyMetaData(ExampleSetMetaData metaData) throws UndefinedParameterError {
    		// counting numerical attributes
    		int numberOfNumerical = 0;
    		for (AttributeMetaData amd : metaData.getAllAttributes()) {
    			if (amd.isNumerical() && !amd.isSpecial())
    				numberOfNumerical++;
    		}

    		// clear regular if needed
    		if (!getParameterAsBoolean(PARAMETER_KEEP_ALL))
    			metaData.clearRegular();

    		// new ones
    		int numberOfAdditionalAttributes = 0;
    		int commutativeNonSelfapplicable = MathFunctions.factorial(numberOfNumerical - 1);
    		int commutativeSelfapplicable = MathFunctions.factorial(numberOfNumerical);
    		int nonCommuatativeSelfApplicable = numberOfNumerical * numberOfNumerical;
    		int nonCommuatativeNonSelfApplicable = numberOfNumerical * numberOfNumerical;

    		if (getParameterAsBoolean(PARAMETER_USE_PLUS))
    			numberOfAdditionalAttributes += commutativeNonSelfapplicable;
    		if (getParameterAsBoolean(PARAMETER_USE_DIFF))
    			numberOfAdditionalAttributes += nonCommuatativeNonSelfApplicable;
    		if (getParameterAsBoolean(PARAMETER_USE_MULT))
    			numberOfAdditionalAttributes += commutativeSelfapplicable;
    		if (getParameterAsBoolean(PARAMETER_USE_DIV))
    			numberOfAdditionalAttributes += nonCommuatativeNonSelfApplicable;
    		if (getParameterAsBoolean(PARAMETER_USE_RECIPROCALS))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_SQUARE_ROOTS)) {
    			numberOfAdditionalAttributes += numberOfNumerical;
    		}
    		if (getParameterAsBoolean(PARAMETER_USE_POWER_FUNCTIONS)) {
    			numberOfAdditionalAttributes += nonCommuatativeSelfApplicable;
    		}

    		if (getParameterAsBoolean(PARAMETER_USE_SIN))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_COS))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_TAN))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_ATAN))
    			numberOfAdditionalAttributes += numberOfNumerical;

    		if (getParameterAsBoolean(PARAMETER_USE_EXP))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_LOG))
    			numberOfAdditionalAttributes += numberOfNumerical;

    		if (getParameterAsBoolean(PARAMETER_USE_ABSOLUTE_VALUES))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_MIN))
    			numberOfAdditionalAttributes += commutativeNonSelfapplicable;
    		if (getParameterAsBoolean(PARAMETER_USE_MAX))
    			numberOfAdditionalAttributes += commutativeNonSelfapplicable;

    		if (getParameterAsBoolean(PARAMETER_USE_CEIL))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_FLOOR))
    			numberOfAdditionalAttributes += numberOfNumerical;
    		if (getParameterAsBoolean(PARAMETER_USE_ROUNDED))
    			numberOfAdditionalAttributes += numberOfNumerical;

    		for (int i = 0; i < numberOfAdditionalAttributes; i++) {
    			if (i == 0)
    				metaData.addAttribute(new AttributeMetaData("gensym", Ontology.REAL));
    			else
    				metaData.addAttribute(new AttributeMetaData("gensym" + i, Ontology.REAL));
    		}

    		return metaData;
    	}

    @Override
    	public ExampleSet apply(ExampleSet exampleSet) throws OperatorException {
    		// set selection mode to restrictive mode
    		FeatureGenerator.setSelectionMode(FeatureGenerator.SELECTION_MODE_RESTRICTIVE);

    		List<FeatureGenerator> generators = getGenerators();
    		List<FeatureGenerator> generatorList = new LinkedList<FeatureGenerator>();
    		Iterator<FeatureGenerator> i = generators.iterator();
    		while (i.hasNext()) {
    			FeatureGenerator generator = i.next();
    			List<Attribute[]> inputAttributes = generator.getInputCandidates(exampleSet, new String[0]);
    			Iterator<Attribute[]> a = inputAttributes.iterator();
    			while (a.hasNext()) {
    				Attribute[] args = a.next();
    				FeatureGenerator newGenerator = generator.newInstance();
    				newGenerator.setArguments(args);
    				generatorList.add(newGenerator);
    			}
    		}

    		// generate all new attributes
    		if (!getParameterAsBoolean(PARAMETER_KEEP_ALL)) {
    			exampleSet.getAttributes().clearRegular();
    		}

    		List<Attribute> newAttributes = FeatureGenerator.generateAll(exampleSet.getExampleTable(), generatorList);
    		for (Attribute newAttribute : newAttributes)
    			exampleSet.getAttributes().addRegular(newAttribute);

    		return exampleSet;
    	}

    	private List<FeatureGenerator> getGenerators() {
    		List<FeatureGenerator> generators = new ArrayList<FeatureGenerator>();
    		if (getParameterAsBoolean(PARAMETER_USE_PLUS))
    			generators.add(new BasicArithmeticOperationGenerator(BasicArithmeticOperationGenerator.SUM));
    		if (getParameterAsBoolean(PARAMETER_USE_DIFF))
    			generators.add(new BasicArithmeticOperationGenerator(BasicArithmeticOperationGenerator.DIFFERENCE));
    		if (getParameterAsBoolean(PARAMETER_USE_MULT))
    			generators.add(new BasicArithmeticOperationGenerator(BasicArithmeticOperationGenerator.PRODUCT));
    		if (getParameterAsBoolean(PARAMETER_USE_DIV))
    			generators.add(new BasicArithmeticOperationGenerator(BasicArithmeticOperationGenerator.QUOTIENT));
    		if (getParameterAsBoolean(PARAMETER_USE_RECIPROCALS))
    			generators.add(new ReciprocalValueGenerator());
    		if (getParameterAsBoolean(PARAMETER_USE_SQUARE_ROOTS)) {
    			generators.add(new SquareRootGenerator());
    		}
    		if (getParameterAsBoolean(PARAMETER_USE_POWER_FUNCTIONS)) {
    			generators.add(new PowerGenerator());
    		}

    		if (getParameterAsBoolean(PARAMETER_USE_SIN))
    			generators.add(new TrigonometricFunctionGenerator(TrigonometricFunctionGenerator.SINUS));
    		if (getParameterAsBoolean(PARAMETER_USE_COS))
    			generators.add(new TrigonometricFunctionGenerator(TrigonometricFunctionGenerator.COSINUS));
    		if (getParameterAsBoolean(PARAMETER_USE_TAN))
    			generators.add(new TrigonometricFunctionGenerator(TrigonometricFunctionGenerator.TANGENS));
    		if (getParameterAsBoolean(PARAMETER_USE_ATAN))
    			generators.add(new TrigonometricFunctionGenerator(TrigonometricFunctionGenerator.ARC_TANGENS));

    		if (getParameterAsBoolean(PARAMETER_USE_EXP))
    			generators.add(new ExponentialFunctionGenerator(ExponentialFunctionGenerator.EXP));
    		if (getParameterAsBoolean(PARAMETER_USE_LOG))
    			generators.add(new ExponentialFunctionGenerator(ExponentialFunctionGenerator.LOG));

    		if (getParameterAsBoolean(PARAMETER_USE_ABSOLUTE_VALUES))
    			generators.add(new AbsoluteValueGenerator());
    		if (getParameterAsBoolean(PARAMETER_USE_MIN))
    			generators.add(new MinMaxGenerator(MinMaxGenerator.MIN));
    		if (getParameterAsBoolean(PARAMETER_USE_MAX))
    			generators.add(new MinMaxGenerator(MinMaxGenerator.MAX));

    		if (getParameterAsBoolean(PARAMETER_USE_CEIL))
    			generators.add(new FloorCeilGenerator(FloorCeilGenerator.CEIL));
    		if (getParameterAsBoolean(PARAMETER_USE_FLOOR))
    			generators.add(new FloorCeilGenerator(FloorCeilGenerator.FLOOR));
    		if (getParameterAsBoolean(PARAMETER_USE_ROUNDED))
    			generators.add(new FloorCeilGenerator(FloorCeilGenerator.ROUND));
    		return generators;
    	}

    @Override
    	public List<ParameterType> getParameterTypes() {
    		List<ParameterType> types = super.getParameterTypes();
    		types.add(new ParameterTypeBoolean(PARAMETER_KEEP_ALL, "If set to true, all the original attributes are kept, otherwise they are removed from the example set.", true, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_PLUS, "Generate sums.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_DIFF, "Generate differences.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_MULT, "Generate products.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_DIV, "Generate quotients.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_RECIPROCALS, "Generate reciprocal values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_SQUARE_ROOTS, "Generate square root values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_POWER_FUNCTIONS, "Generate the power of one attribute and another.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_SIN, "Generate sinus.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_COS, "Generate cosinus.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_TAN, "Generate tangens.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_ATAN, "Generate arc tangens.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_EXP, "Generate exponential functions.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_LOG, "Generate logarithmic functions.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_ABSOLUTE_VALUES, "Generate absolute values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_MIN, "Generate minimum values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_MAX, "Generate maximum values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_CEIL, "Generate ceil values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_FLOOR, "Generate floor values.", false, false));
    		types.add(new ParameterTypeBoolean(PARAMETER_USE_ROUNDED, "Generate rounded values.", false, false));
    		return types;
    	}
    }
    and the example XML is just a loop that keeps the best formulae in the 'Construction" column of the meta-data. So where is  the genetic algorithm hiding? Ah, Eureka! It is there! http://www.hakank.org/eureqa/



  • IngoRM
    IngoRM New Altair Community Member
    Hi,

    want to add a note (since I did a lot of research in this topic like 10 years ago):

    - if you want an automatic way for creating those complex features and their interactions, you could try the operator Yagga2
    - personally I think that having this automatic feature construction with a robust inner learner is much more stable than for example Genetic Programming, but of course this would also be an option for you
    - if you want to read more about this, you could try some of my papers including the second part of my PhD: http://www-ai.cs.uni-dortmund.de/PERSONAL/mierswa.html

    Cheers,
    Ingo

Categories