--- title: "DecisionTree" language: "en" type: "Method" summary: "DecisionTree (Machine Learning Method) Method for Predict, Classify and LearnDistribution. Use a decision tree to model class probabilities, value predictions or probability densities. A decision tree is a flow chart\\[Dash]like structure in which each internal node represents a test on a feature, each branch represents the outcome of the test, and each leaf represents a class distribution, value distribution or probability density. For Classify and Predict, the tree is constructed using the CART algorithm. For LearnDistribution, the splits are determined using an information criterion trading off the likelihood and the complexity of the model. The following options can be given:" keywords: - Machine Learning - CART algorithm - Prediction - Classification - Regression - Tree - Decision algorithm canonical_url: "https://reference.wolfram.com/language/ref/method/DecisionTree.html" source: "Wolfram Language Documentation" --- # "DecisionTree" (Machine Learning Method) * Method for ``Predict``, ``Classify`` and ``LearnDistribution``. * Use a decision tree to model class probabilities, value predictions or probability densities. --- ## Details & Suboptions * A decision tree is a flow chart–like structure in which each internal node represents a "test" on a feature, each branch represents the outcome of the test, and each leaf represents a class distribution, value distribution or probability density. * For ``Classify`` and ``Predict``, the tree is constructed using the CART algorithm. * For ``LearnDistribution``, the splits are determined using an information criterion trading off the likelihood and the complexity of the model. * The following options can be given: | | | | | ------------------------ | - | -------------------------------------------------------------------------------------------- | | "DistributionSmoothing" | 1 | regularization parameter | | "FeatureFraction" | 1 | the fraction of features to be randomly selected for training (only in Classify and Predict) | --- ## Examples (7) ### Basic Examples (3) Train a predictor function on labeled examples: ```wl In[1]:= p = Predict[{1, 2, 3, 4} -> {.3, .4, .6, 9}, Method -> "DecisionTree"] Out[1]= PredictorFunction[Association["ExampleNumber" -> 4, "Input" -> Association["Preprocessor" -> MachineLearning`MLProcessor["ToMLDataset", Association["Input" -> Association["f1" -> Association["Type" -> "Numerical"]], "Output" -> As ... " -> DateObject[{2025, 6, 25, 0, 54, 16.0945732`8.95925440996189}, "Instant", "Gregorian", -5.], "ProcessorCount" -> 4, "ProcessorType" -> "x86-64", "OperatingSystem" -> "Windows", "SystemWordLength" -> 64, "Evaluations" -> {}]]] ``` Look at the information about the predictor: ```wl In[2]:= Information[p] Out[2]= MachineLearning`MLInformationObject[PredictorFunction[Association["ExampleNumber" -> 4, "Input" -> Association["Preprocessor" -> MachineLearning`MLProcessor["ToMLDataset", Association["Input" -> Association["f1" -> Association["Type" -> ... DateObject[{2025, 6, 25, 0, 54, 16.0945732`8.95925440996189}, "Instant", "Gregorian", -5.], "ProcessorCount" -> 4, "ProcessorType" -> "x86-64", "OperatingSystem" -> "Windows", "SystemWordLength" -> 64, "Evaluations" -> {}]]]] ``` Extract option information that can be used for retraining: ```wl In[3]:= Information[p, "MethodOption"] Out[3]= Method -> {"DecisionTree", "DistributionSmoothing" -> 1, "FeatureFraction" -> 1} ``` Predict a new example: ```wl In[4]:= p[1.3] Out[4]= 2.575 ``` --- Generate some data and visualize it: ```wl In[1]:= data = Table[x -> Sin[x] + RandomVariate[NormalDistribution[0, .2]], {x, RandomReal[{-10, 10}, 400]}]; ListPlot[List@@@data] Out[1]= [image] ``` Train a predictor function on it: ```wl In[2]:= p = Predict[data, Method -> "DecisionTree"] Out[2]= PredictorFunction[Association["ExampleNumber" -> 400, "Input" -> Association["Preprocessor" -> MachineLearning`MLProcessor["ToMLDataset", Association["Input" -> Association["f1" -> Association["Type" -> "Numerical"]], "Output" -> ... "Date" -> DateObject[{2018, 11, 7, 17, 37, 36.43681`8.314115327333338}, "Instant", "Gregorian", -8.], "ProcessorCount" -> 4, "ProcessorType" -> "x86-64", "OperatingSystem" -> "MacOSX", "SystemWordLength" -> 64, "Evaluations" -> {}]]] ``` Compare the data with the predicted values and look at the standard deviation: ```wl In[3]:= Show[Plot[ {p[x], p[x] + StandardDeviation[p[x, "Distribution"]], p[x] - StandardDeviation[p[x, "Distribution"]]}, {x, -2, 6}, PlotStyle -> {Blue, Gray, Gray}, Filling -> {2 -> {3}}, Exclusions -> False, PerformanceGoal -> "Speed", PlotLegends -> {"Prediction", "Confidence Interval"}], ListPlot[List@@@data, PlotStyle -> Red, PlotLegends -> {"Data"}]] Out[3]= [image] ``` --- Learn a distribution using the method ``"DecisionTree"`` : ```wl In[1]:= data = RandomVariate[NormalDistribution[], 1000]; In[2]:= ld = LearnDistribution[data, Method -> "DecisionTree"] Out[2]= LearnedDistribution[Association["ExampleNumber" -> 1000, "Preprocessor" -> MachineLearning`MLProcessor["ToMLDataset", Association["Input" -> Association["f1" -> Association["Type" -> "Numerical"]], "Output" -> Association["f1" -> Asso ... ["Quantiles" -> CompressedData["«620»"], "LeftBoundary" -> -3.7971842989176867, "LeftScale" -> 0.548251903313647, "LeftTailNorm" -> 0.025]], "Entropy" -> Around[1.4539717772634453, 0.051328472950245806], "EntropySampleSize" -> 200]] ``` Visualize the PDF obtained: ```wl In[3]:= Plot[PDF[ld, x], {x, -5, 5}, Filling -> Bottom] Out[3]= [image] ``` Obtain information about the distribution: ```wl In[4]:= Information[ld] Out[4]= MachineLearning`MLInformationObject[LearnedDistribution[Association["ExampleNumber" -> 1000, "Preprocessor" -> MachineLearning`MLProcessor["ToMLDataset", Association["Input" -> Association["f1" -> Association["Type" -> "Numerical"]], ... antiles" -> CompressedData["«621»"], "LeftBoundary" -> -3.7971842989176867, "LeftScale" -> 0.548251903313647, "LeftTailNorm" -> 0.025]], "Entropy" -> Around[1.4539717772634453, 0.051328472950245806], "EntropySampleSize" -> 200]]] ``` ### Options (4) #### "DistributionSmoothing" (2) Use the ``"DistributionSmoothing"`` option to train a classifier: ```wl In[1]:= c = Classify[{1, 2, 3, 4, 5, 6} -> {1, 1, 3, 3, 1, 3}, Method -> {"DecisionTree", "DistributionSmoothing" -> .3}] Out[1]= ClassifierFunction[…] ``` --- Use the mushrooms training set to train a classifier with the default value of ``"DistributionSmoothing"`` : ```wl In[1]:= data = ExampleData[{"MachineLearning", "Mushroom"}, "TrainingData"]; In[2]:= classifier = Classify[data, Method -> "DecisionTree"]; ``` Train a second classifier using a large ``"DistributionSmoothing"`` : ```wl In[3]:= smoothed = Classify[data, Method -> {"DecisionTree", "DistributionSmoothing" -> 100}] Out[3]= ClassifierFunction[«1»] ``` Compare the probabilities for examples from a test set: ```wl In[4]:= testdata = ExampleData[{"MachineLearning", "Mushroom"}, "TestData"]; In[5]:= sample = RandomSample[testdata, 4]; Dataset@<|"AutomaticClassifier" -> classifier[sample[[All, 1]], "Probabilities"], "SmoothedClassifier" -> smoothed[sample[[All, 1]], "Probabilities"]|> Out[5]= Dataset[Association["AutomaticClassifier" -> {Association["edible" -> 0.9999913194522589, "poisonous" -> 8.680547741058685*^-6], Association["edible" -> 0.9999913194522589, "poisonous" -> 8.680547741058685*^-6], Association["edible" -> ... ble" -> 0.9999524924913398, "poisonous" -> 0.00004750750866022736], Association["edible" -> 0.9999524924913398, "poisonous" -> 0.00004750750866022736], Association["edible" -> 0.00006672995089214902, "poisonous" -> 0.9999332700491078]}]] ``` #### "FeatureFraction" (2) Use the`` "FeatureFraction"`` option to train a classifier: ```wl In[1]:= c = Classify[{{1, 2.3, 4, 5.3}, {2, 2.3, 2.4, 5}, {2, 2.3, 2.4, 5}, {1, 3, 4, -5.2}, {2, -5, -3.2, 5}, {2, 1.3, -8.1, 3.3}} -> {1, 1, 3, 3, 1, 3}, Method -> {"DecisionTree", "FeatureFraction" -> .5}] Out[1]= ClassifierFunction[…] ``` --- Use the mushrooms training set to train two classifiers with different values of ``"FeatureFraction"`` : ```wl In[1]:= data = ExampleData[{"MachineLearning", "Mushroom"}, "TrainingData"]; In[2]:= c1 = Classify[data, Method -> {"DecisionTree", "FeatureFraction" -> 1}] Out[2]= ClassifierFunction[…] In[3]:= c2 = Classify[data, Method -> {"DecisionTree", "FeatureFraction" -> .1}] Out[3]= ClassifierFunction[…] ``` Look at the accuracy of these classifiers on a test set: ```wl In[4]:= testdata = ExampleData[{"MachineLearning", "Mushroom"}, "TestData"]; In[5]:= ClassifierMeasurements[c1, testdata, "Accuracy"] Out[5]= 0.993437 In[6]:= ClassifierMeasurements[c2, testdata, "Accuracy"] Out[6]= 0.855209 ``` ## See Also * [`Classify`](https://reference.wolfram.com/language/ref/Classify.en.md) * [`Predict`](https://reference.wolfram.com/language/ref/Predict.en.md) * [`LearnDistribution`](https://reference.wolfram.com/language/ref/LearnDistribution.en.md) * [`ClassifierFunction`](https://reference.wolfram.com/language/ref/ClassifierFunction.en.md) * [`PredictorFunction`](https://reference.wolfram.com/language/ref/PredictorFunction.en.md) * [`LearnedDistribution`](https://reference.wolfram.com/language/ref/LearnedDistribution.en.md) * [`Information`](https://reference.wolfram.com/language/ref/Information.en.md) * [`RandomForest`](https://reference.wolfram.com/language/ref/method/RandomForest.en.md) * [`GradientBoostedTrees`](https://reference.wolfram.com/language/ref/method/GradientBoostedTrees.en.md) * [`NearestNeighbors`](https://reference.wolfram.com/language/ref/method/NearestNeighbors.en.md) * [`KernelDensityEstimation`](https://reference.wolfram.com/language/ref/method/KernelDensityEstimation.en.md) * [`SupportVectorMachine`](https://reference.wolfram.com/language/ref/method/SupportVectorMachine.en.md) * [`GaussianProcess`](https://reference.wolfram.com/language/ref/method/GaussianProcess.en.md) * [`NaiveBayes`](https://reference.wolfram.com/language/ref/method/NaiveBayes.en.md) ## Related Links * [An Elementary Introduction to the Wolfram Language: Machine Learning](https://www.wolfram.com/language/elementary-introduction/22-machine-learning.html) ## History * [Introduced in 2017 (11.2)](https://reference.wolfram.com/language/guide/SummaryOfNewFeaturesIn112.en.md)