Paper Example on Ensemble Classification Model

Introduction

The ensemble is a method that involves combining various algorithms of the same or different types known as base learners (Elizabeth et al., 2015). The purpose of these is to create a more efficient system that will effectively integrate the predictions from the various base learners. It is well elaborated by a meeting room involving multiple traders to decide if a specific price of a stock will increase or decrease. Since all the traders have their understanding of the stock market, they will produce unlike mapping functions from the issue regarding the desired outcome of the problem. Hence, they are required to make different predictions concerning the prices of stock that are based on their level of understanding of the stock market.

Is your time best spent reading someone else’s essay? Get a 100% original essay FROM A CERTIFIED WRITER!

Save 25%

All the predictions regarding the stock market will make the final decision tree stronger, accurate and less biased. The final decision would form the opposite conclusion that would not be in line with the stock market if one trader made the decision. Basic concepts for ensemble include averaging, majority rate and weighted average.

Average involve combining the average of predictions in a prediction model. It occurs when a problem arises during the regression computation or while forecasting the prospects of the classification. Majority rate involves considering the predictions that have the maximum number of occurrence or the reference from a multiple model prediction when predicting the outcomes of a grouping difficulty. The weighted average involves applying the different predictions from various models and taking the standard, which enables one to give a high or low position to correct model output.

Techniques commonly used different ensembling models include bagging, boosting and stacking. Bagging is also referred to as bootstrap aggregating that is designed to increase the strength and precision in the machine learning algorithms used in statistics and regression models. It is also used to decrease the level of variance that is key to preventing overfitting. Bootstrapping is a technique is used in sampling where one chooses 'n' number of rows and every row is selected with replacements from the initial data; rows equally selected in each iteration. There can be numerous bootstrapped samples from a single dataset. After establishing a bootstrap sample, trees can be grown from each of the datasets and average them to have the final prediction.

Random Forest Algorithm

It is an ensemble learning technique for the grouping and regression. Random Forest involves the construction of numerous decision trees at preparation class and produces a modal class output with specific trees (Rogozhnikov & Ikhomanenko, 2017). It is a mixture of tree interpreters affecting a tree reliant on the standards of the vector selected individually with a similar spread of trees in the forest. Random Forest principle is that a collection of "weak learners" collaborates to establish a "strong leaner". Generally, the more the number of trees in the forest, the sturdier the forest would seem. Similarly, a random forest classifier relies on a high number of trees in the forest to produce highly accurate and more reliable results. The information gain and Gini index approach utilised when forming a model with numerous decision trees.

Attribute Selection

The most common methods for selecting attribute measures and the information gain and Gini index. When a dataset comprises of a certain number of features, deciding on which attribute should be the root at the various levels of the trees as interior nodes is a sophisticated step. Choosing the node type randomly to be placed at the root is not able to provide a solution to the problem. Following a random approach may yield low accurate and poor quality results. Information gain and Gini index criterions were devised to compute the values for each feature. Values are organized, and attributes placed considering that characteristics with a high value are positioned at the root of individual trees. It is important to note that when using information gain, we assume that features are categorical while in the Gini index, elements are presumed to be constant.

Gain Index

It is a criterion used to estimate the amount of information limited in the attributes. Entropy can well define the certainty or improbability of a random variable X and by calculating the entropy, establishing the information gain is possible. The gain criterion helps calculate the anticipated entropy because attributes are sorting. A binary classification has only negative and positive classes when a model contains only one class, and the entropy will be zero (low) while the entropy is one (high) when the model contains the two classes. It can be represented as:HX=ExI(X)=xXpxlogp(x)

Gini Index

The Gini index is a criterion used in measuring how the frequency of a randomly selected element identified wrongly. The formula for calculating the Gini index is: 1-jp2jTree Bagging

The training system model for a random forest relates the overall methods for bagging among the tree learners. Assuming a training set B=b1......bn respondingC=c1......cn; repeatedly bagging (X times) chooses a random sample with substitutes of the tree in the following samples.

For X=1.....X. The regeneration tree fx on BxCx. Sampling with replacements of n (from B, C to BxCx). After sampling, the predictions for the unnoticed b' is achieved by finding the average of the predictions forms each separate regression trees on b'.

f=1Xx=1Xfx(b') or by considering the typical case of the classification trees.

The above technique results in a much improved model presentation as it reduces the variance and the bias remains constant. It means that when predictions of the individual trees are consistent, the average of the majority of the trees also remain unchanged with the condition that the trees are not correlated. If a single algorithm is defined, training several trees in one training would result in big interrelated trees. Therefore, bootstrap sampling is a technique utilised to de-correlate trees by displaying the trees in an, unlike training sets. Moreover, the estimation of the level of uncertainty can be determined by computing the SD of the estimates from the single regression trees.s=x=1X(fx(b')-f)2X-1X is a parameter that represents trees depending on the size of a set. The optimal X representing the number of trees can be computed by cross-validating.

Overfitting

Overfitting is a standard issue while modelling a decision tree model. A model is considered to have overfitting issues when the algorithm is constant while reducing the training set errors but the model still has a low accuracy of prediction. It commonly happens when many branches are built because of outliers and abnormalities in the data. Pre-pruning and post-pruning are the conventional methods used to avoid overfitting.

Pre-pruning assists in stopping the construction of a tree earlier because it is desired not to fragment a node it is a measure of precision is under a threshold value. Although it is sometimes tricky in choosing the suitable ending point.

Post-pruning goes to an advanced level to build a complete tree. In the occurrence of overfitting, pruning is conducted at the post-pruning step. The effect of post-pruning is measured by performing cross-validation of the data; it tests the effect of increasing a node. When an improvement is recorded, further expanding the node is allowed while a reduction in accuracy means that it should not be improved and should be transformed to a leaf node.

Assumptions While Creating a Decision Tree

When starting, a whole training set is regarded as root. A statistical method is utilised when placing features as roots or internal nodes of a tree.

Training sets are spread recursively based on the feature values and their categories.

Decision Tree Algorithm Advantage

They can be explained easily because it results in a set of rules.

It follows a similar methodology that humans use in decision-making.

Decision tree visualisation helps make simpler the understanding of a complicated decision.

The insignificant number of hyper-parameters adjusted.

Disadvantages

The probability of overfitting is high.

The low estimate precision of a dataset compared to other machine learning algorithms.

When using information gain in a decision tree with a dataset containing categorical variables, it yields a biased response for features that have a more significant number of categories.

In instances where there are several class labels, calculations can become difficult.

eXtreme Gradient Boosting (XGBoost)

Boosting is an ensemble technique that involves the predictor made in sequence and not individually. Boosting follows the reasoning that successive predictors utilise on the errors of the preceding predictors and try to improve on them. Consequently, the results possess an unequal probability or reoccurring in the succeeding models and the ones with the highest errors appear frequently. Predictors are selected based on vast models including decision trees, regressors. It takes a shorter time to make actual predictions because the fresh predictors are learning from mistakes. Choosing the appropriate stopping criteria will assist in avoiding overfitting of the dataset.

Gradient boosting algorithm is a regression-based and classification method that yields a prediction model showing weak decision trees (Introduction to Boosted Trees, n.d.) It utilised in supervising learning problems where training data with several features are used in predicting a target variable. In gradient boosting, the regression trees are used as the weaker learners. Decision trees are used to produce real values whose product can be summed to allow the output of the successive model to be added and fix the residuals. Compelling a weak learner in particular ways such as on the number of leaf nodes, and layers is a common phenomenon.

Model

In supervised learning, the model refers to the mathematical structure where the prediction ai made from the output bi. It can be represented by the grouping of evaluated input attributes described in a linear regression as ai=jthjbij. The estimate value's interpretation solely depends on the task; it is occasionally converted to the probability of a definite class in logistic regression. It is also utilised as a ranking score when the outputs are categorised (Chen & Guestrin, 2016).

Similar to other boosting techniques, XP boost combines weak learners to a sole strong learner. Any supervised learning algorithm aims to strive to describe and reduce and loss function by permanently updating the predictions ensuring that the summation of the residuals is at its minimum or zero. Predicted values are provided to be appropriately nearer to the actual value.

The Logic behind boost algorithm is to advantage the residual patterns repetitively with the purpose of strengthening the weaker predictions and improving them. Modelling can be stopped when residuals do not form any more modelling patterns if not, it will result in overfitting. Algorithmically, the loss function is being minimised to attain its minima. The impression is to use weak learning quite a lot of times to get a sequence of hypotheses, with each assumptions progressing on the basis that the earlier ones found a substantial level of misclassification.

In the additive model, trees are added one each time while the existing trees in the model are not altered. A gradient succession procedure is utilised to reduce the loss while adding trees. Conventionally, a gradient succession technique is to get control of a set of paramete...