Help Manual

Contents






Sigma Magic Help Version 15

XGBoost Analysis

Overview

XGBoost is a popular machine learning algorithm. It can handle both regression and classification problems and is well-known to provide better solutions than other algorithms. XGBoost stands for Extreme Gradient Boosting and is an optimized distributed gradient boosting library. It uses parallel computing, regularization to prevent overfitting, uses cross-validation, and can handle missing values. XGBoost is a family of boosting algorithms that convert weak learners to strong learners. Boosting is a sequential process - trees are grown using information from a previously grown tree one after the other. The process slowly learns from the data and tries to improve its prediction in subsequent iterations. Classification problems can be solved using the booster gbtree, while regression problems can be solved using gbtree and gblinear. It builds generalized linear models and optimizes them using regularization and gradient descent. Even though it is a very powerful algorithm, tuning the parameters can be a challenge.

The main menu of the Sigma Magic software is shown below. To add the XGBoost Analysis tool to your worksheet, click on Analytics and then select XGBoost Analysis.

Inputs

Click on Analysis Setup to open the menu options for this tool. A sample screenshot of the menu is shown below: XGBoosts Menu 1
1
Data Type: The data type for this analysis. The available options are:
OptionDescription
ContinuousPick this option is your Y metric is continuous data.
CategoricalUse this option if your Y metric is categorical data.
2
Input Type: Specify the format for input data. The only option currently available is Raw Data.
3
Training Set: Specify the data to be used for training the algorithms.
OptionDescription
AutoRandom method for selection of the training data set is used./td>
RandomA random number algorithm is used to select the training set.
SequentialA sequential set of samples are picked. For example: 1st point, 3rd point, 5th point etc.
4
Training Size: Specify what % of the data points should be picked for the training set. These are used to build the model and the remaining points are used to test the model.
5
Method: Specify the method for this analysis. The available options are:
OptionDescription
AutoSelect based on data type.
Reg:LinearUsed for linear regression./td>
Binary:LogisticUsed for binary logistic regression. It returns class probabilities.
multi:softmaxUsed for multiclassification. It returns class labels.
multi:softprobUsed for multiclassification. It returns predicted class probabilities.
6
Booster: Specify the booster type to use.
OptionDescription
AutoSelect based on data type
gbtreeUsed for categorical data./td>
gblinearUsed for continuous data.
7
Num Rounds: Specify the maximum number of iterations. For classification, it is similar to the number of trees to grow. The default value is 100.
8
Eta: Specify the learning rate, the rate at which the model learns from the data. After every round, it shrinks the feature weights to reach the optimum. The default value is 0.3 and range of this parameter is between 0 and 1.
9
Max Depth: It controls the depth of the tree. Larger the depth, more complex the model but higher the chance of overfitting. Smaller trees are faster to fit. The default value is 6.

For continuous data, specify the alpha value. Which is the L1 regularization on weights. In addition to shrinkage, enabling alpha results in feature selection. Hence, it is more useful in high-dimensional data sets. The default value is 1.
10
Gamma: Specify the parameter that controls regularization and hence prevents overfitting. Regularization penalizes large coefficients which don't improve model performance. The default value is 0 which means no regularization and the range for this parameter is from 0 to infinity.

For continuous data, specify the lambda value. Which is the L2 regularization of weights. It is used to avoid overfitting. The default value is 0.
11
Col Sample: Specify the number of variables supplied to the tree. The default value is 1 and the range for this parameter is from 0 to 1.
12
Sub Sample: Specify the number of samples (observations) supplied to a tree. The default value is 1 and the range for this parameter is between 0 and 1.
13
Additional Options: This field is optional. You can specify any additional options for the R software program directly by typing it here.
14
Help Button: Click on the Help Button to view the help documentation for this tool.
15
Cancel Button: Click on the Cancel Button to discard your changes and exit this menu.
16
OK Button: Click on the OK Button to save your changes and try to execute the program. Note that you will need to specify the required data in order to complete the analysis and generate outputs. If there are any missing data, then the software will remind you to specify the data and click on Compute Outputs to generate analysis results.

Data

If you click on the Data button, you will see the following dialog box. Here you can specify the data required for this analysis. Data
1
Search Data: The available data displays all the columns of data that are available for analysis. You can use the search bar to filter this list and to speed up finding the right data to use for analysis. Enter a few characters in the search field and the software will filter and display the filtered data in the Available Data box.
2
Available Data: The available data box contains the list of data available for analysis. If your workbook does not have any data in tabular format, this box will display "No Data Found". The information displayed in this box includes the row number, whether the data is Numeric (N) or Text (T), and the name of the column variable. Note that the software displays data from all the tables in the current workbook. Even though data within the same table have unique column names, columns across different tables can have similar names. Hence, it is important that you not only specify the column name but also the table name.
3
Add or View Data: Click on this button either to add more data into your workbook for analysis or to view more details about the data listed in the available data box. When you click on this button, it opens up the Data Editor dialog box where you can import more data into your workbook, or you can switch from the list view to a table view to see the individual data values for each column.
4
Required Data: The code for the required data specifies what data can be specified for that box. An example code is N: 2-4. If the code starts with an N, then you will need to select only numeric columns. If the code starts with a T, then you can select both numeric and text columns. The numbers to the right of the colon specify the min-max values. For example, if the min-max values are 2-4, then you need to select a minimum of 2 columns of data and a maximum of 4 columns of data in this box. If the minimum value is 0, then no data is required to be specified for this box.
5
Select Button: Click on this button to select the data for analysis. Any data you select for the analysis is moved to the right. To select a column, click on the columns in the Available Databox to highlight them and then click on the Select Button. A second method to select the data is to double click on the columns in the list of Available Data. Finally, you can also drag and drop the columns you are interested in by holding down the select columns using your left mouse key and dragging and dropping them in one of the boxes on the right.
6
Selected Data: If the right amount of data columns has been specified, the list box header will be displayed in the black color. If sufficient data has not been specified, then the list box header will be displayed in the red color. Note that you can double-click on any of the columns in this box to remove them from the box.
7
View Selection: Click on this button to view the data you have specified for this analysis. The data can be viewed either in the tablular format or you can view a graphical summary of the data selected.

Program

If you click on the Program button, the software will display the program code - an example screenshot is shown in the figure below. Pre-Process Inputs 3
1
R Program: You can view the R program that will be executed here. This program is usually automatically generated from the options you have specified in the setup earlier. This is the program that will be executed by the R program to generate analysis outputs. If you like, you can edit this program.
2
Auto Mode: If the radio button is selected as Auto, then the software will automatically update this code based on any changes you make in the input dialog box. We recommend that you use this option to generate the R program so that all your input settings are used to generate analysis results.
3
Manual Mode: If you use the Manual option, then you will be allowed to edit the R program before the program is executed. Make sure that you specify a syntactically correct program; otherwise, the R program may report errors.

Verify

If you click on the Verify button, the software will perform some checks on the data you have entered. A sample screenshot of the data is shown in this figure. Pre-Process Inputs 4
1
Verify Checks: The objective of this analysis as well as any checks that are performed are listed in this dialog box. For example, the software may check if you have correctly specified the input options and if you have specified the data correctly for analysis.
2
Check Status: The results of the analysis checks are listed here. If the checks are passed, then they are shown as a green-colored checkmark. If the verification checks fail, then they are shown as a red-colored cross. If the verification checks result in a warning, they are shown in the orange color exclamation mark and finally, any checks that are required to be performed by the user are shown as blue info icons.

Outputs

Click on Compute Outputs to generate the outputs from this analysis. A sample screenshot of the worksheet is shown below. XGBoost Example
1
Notes Section: The analysis results section shows a summary of the inputs used to build the XGBoost model, the number of trees that were grown and the number of variables tried at each split. It also provides a summary of the mean square residual errors for continuous data and accuracy and Kappa values for categorical data.
2
Graph Section: The first graph shows the importance of each variable in model development. The second graph either shows a histogram of the errors or the confusion matrix. The prediction column shows the results of prediction using this model and the conclusion shows the mean square error or the accuracy values for the training and test data sets depending on if the results are continuous or categorical.

Notes

Here are a few notes regarding this analysis:
  • If you want to review the R program that was used to generate this output, go to the %TMP%/Sigma Magic/ folder and look for the file smRScript.R after you execute the program. Note that this folder also contains other files such as error or warning output, text, and graphical output from running R. However, these files are stored in a temporary location and are not persistent between sessions.

Examples

Following examples can be found in the Examples folder within the software.
  • Use the XGBoost model to estimate the number of gears in a car given the rest of the features (XGBoost 1.xlsm).

References

For more information on this topic, please refer to the following articles. Do note that if any external links are mentioned below, they are for reference purposes only.