Continuing on from the discussion about needing beefier computation, sometimes throwing more money and power at the problem isn’t the answer. Well it rarely is, and even though I prefer to take time to find more optimal and efficient solutions, in the real world I only have so much time and resources to get a problem solved.

Following a procedure

Coming from a more classical statistics background, my view of the more recent advances in machine learning is that I am always a bit behind in the newer advancements, especially ones which require a more technical background to approach. It is always best to start off with more basic (and fast) algorithms to establish a baseline before diving deep into models that take hours to train. Following a routine of basic linear regression first, and then basic tree models, and then more complicated mixtures allows for easy to follow analysis. This way, if you have a linear model that is quick to train that performs better than more advanced models, you won’t be wasting time.

The other benefit of trying out the older and more understood models is that it helps to develop an intuition into your data. The dark art of model building and choosing what fits best comes from practice and doing it over and over again. Which brings me to the main topic of LightGBM.

A new challenger

XGBoost came into the scene awhile back and drew many followers through winning Kaggle competitions. Without going too deep into the technical bits of gradient boosted frameworks, XGboost makes usage of trees that run in parallel to perform decisions.

With XGBoost, I used it within the h2o machine learning engine, which make it easy to set up and test multiple models. But I quickly found that it became very memory intensive, so I looked to other algorithms.

LightGBM offers better memory management, and a faster algorithm due to the “pruning of leaves” to manage the number and depth of trees that are grown.

R and LightGBM

Compiler set up

    # for linux
    sudo apt-get install cmake
    # for os x
    brew install cmake
    brew install gcc --without-multilib

Then to install LightGBM as a R-package “` git clone -recursive https://github.com/Microsoft/LightGBM cd LightGBM/R-package

export CXX=g++-7 CC=gcc-7 # for OSX

R CMD INSTALL -build . -no-multiarch “`

Organising the dataset

As usual, each algorithm wants specific formats for the input data, usually in the form of a matrix (sparse or not).

    categoricals = c('V1', 'V2', 'V3')
    
    dtrain <- lgb.Dataset((data.matrix(train[,x_features, with=FALSE])),
                         categorical_feature = categoricals,
                         label = train$target)

With your dataset partitioned into train and test dataframes/datatables, the simple way is to convert into a matrix. This will create a lgb dataset object with the training data.

Building the model

    system.time(
    model <- lgb.train(params,
                       dtrain,
                       nrounds = 5500,
                       num_leaves = 50,
                       learning_rate = 0.1,
                       max_bin = 550,
                       verbose = 0
                       )
    )
         user    system   elapsed
    24761.032     5.844  2107.373
    
    preds <- predict(model, data.matrix(test[,x_features, with=FALSE]))

I have included the computation numbers here as a comparison for other models. The training data is 33m rows, which ends up being the largest dataset I’ve ever worked out. The size itself was the issue for me, which lead me to use LightGBM.

Comparison

A semi-tuned gbm model gave me RMSE results of around 4, which is the number I’ve been working towards. The issue arose when I wanted to tune xgb models with h2o, which took around 2 hours to train a model. Of course, there will be a factor of me not knowing what I’m really doing. Being limited in computational power, I didn’t know how to properly set up a distributed xgb model with h2o, which I’m not sure is even possible with the current versions. This lead me to not be able to properly figure out what the optimal parameters for the model are.

To me, LightGBM straight out of the box is easier to set up, and iterate. Which makes it easier do tuning and iteration of the model. I can train models on my laptop with 8gb of RAM, with 2000 iteration rounds. This I could’ve never done with h2o and xgb.

Of course, there is still a lot of other tuning and investigation that needs to be done, but that’s for next time.