Target transform¶

Robot 1

How does transforming your target (log(target), target^2, etc.) help ML models and when should you use each?

Robot 2

This is not going to be an ELI5 answer, but here is a reason you log transform your target.

TL;DR: When you run a linear regression, you try to approximate your response variable (e.g., target) by drawing a line through a bunch of points and you want the line to be as close to the response variable for those points as possible. Sometimes though, those points don’t follow a straight line. They might follow a curvy line, in which case a simple line through the points doesn’t approximate your target very well. In some of those scenarios, you can log transform your target to make the relationship between those points and your response variable more like a straight line. The following images from Code Academy show the bad:

And the good:

Robot 3

This is specific to linear models that fit a straight line. For tree-based models like XGBoost, you don’t need to transform your target (or any other variable)!

Robot 4

Yeah—log-transforming specifically was born out of trying to better meet the assumptions in linear regression (when they're violated). I have seen some cases where log-transformations can help from the predictive performance standpoint. (AKA when your target has a really long tail, log-transforming makes this tail smaller and this sometimes helps models understand the target better.)

Robot 5

Honey, I shrunk the target variable.

Robot 3

Robot 4 you can also use a log link loss function (such as poisson loss) on both XGBoost and many linear regression solvers. I prefer that over the log transform, as the log transform biases the predicted mean, which makes your lift charts look funny on the scale of the original target.

But it really depends on the problem and what you’re trying to model!

Robot 4

Or use inverse hyperbolic sine amirite? 😂

Robot 1

Thanks all! Very helpful. Robot 4 hyperbolic sine amirite is the name of my favorite metal band.