Data science¶

• General data science
• Types of learning

General data science¶

As you can imagine, there are a lot of general questions that come through.

What is a sidecar model?¶

ELI5: Imagine you have a motorcycle driver who is navigating just by what they see on the road. Sometimes, the driver gets a bit confused and needs some help to make the right decisions. That’s where the sidecar comes in. The sidecar is like having a passenger sitting beside the motorcycle driver. This passenger has a different perspective (or maybe even a map) and can give advice or guidance to the driver. They might notice things that the driver doesn’t see, like a pothole, a shortcut, or a prompt injection attack.

I would also probably consider guard models to be a kind of sidecar model, but idk if we refer to them that way internally.

With DataRobot's hosted custom metrics, the sidecar model is a different model from the LLM that’s serving back the actual responses, but it can make a determination about whether the prompt was toxic, an injection attack, etc.

What are rating tables and GAMs?¶

A rating table is a ready-made set of rules you can apply to insurance policy pricing, like, "if driving experience and number of accidents is in this range, set this price."

A GAM model (generalized additive models) is interpretable by an actuary because it models things like, "if you have this feature, add $100; if you have this, add another $50."

GAMs allow you to automatically learn ranges for the rating tables.

Learn more about rating tables.

Loss reserve vs. loss cost modeling¶

In insurance, loss reserving is estimating the ultimate costs of policies that you've already sold (regardless of what price you charged). If you sold 1000 policies this year, at the end of the year lets say you see that there have been 50 claims reported and only $40k has been paid. They estimate that when they look back 50 or 100 years from now, they'll have paid out a total of $95k, so they set aside an additional $55k of "loss reserve". Loss reserves are by far the biggest liability on an insurer's balance sheet. A multi-billion dollar insurer will have hundreds of millions if not billions of dollars worth of reserves on their balance sheet. Those reserves are very much dependent on predictions.

ELI5: You just got paid and have $1000 in the bank, but in 10 days your $800 mortgage payment is due. If you spend your $1000, you won't be able to pay your mortgage, so you put aside $800 as a reserve to pay the future bill.

Algorithm vs. model¶

ELI5: An example model for sandwiches: a sandwich is a savory filling (such as pastrami, a portobello mushroom, or a sausage) and optional extras (lettuce, cheese, mayo, etc.) surrounded by a carbohydrate (bread). This model allows you to describe foods simply (you can classify all foods as "sandwich" or "not sandwich"), and allows you to predict new sets of ingredients to make a sandwich.

Robot 1

Not the "is it a sandwich?" debate!!

An algorithm for making a sandwich would consist of a set of instructions:

1. Slice two pieces of bread from a loaf.
2. Spread chunky peanut butter on one side of one slice of bread.
3. Spread raspberry jam on one side of the other slice.
4. Place one slice of bread on top of the other so that the sides with the peanut butter and jam are facing each other.

PCA and K-Means clustering¶

Robot 1

What is the impact of principal component analysis (PCA) on K-Means clustering?

Hi team, a customer is asking how exactly a PCA > k-means is being used during modeling. I see that we create a CLUSTER_ID feature in the transformed dataset and I am assuming that is from the k-means. My question is, if we are creating this feature, why aren't we tracking it in, for example, feature impact?

Robot 2

Feature impact operates on the level of dataset features, not derived features. If we have one-hot encoding for categorical feature CAT1—we also calculate feature impact of just CAT1, not CAT1-Value1, CAT1-Value2,...

Permutation of original features would also produce permutation of KMeans results—so if those are important for the modeling result, its impact will be assigned to the original columns.

Robot 3

Some blueprints use the one-hot-encoded cluster ID as features, and other blueprints use the cluster probabilities as features.

If you wish to assess the impact of the kmeans step on the outcome of the model, delete the kmeans branch in composable ML and use the Leaderboard to assess how the model changed.

As Robot 2 says, feature impact operates on the RAW data and is inclusive of both the preprocessing AND the modeling.

What does monotonic mean?¶

Examples

To normalize or not to normalize, that is the question:

Robot 1

1. When we apply monotonic Increasing/Decreasing constraints to attributes, is DataRobot doing some kind of normalization (capping amd flooring, binning etc.)?
2. When we apply try only monotonic models, will it try GBM, XGBOOST, RF etc.?

Robot 2

No normalization that I know of and just xgboost and gam. You don't really need to normalize data for xgboost models.

Robot 3

For docs you can check out how to configure feature constraints and then the workflow to build them.

What is a ridge regressor?¶

There are two kinds of penalized regression—one kind of penalty makes the model keep all the features but spend less on the unimportant features and more on the important ones. This is Ridge. The other kind of penalty makes the model leave some unimportant variable completely out of the model. This is called Lasso.

Calibration for XGBoost probabilities¶

Robot 1

Customer asked a stumper for me. They mentioned that XGBoost probabilities for binary classifiers can sometimes be off base and need to be “calibrated”. I’ll admit this is over my head, but is this just the XGBoost loss function?

Robot 2

Is it important for the probabilities to be properly calibrated, or good to have? What's the use case? CC @Robot 3, who almost certainly knows the technical answer about the loss function.

Robot 1

It’s 90/10 unbalanced and is likely some sort of medical device failure or bad outcome.

Robot 3

We use Logloss for our XGBoost models, which usually leads to pretty well calibrated models.

If we used a different loss function, we would need to calibrate (but we don’t).

We’ve investigated ourselves and determined that using Logloss is a good solution.

Robot 2

Should we have a DR link that explains "we've thought about this, and here's our answer"?

Robot 1

How about:

There was a great question from this morning on the calibration of the probabilities for XGBoost models. I discussed this with some of the data scientists who work on core modeling. Based on their research on this issue, Using the LogLoss loss function generally produces well-calibrated probabilities and this is the default function for unbalanced binary classification datasets.

For other optimization metrics, calibration may be necessary and is not done by DataRobot at this time.

Robot 4

If they wanted to, they could add a calibration step in the blueprint like here:

Robot 5

Maybe worth noting that another quick way to check calibration is by looking at the lift chart. Not the 100% answer but still helps.

What are tuning parameters and hyperparameters?¶

Tuning parameters and hyperparameters are like knobs and dials you can adjust to make a model perform differently. DataRobot automates this process to make a model fit data better.

NPS in DataRobot¶

Robot 1

Hello NLP team. I was wondering if anyone has implemented an NPS (net promoter scores) solution in DataRobot. I have a customer that wants to use a multilabel project that not only labels Good, Bad, Neutral, but also tags the cause of the bad/good review. Say for example someone responds with:

“I loved the product but the service was terrible.”

How can we use DataRobot to tell us that that is contains both a good and bad comment, the bad comment is assigned to “service” and the good is assigned to “product”?

Robot 2

Multilabel with classes like good_product, bad_product , good_service, etc ?

Robot 3

I would use the raw 1-10 score as a target. A good model should be able to learn something like:

Target: 7 Text: “I loved the product but the service was terrible.”

coefficients: * intercept: +5.0 * "loved the product": +4.0 * "service was terrible": -1.0

prediction: 5 + 4 - 1 = 7

Robot 3

Don't aggregate the data to get a NPS and then try to label and bin. Just use the raw survey scores directly and look at how the words/phrases in the word cloud drive the score up and down. Multilabel (and multiclass) both feel like they are overcomplicating the problem—great for other things but you don't need it here!

Robot 1

“don’t aggregate the data to get a NPS and then try to label and bin”

Can you elaborate a bit more on this ^^ sentence

Robot 3

So a "net promoter score" is an aggregate number. It doesn't exist in individual surveys. This is a great article on it.

Typically, a net promotor score survey has 2 questions:

1. On a scale of 1-10, how likely are you to recommend this product to a friend?
2. Free form text: Why?

Robot 1

Gotcha, I see what you mean.

Robot 3

Robot 3

Ok, now why is it bad? Well, you hire some consultant to read the survey and tell you, or you can use DataRobot to read the surveys instead!

The concept of a net promotor score at the level doesn't apply—you can't look at one person and compute their NPS. NPS is a property of a group of users. At the user level you could bin people up into multiclass "detractor" "passive" and "promotor" but you lose information, particularly in the detractor class.

I personally think a 6 is really different from a 1. A 1 hates your product, and a 6 is pretty much a passive.

So it's useful to build an individual-level model where the target is the direct, raw score from 1-10, and then the predictor is the text of the response. And as I pointed out above, DataRobot's word cloud and coefficients will tell you which pieces of text increase the users score and which pieces decrease their score, adding up to a total predicted score for a user based on what they said.

Robot 2

You can also use text prediction explanations to look at individual reviews.

Robot 3

Oh that’s right! That will give you word level positive/negative/neutral for each review.

Thanks Robot 2 and Robot 3! This is all great information. I’ll see what we can come up with, but I’d definitely like to leverage the text prediction explanations for this one.

Scoring data vs. scoring a model¶

You want to compete in a cooking competition, so you practice different recipes at home. You start with your ingredients (training data), then you try out different recipes on your friend to optimize each of your recipes (training my models). After that, you try out the recipes on some external guests who you trust and are somewhat unbiased (validation), ultimately, choosing the recipe that you will try in the competition. This is the model that you will be using for scoring.

Now you go to the competition where they give you a bunch of ingredients—this is your scoring data (new data that you haven't seen). You want to run these through your recipe and produce a dish for the judges—that is making predictions or scoring using the model.

You could have tried many recipes with the same ingredients—so the same scoring data can be used to generate predictions from different models.

Bias vs. variance¶

You're going to a wine tasting party and are thinking about inviting one of two friends:

• Friend 1: Enjoys all kinds of wine, but may not actually show up (low bias/high variance).
• Friend 2: Only enjoys bad gas station wine, but you can always count on them to show up to things (high bias/low variance).

Best case scenario: You find someone who isn’t picky about wine and is reliable (low bias/low variance). However, this is hard to come by, so you may just try to incentivize Friend 1 to show up or convince Friend 2 to try other wines (hyperparameter tuning). You avoid friends who only drink gas station wine and are unreliable about showing up to things (high bias/high variance).

Log scale vs. linear scale¶

In log scale the values keep multiplying by a fixed factor (1, 10, 100, 1000, 10000). In linear scale the values keep adding up by a fixed amount (1, 2, 3, 4, 5).

Interesting facts:

• In economics and finance, log scale is used because it's much easier to translate to a % change.
• It's possible that a reason log scale exists is because many events in nature are governed by exponential laws rather than linear, but linear is easier to understand and visualize.
• If you have large linear numbers and they make your graph look bad, then you can log the numbers to shrink them and make your graph look prettier.

Offset/exposure with Gamma distributions¶

Robot 1

How does DataRobot treat exposure and offset in model training with the target following a Gamma distribution?

The target is total claim cost while exposure = claim count. So, in DataRobot, one can either set exposure equal to “claim count” or set offset = ln(“claim count”). Should I reasonably expect that both scenarios are mathematically equivalent?

Thanks!

Robot 2

Yes, they are mathematically equivalent. You either multiply by the exposure or add the ln(exposure).

Robot 1

Thanks, that was my impression as well. However, I did an experiment, setting up projects using the two approaches with the same feature list. One project seems to overpredict the target, while the other underpredicts. If they are mathematically equal, what might have caused the discrepancy?

Robot 2

Odd. Are you using the same error metric in both cases?

Robot 1

Yes, both projects used the recommended metric—Gamma Deviance.

Robot 2

Can you manually compare predictions and actuals by downloading the validation or holdout set predictions?

Robot 1

Upon further checking, I see I used the wrong feature name (for the exposure feature) in the project with the exposure setting. After fixing that, predictions from both projects match (by downloading from the Predict tab).

I did notice, however, that the Lift Charts are different.

Robot 2

That is likely a difference in how we calculate offset vs. exposure for Lift. I would encourage making your own Lift Charts in a notebook. Then you could use any method you want for handling weights, offset, and exposure in the Lift Chart.

Robot 3

We do have a great AI Accelerator for customizing lift charts.

Robot 1

Amazing. Thank you!

Target transform¶

Robot 1

How does transforming your target (log(target), target^2, etc.) help ML models and when should you use each?

Robot 2

This is not going to be an ELI5 answer, but here is a reason you log transform your target.

TL;DR: When you run a linear regression, you try to approximate your response variable (e.g., target) by drawing a line through a bunch of points and you want the line to be as close to the response variable for those points as possible. Sometimes though, those points don’t follow a straight line. They might follow a curvy line, in which case a simple line through the points doesn’t approximate your target very well. In some of those scenarios, you can log transform your target to make the relationship between those points and your response variable more like a straight line. The following images from Code Academy show the bad:

And the good:

Robot 3

This is specific to linear models that fit a straight line. For tree-based models like XGBoost, you don’t need to transform your target (or any other variable)!

Robot 4

Yeah—log-transforming specifically was born out of trying to better meet the assumptions in linear regression (when they're violated). I have seen some cases where log-transformations can help from the predictive performance standpoint. (AKA when your target has a really long tail, log-transforming makes this tail smaller and this sometimes helps models understand the target better.)

Robot 5

Honey, I shrunk the target variable.

Robot 3

Robot 4, you can also use a log link loss function (such as poisson loss) on both XGBoost and many linear regression solvers. I prefer that over the log transform, as the log transform biases the predicted mean, which makes your lift charts look funny on the scale of the original target.

But it really depends on the problem and what you’re trying to model.

Robot 4

Or use inverse hyperbolic sine amirite? 😂

Robot 1

Thanks all! Very helpful. Robot 4, hyperbolic sine amirite is the name of my favorite metal band.

What is target encoding?¶

Machine learning models don't understand categorical data, so you need to turn the categories into numbers to be able to do math with them. Some example methods to do target encoding include:

• One-hot encoding is a way to turn categories into numbers by encoding categories as very wide matrices of 0s and 1s. This works well for linear models.

• Target encoding is a different way to turn a categorical into a number by replacing each category with the mean of the target for that category. This method gives a very narrow matrix, as the result is only one column (vs. one column per category with a one-hot encoding).

Although more complicated, you can also try to avoid overfitting while using target encoding—DataRobot's version of this is called credibility encoding.

What is credibility weighting?¶

Credibility weighting is a way of accounting for the certainty of outcomes for data with categorical labels (e.g., what model vehicle you drive).

What is overfitting?¶

You tell Goodreads that you like a bunch of Agatha Christie books, and you want to know if you'd like other murder mysteries. It says “no,” because those other books weren't written by Agatha Christie.

Overfitting is like a bad student who only remembers book facts but does not draw conclusions from them. Any life situation that wasn't specifically mentioned in the book will leave them helpless. But they'll do well on an exam based purely on book facts (that's why you shouldn't score on training data).

What are offsets?¶

Let's say you are a 5-year-old who understands linear models. With linear regression, you find that betas minimize the error, but you may already know some of the betas in advance. So you give the model the value of those betas and ask it to go find the values of the other betas that minimize error. When you give a model an effect that’s known ahead of time, you're giving the model an offset.

What is SVM?¶

Let's say you build houses for good borrowers and bad borrowers on different sides of the street so that the road between them is as wide as possible. When a new person moves to this street, you csn see which side of the road they're on to determine if they're a good borrower or not. SVM learns how to draw this "road" between positive and negative examples.

SVMs are also called “maximum margin” classifiers. You define a road by the center line and the curbs on either side, and then try to find the widest possible road. The curbs on the side on the road are the “support vectors”.

Closely related term: Kernel Trick.

In the original design, SVM could only learn roads that are straight lines, however, kernels are a math trick that allows them to learn curve-shaped roads. Kernels project the points into a higher dimensional space where they are still separated by a linear "road," but in the original space, they are no longer a straight line.

The ingenious part about kernels compared to manually creating polynomial features in logistic regression, is that you don't have to compute those higher-dimensional coordinates beforehand because kernel is always applied to a pair of points and only needs to return a dot product, not the coordinates. This makes it very computationally efficient.

Interesting links:

• Help me understand Support Vector Machines on Stack Exchange.

Particle swarm vs. GridSearch¶

GridSearch takes a fixed amount of time but may not find a good result.

Particle swarm takes an unpredictable, potentially unlimited amount of time, but can find better results.

Keras vs. TensorFlow¶

In DataRobot, “TensorFlow" really means “TensorFlow 0.7” and “Keras” really means “TensorFlow 1.x".

In the past, TensorFlow had many interfaces, most of which were lower level than Keras, and Keras supported multiple backends (e.g., Theano and TensorFlow). However, TensorFlow consolidated these interfaces and Keras now only supports running code with TensorFlow, so as of Tensorflow 2.x, Keras and TensorFlow are effectively one and the same.

Because of this history, upgrading from an older TensorFlow to a new TensorFlow is easier to understand than switching from TensorFlow to Keras.

What is quantile regression loss?¶

Typical loss function: I want to predict a value associated with what happens on average.

Quantile loss: I want to predict a value associated with what happens in a certain percentile.

Why would we do this?

• Intentionally over or under predict: I'm okay with overpredicting supply because stock-outs are very costly.
• Understanding the features that drive behavior in the extremes of your target: Say I wanna know what player-tracking metrics lead to better 3-point shooting in the NBA. I can use quantile loss to surface the metrics related to the best 3-point shooter rather than the average 3-point shooter.

What is F1 score?¶

Let's say you have a medical test (ML model) that determines if a person has a disease. Like many tests, this test is not perfect and can make mistakes (call a healthy person unhealthy or otherwise).

We might care the most about maximizing the % of truly sick people among those our model calls sick (precision), or we might care about maximizing the % of detection of truly sick people in our population (recall).

Unfortunately, tuning towards one metric often makes the other metric worse, especially if the target is imbalanced. Imagine you have 1% of sick people on the planet and your model calls everyone on the planet (100%) sick. Now it has a perfect recall score but a horrible precision score. On the opposite side, you might make the model so conservative that it calls only one person in a billion sick but gets it right. That way it has perfect precision but terrible recall.

F1 score is a metric that considers precision and recall at the same time so that you could achieve balance between the two.

How do you consider precision and recall at the same time? Well, you could just take an average of the two (arithmetic mean), but because precision and recall are ratios with different denominators, arithmetic mean doesn't work that well in this case and a harmonic mean is better. That's exactly what an F1 score is—a harmonic mean between precision and recall.

Interesting links:

• Explanations of Harmonic Mean.

Types of learning¶

Contextual bandits¶

Robot 3

Does this mean contextual bandits are stopping to make a decision of what their next step is when they finish their current one? While a standard multi-armed bandit has a planned path at the start that it follows?

Robot 2

Neither necessarily has planned paths from the beginning. In each setting, one can see how things are going...decide the option just tried isn't as good as expected, make a change to something. The difference is that in contextual bandits, there is some extra information about each decision that you might have that makes it different from the previous decision. I think an example might be helpful

Robot 3

So contextual bandits include context outside of 1. choices being made and 2. the outcome of all previous choices, while non-contextual multi-arm bandits focus internally to those factors only?

Robot 2

Yep. It’s spot on now.

Robot 1

It should be noted that this whole bandits thing is more of a theoretical simplification to reason about reinforcement learning strategies. The real world rarely is as simple. Next best action type use cases can be approximated by a multi-armed bandit. But as soon as you know some information about your users, the actions, the current situation, the recent past...sure it's now a contextual bandit, but it's also just as well, and in many people's opinion better, to just use supervised ML.

Reinforcement learning¶

Reinforcement learning is about how you approach the definition of the problem and the process of learning (which requires and environment able to communicate the reward signal).

Interesting facts

• DataRobot offers an AI Accelerator on reinforcement learning that shows a basic form that doesn't require a deep understanding of neural networks or advanced mathematics.

• Reinforcement learning works better if you can generate an unlimited amount of training data, like with Doom/Atari, AlphaGo games, and so on. You need to emulate the training environment so the model can learn its mechanics by trying different approaches a gazillion times.

• A good reinforcement learning framework is OpenAI Gym. In it you set some goal for your model, put it in some environment, and keep it training until it learns something.

• Tasks that humans normally consider "easy" are actually some of the hardest problems to solve. It's part of why robotics is currently behind machine learning. It is significantly harder to learn how to stand up or walk or move smoothly than it is to perform a supervised multiclass prediction with 25 million rows and 200 features.

Deep learning¶

Imagine your grandma Dot forgot her chicken matzo ball soup recipe. You want to try to replicate it, so you get your family together and make them chicken matzo ball soup.

It’s not even close to what grandma Dot used to make, but you give it to everyone. Your cousin says “too much salt,” your mom says, “maybe she used more egg in the batter,” and your uncle says, “the carrots are too soft.” So you make another one, and they give you more feedback, and you keep making chicken matzo ball soup over until everyone agrees that it tastes like grandma Dot's.

That’s how a neural network trains—something called backpropagation, where the errors are passed back through the network, and you make small changes to try to get closer to the right answers.

Transfer learning¶

Short version: When you teach someone how to distinguish dogs from cats, the skills that go into that can be useful when distinguishing foxes and wolves.

Example

You are a 5-year old whose parents decided you need to learn tennis, while you are wondering who "Tennis" is.

Scenario 1 is similar to how a regular machine learning algorithm starts learning. With the fear of being punished, it starts looking for a way to learn what is being taught and slowly starts learning stuff from scratch. On the other hand, by using Transfer Learning the same ML algorithm has a much better guidance/starting ground or in other words it is using the same ML algorithm that was trained on a similar data as an initialization point so that it can quickly learn the new data but at a much faster rate and sometimes with better accuracy.

Federated machine learning¶

The idea is that once a central model is built, you can retrain the model to use in different edge devices.