NPS in DataRobot¶
Has anyone implemented an NPS solution in DataRobot?
Hello NLP team. I was wondering if anyone has implemented an NPS (net promoter scores) solution in DataRobot. I have a customer that wants to use a multilabel project that not only labels Good, Bad, Neutral, but also tags the cause of the bad/good review. Say for example someone responds with:
“I loved the product but the service was terrible.”
How can we use DataRobot to tell us that that is contains both a good and bad comment, the bad comment is assigned to “service” and the good is assigned to “product”?
Multilabel with classes like
good_service, etc ?
I would use the raw 1-10 score as a target. A good model should be able to learn something like:
Target: 7 Text: “I loved the product but the service was terrible.”
coefficients: * intercept: +5.0 * "loved the product": +4.0 * "service was terrible": -1.0
prediction: 5 + 4 - 1 = 7
Don't aggregate the data to get a NPS and then try to label and bin. Just use the raw survey scores directly and look at how the words/phrases in the word cloud drive the score up and down. Multilabel (and multiclass) both feel like they are overcomplicating the problem—great for other things but you don't need it here!
“don’t aggregate the data to get a NPS and then try to label and bin”
Can you elaborate a bit more on this ^^ sentence
So a "net promoter score" is an aggregate number. It doesn't exist in individual surveys. This is a great article on it.
Typically, a net promotor score survey has 2 questions:
- On a scale of 1-10, how likely are you to recommend this product to a friend?
- Free form text: Why?
gotcha, i see what you mean.
So lets say you get 100 surveys, and the distribution of scores is something like:
- 1: 1 respondent
- 2: 3 respondents
- 3: 5 respondents
- 4: 7 respondents
- 5: 15 respondents
- 6: 25 respondents
- 7: 15 respondents
- 8: 15 respondents
- 9: 10 respondents
- 10: 4 respondents
And the net promotor methodology bins these up:
- Detractors: 1-6
- Passives: 7-8
- Promotors: 9-10
So in our case we have:
- Detractors: 1+3+5+7+15+25 = 56
- Passives: 15+15 = 30
- Promotors: 10+4 = 14
Converting to %'s we get:
- Detractors: 56%
- Passives: 30%
- Promotors: 14%
The net promotor score is
(Promotors %) - (Detractors %), or in this case 14-56.
So the NPS for this survey is -42, which is bad.
Ok, now why is it bad? Well, you hire some consultant to read the survey and tell you, or you can use DataRobot to read the surveys instead!
The concept of a net promotor score at the level doesn't apply—you can't look at one person and compute their NPS. NPS is a property of a group of users. At the user level you could bin people up into multiclass "detractor" "passive" and "promotor" but you lose information, particularly in the detractor class.
I personally think a 6 is really different from a 1. A 1 hates your product, and a 6 is pretty much a passive.
So it's useful to build an individual-level model where the target is the direct, raw score from 1-10, and then the predictor is the text of the response. And as I pointed out above, DataRobot's word cloud and coefficients will tell you which pieces of text increase the users score and which pieces decrease their score, adding up to a total predicted score for a user based on what they said.
You can also use text prediction explanations to look at individual reviews.
Oh that’s right! That will give you word level positive/negative/neutral for each review.
thanks Robot 2 and Robot 3! This is all great information. I’ll see what we can come up with, but I’d definitely like to leverage the text prediction explanations for this one.