Assessing relevancy of Marketing Mix Model through Population Stability Index (PSI)
Know whether your MMM model needs updating or not through PSI
After a lot of hard work you have built a great MMM model. The model is now powering saturation / reach frequency curves, scenario planning and Budget optimization.
The client is happy using all of these.
But the client asks you the following question -
"How long can I continue to use the MMM models, When should we update them ?"
At Aryma Labs, we liken MMM models to movie making. Just like a series of snapshots is put together to make a movie, a series of model updates gives you the 'motion picture' of your market.
At Aryma Labs, we liken MMM models to movie making. Just like a series of snapshots is put together to make a movie, a series of model updates gives you the 'motion picture' of your market.
So it is pretty clear that frequent model updates are required. But how to know exactly when to update the model?
The answer lies in Population Stability Index.
Population Stability Index
Population Stability Index (PSI) is a statistic that tells you how much your population (data) has shifted over time or between any interval of time. An excellent paper on PSI by Dr. Bilal Yurdakul can be found in the comments.
PSI is a close cousin to KL Divergence. It is a symmetrised KL Divergence. In my last post (link in resources section), I wrote about how we use KL divergence as a MMM calibration metric.
PSI and KL Divergence answer different questions
PSI informs you about data shift.
KL Divergence informs you about model shift.
How we leverage PSI in MMM
So once the MMM model is built and put in production (we do this in our product ArymaEdge), we compare the data on which the model was built with the latest data given by the client.
The cadence of providing new data depends on the client. It could be weekly or monthly.
We then compute the PSI between the old data and the new data.
There are some thumb rules as follows:
▪ PSI <0.10 : Little shift, no action required
▪ PSI between 0.10 - 0.25 : Moderate shift. Investigate data shift, investigate variable shift through CSI (Characteristic Stability Index)
▪ PSI > 0.25 : Significant shift. Rebuild model
So using these rules, we take a decision on whether the MMM model needs to be updated or not.
PSI as MMM relevancy metric
Marketing environment is always dynamic. One must make sure that the business decisions being made are relevant to current circumstances. PSI helps in evaluating the relevancy of your MMM model.
In summary:
Use KL Divergence to assess model fit.
Use PSI to assess MMM model relevancy.
Resources Section:
Dr. Bilal's paper: https://scholarworks.wmich.edu/cgi/viewcontent.cgi?article=4249&context=dissertations
https://stats.stackexchange.com/questions/219822/what-is-the-intuition-behind-the-population-stability-index
Thanks for reading.
For consulting and help with MMM implementation, Click here
Stay tuned for more articles on MMM.