How to Deliver Predictive Models: Discover, Modeling, Pilot

However, it is easy to pour hundreds of thousands (or millions) of dollars into data endeavors without seeing a return on investment. In this blog post, we have compiled some of Sagepath Reply’s top lessons learned for delivering successful Data Science Predictive Modeling use cases.

Discovery

Keep the Discovery phase of the project efficient and concise. While effort here goes a long way to having everyone happy in the end, it’s important that it’s time bound. Remember, the use case may not be viable for several reasons - not enough data, business value isn’t there, end users won’t adopt the model recommendations, etc. This is totally normal for data science initiatives, but in order to ensure you are positively impacting your bottom line - it’s best to get to this point as quickly as possible so that the team can move on to the next use case (if needed). We have found the following tips helpful:

1. Send questionnaires to subject matter experts before discovery sessions.
2. Get a sample dataset from each potential data source as soon as possible
3. Keep the focus on defining the target variable of the model - it is way too easy to start chasing various requests for insights (or as stakeholders put it “It would be good to know…”). Create a parking lot for all Insight requests, so that you do not distract from the modeling efforts.
4. Validate that there is sufficient value potential by calculating the target ROI

Modeling

The modeling phase of the project is where the rubber hits the road. Once again, it’s important that the modeling phase is time-bound and that the focus remains on an MVP model. Tuning the model can be a never-ending process, which is okay if you have the ROI to justify the resource investment. Sagepath Reply’s recommendation is to focus on an MVP model that can be piloted by the business. Here are some tips that we have found helpful in ensuring an efficient modeling phase:

1. Show sample model outputs to the business, early and often. If you have the wrong target variable or if the model outputs are way off, the business can provide feedback quickly.
2. If the model accuracy seems too good to be true, then it probably is. Once again, tight collaboration with the business and asking the business to validate a batch of results early in the modeling phase can typically help the team realign.
3. There are tradeoffs between model explainability and model accuracy. Ensure your Data Science team selects the candidate model based on the business’ priority.
4. While a model’s accuracy can improve with additional data, it’s important to consider the long-term costs associated with maintaining each additional data feed and whether it’s realistic to have the additional data on an ongoing basis post-pilot.
5. When explaining model coverage to the business, ensure that you show coverage in terms that the business understands. This is also a great opportunity to revisit your ROI estimate to ensure that the model coverage is taken into consideration.

Pilot Phase

Sagepath Reply recommends that every data science use case includes a Pilot Phase. We view the Pilot phase as the gatekeeper before our clients invest into “productionalizing” the model. Here are some ideas that can be used to drive a successful pilot phase:

1. The purpose of the Pilot phase is to ensure that the model outputs can bring value to the business, and to ultimately validate the ROI of the use case. Ensure that the use case can achieve the target ROI that was defined during Discovery
2. For the business to pilot the model, the business must have a way to interact with the model. Having the model outputs live in some database, or on a data scientist’s computer, or making them only accessible via API will not cut it! If the model requires input parameters, then there needs to be a mechanism for the business to provide inputs and view model outputs.
3. If the pilot phase is not successful, spend time determining the reason for failure. This retrospective will not only help you with your current Data Science use case, but likely, also your next use case as well. Getting access to new data sources, changing the target variable, and redefining how the business will use the model outputs may feel like starting all over, but this is exactly the reason why we have the pilot phase.
4. Once the pilot phase is successful, it's time to “productionalize” the model, roll it out, realize that ROI, and move on to the next Data Science use case. Automate as much as possible so that your Data Science resources don’t get tied up in maintenance - we need those resources to focus on making your next use case successful!

About Sagepath Reply

It takes more than a Data Science team to successfully deliver Data Science use cases. Sagepath Reply is a full-service digital agency who can help in many aspects of use case delivery:

• We have a very strong strategy team who can lead discovery, refine the use case, estimate the value potential, design the future state process (on how predictive analytics will be consumed by the business), and put together effort and timeline estimates.
• We have an engagement team (strong business analysts and project managers) who can drive the data wrangling, feature engineering, model development and tuning phase. Our engagement team can also work with the business to execute the pilot and lead the operations-phase.
• We have a creative team (User Experience and Visual Designers) who can help put together preliminary concepts (to help get executive buy-in for the use case) but also design the end-user interfaces for the predictions.
• We have a development/QA team who can do any of the ETL work needed to feed the model data, integrate with the model APIs, develop any post-processing logic needed before the predictions are integrated into the business, etc.