Transform forecasting insights¶

概要¶

What it is¶

In many industries, forecasts are still often created in spreadsheets for the sake of transparency and ease of forecast model interpretation. But these spreadsheet forecasts require a lot of manual effort, are resistant to backtesting, and are often difficult for non-technical stakeholders to understand.

For these reasons, many organizations are already utilizing machine learning for time series forecasting. However, one of the problems with these predictive models persists. It still may be hard for non-technical or business stakeholders without a quantitative background to interpret these insights, even with advanced explainability that predictive AI can provide.

Generative AI can bridge the gap by conveying predictions and methodologies for non-technical stakeholders, elevating visibility of this data and the underlying data science work across the organization. The solution would process the contextual information, as well as quantitative prediction insights, explaining the key drivers of these forecasts in human language and even learning to interpret and explain the underlying dynamics of the market at hand based on the data. This creates a seemingly all-knowing human-like assistant that’s able to back its decisions with the highest degree of quantitative data possible.

仕組み¶

For this solution, prediction explanations (a quantitative indicator of the effect variables have on the predictions) from the forecasting model are fed to the generative AI model. Post-processed prediction data with prediction explanations for every time series is ingested and stored in a .csv format. Then it’s converted to a string when injected into the LLM prompt. But could easily also be stored in a database table and converted to a string later.

Organizations may need to post-process the data and do aggregations up to the series-level in order to make it easier for the LLM to understand the prediction explanations, as the individual row level of predictions might be too granular, the series-level is easier to understand and work with. Since the generative AI model and its explanations are only as powerful as the underlying forecast, it’s important to use as many features in the underlying forecasting model as possible. Once the data is fed to the LLM, it summarizes prediction explanations into powerful narratives through an intricate prompting strategy.

User experience¶

The user can interact with the model through a number of ways, depending on how the solution is set up. It can be a standalone application that has access to different forecasting reports from the predictive model, with all of the data already pre-processed for the LLM. This application can also include prompting templates for the user to choose from and modify if necessary since, as you can see above, the prompting strategy can be complicated, depending on the forecasting needs.

The organization may also choose to obfuscate some of the complicated prompting details by offering the user the ability to select necessary prompt elements via a dropdown, like the specific geos they might be interested in. This will then be automatically added to the final prompt. Once they’ve set up their prompt, they run the application and receive the final report within the application (text field, .pdf, or other formats).

Why it might benefit your business¶

Extremely powerful forecasting models built with predictive AI and backed by transparent explanations built with generative AI are extremely easy to understand. This improved decision-making processes by delivering reliable and understandable forecasts, which can have a multiplying positive effect on long-term business decisions, like investment choices and resource allocation. Getting the powerful predictive AI insights into the hands of consumers who otherwise would have been able to make decisions based on them can become a force multiplier for an organization.

Such a comprehensive solution also Increases efficiency and productivity of analytical teams by augmenting and automating forecasting processes. These teams spend a lot of time interpreting the data, but a significant investment is also made in storytelling to explain these findings to decision makers. Generative AI can simplify this process, while simultaneously improving the robustness of insights.

As a unified generative and predictive AI workflow, this can be a visible competitive advantage through improved velocity and accuracy of forecasting insights, as well as their transparency.

What You Need To Implement This Use Case Successfully

• A robust time series forecasting solution or framework that is able to provide context for its findings, “explaining” each prediction, row-by-row.
• An elaborate post-processing pipeline for getting the predictive insights into the LLM workflow.
• A solid prompting strategy that allows to shape the LLMs outputs in a digestible and useful manner.

Potential risks¶

Risks associated with the generated output:

• Potential toxicity or readability issues associated with the prompt, as well as any potential cost implications, based on the scale of the user base and the complexity of inputs and outputs.

• Data quality issues on the predictive side of the workflow, such as inaccurate or incomplete data, which can impact the accuracy and reliability of the predictions and lead to downstream effects for generative AI outputs.

Baseline mitigation tactics¶

• Custom metrics monitoring for toxicity, readability (Flesch Reading Score), as well as informative/ uninformative responses to ensure that responses are appropriate. Additional tokens/ cost monitoring ensures the financial viability of the solution.

• Guard models that would prevent unwanted or unrelated outputs and ensure that the final answers don’t include any hallucinated answers by the LLM

• Extensive testing of the LLM, its parameters, like the system prompts.

• Ongoing monitoring of the underlying predictive model that supplies the forecasts (accuracy, data drift, etc.).