Context is Key: A New Benchmark for Time-Series Forecasting with Textual Information

Forecasting has always been a critical task in decision-making, spanning various domains from finance to healthcare. Traditional methods often rely on historical numerical data, but these alone can fall short when it comes to providing the full context needed for accurate predictions. Human forecasters, however, frequently leverage additional information such as background knowledge and constraints, which are naturally expressed through text. Despite recent advancements in large language models (LLMs), their ability to effectively integrate this textual context remains an open question.

To address this gap, a team of researchers from multiple institutions has introduced "Context is Key" (CiK), a novel benchmark for time-series forecasting that pairs numerical data with diverse types of textual context. This benchmark aims to evaluate and improve the performance of models that can integrate both modalities, ensuring that they are both accurate and accessible to decision-makers with varying levels of technical expertise.

What Changed Technically?

• Multimodal Integration: CiK introduces a new standard for evaluating forecasting models by requiring them to process and understand textual context in addition to numerical data. This is a significant shift from traditional benchmarks, which often focus solely on numerical inputs.
• Diverse Context Types: The benchmark includes various types of textual information, such as background knowledge, constraints, and event descriptions. Each task in CiK requires models to integrate this contextual information to make accurate predictions.
• Benchmark Structure:
  • Tasks: A wide range of forecasting tasks, each requiring the integration of numerical data and textual context.
  • Datasets: Diverse datasets with carefully crafted textual contexts that reflect real-world scenarios.
  • Evaluation Metrics: Standard metrics for time-series forecasting, along with additional measures to assess the model's ability to understand and utilize textual information.

Key Findings

• LLM Performance: The researchers evaluated a variety of models, including statistical models, time series foundation models, and LLM-based forecasters. They found that a simple yet effective LLM prompting method outperformed all other tested methods on the CiK benchmark.
  • Prompting Method: The method involves carefully designing prompts to guide the LLM in understanding and integrating the textual context with the numerical data.
• Surprising Results:
  • Accuracy: LLMs demonstrated surprisingly high accuracy when properly prompted, suggesting that they can effectively leverage textual information for forecasting.
  • Shortcomings: However, the experiments also revealed some critical shortcomings of LLM-based models. For instance, they can struggle with certain types of context, particularly when the text is complex or ambiguous.

Implementation Details

• Model Architectures:
  • Statistical Models: Traditional models like ARIMA and Exponential Smoothing.
  • Time Series Foundation Models: Modern deep learning models designed for time-series forecasting, such as LSTM and Transformer-based architectures.
  • LLM-Based Forecasters: Large language models fine-tuned for forecasting tasks, with a focus on integrating textual context.
• Benchmarks:
  • The CiK benchmark is available for visualization and further research at this link.
  • It includes detailed documentation on the datasets, tasks, and evaluation metrics.

Why It Matters to Practitioners

• Improved Accuracy: By incorporating textual context, models can make more informed and accurate predictions, which is crucial for decision-making in various domains.
• Accessibility: The benchmark promotes the development of models that are not only accurate but also accessible to users with varying levels of technical expertise. This makes advanced forecasting techniques more widely applicable.
• Research Opportunities: CiK provides a standardized platform for researchers to evaluate and improve multimodal forecasting models, driving innovation in the field.

Conclusion

The "Context is Key" benchmark represents a significant step forward in time-series forecasting by emphasizing the importance of integrating textual context. The findings highlight both the potential and the challenges of using LLMs for this task, providing valuable insights for practitioners and researchers alike. As the field continues to evolve, benchmarks like CiK will play a crucial role in advancing the state of the art in multimodal forecasting.