Sponsored Content

It’s hard to keep up with the breakneck pace of advancement in earth system forecasting (let alone integrate it into daily hydropower operations and decision-making). Each month has produced a gasp-inducing paper. Most of the recent investment has focused on AI-based weather forecasting, driven by interest from large technology companies to generate AI domination PR.

Recently, AI heavy hitters have been expanding focus from language models to earth system models. Why? To take an overly simplified view: large language models (LLMs) are trained on a dataset containing humanity’s written works, including the “text of the internet,” which works out to a few petabytes – a lot of data relative to day-to-day computing. But there is no second internet for the LLMs to train on. No stashed-away second Github of open source code. Further progress in LLMs will require breakthroughs outside of simply scaling training data and computation.

Earth system data, on the other hand, is big – really big. We are talking about 1000s of petabytes of historical and real-time data that could potentially be utilized to train massive earth system AI models. 

I wanted to give myself a reason to really stay in the loop on this fast-moving corner of AI, so I started a podcast, called Weathering, to discuss the most interesting papers coming out at the intersection of AI and earth systems. 

Weathering: At the intersection of weather forecasting, technology, and the unknowable. A scattered mix of academic papers, good books, philosophy, and the human relationship with weather. Hosted by Marshall, Marta, and Alden.

Reflecting on all the papers I’ve read this year, combined with the perspective gained from providing operational hydrology forecasts to hydropower users for almost a decade (!) at my company Upstream Tech, I’ve collected my predictions for where weather and water forecasts are going in the next year, and how the hydropower industry can make the most of this fast-evolving space.

Prediction #1: AI forecasts’ malleability and resource efficiency will make them fit-for-purpose

AI weather models have demonstrated steady global accuracy improvements over their numerical weather prediction (NWP) cousins. They have also demonstrated order of magnitude reductions in resource cost to produce a comparable forecast. This is because the world’s primary NWP models are so complex, they must run on supercomputers.

This means you (yes you!) can run a solid-performing model on your desktop computer. This may seem superfluous, and heaven forbid we get IT involved, but it allows for a level of resilience that in a critical moment, could be quite powerful. Think: an air-gapped system in the midst of a cyber attack or internet outage. 

Is there a specific problem that creates millions of dollars of operations and compliance headaches that you wish you could forecast? Particularly in situations that are poorly gauged, AI models can often deliver valuable forecasts with smaller gauging investments and shorter spin up times than building and calibrating conceptual models. It’s possible, and getting easier for organizations without deep ML expertise to adapt frameworks to their specific use case. Water quality forecasting is a particularly ripe target here.  

Prediction #2: Decisions will be made using many forecasts

There is no singular best forecast. There never will be. The world is heterogeneous across space and time. More importantly, different users have different needs. There are so many characteristics for forecasts to specialize: one could optimize for high-frequency issuance, another for very high spatial resolution; there are various temporal resolutions and horizons, output variables, resiliency to missing inputs, etc. 

It sometimes seems that every paper and startup claims to be “the best.” Take these boasts with a grain of salt (unless I make them, in which case…). But these claims are often looking at a global average, or a value function that is meaningful to one user but not another.

The answer? Leverage many forecasts and build solid decision frameworks that take advantage of the fuller inputs.

Adapting each forecast for use is not easy – not even for an organization like mine, full of smart software engineers dealing with weather data on a daily basis. I started a not-for-profit organization dynamical.org to make utilizing weather forecasts easier and more consistent across models. If you aren’t afraid of a bit of code, check it out.

It’s safe to say that the future holds many more earth system forecast models – so to prepare, design decision support models that maximize value based on fuller knowledge of potential futures. Build ensembling functions to combine forecasts in a way that matters to the decision you need to make. Leading organizations will move beyond the single-best-model mindset and focus on how to intelligently incorporate diverse forecasts into their operational workflows.

Prediction #3: Uncertainty and risk will be navigated with probability

In our third episode of Weathering, we dove into next-generation large-ensemble forecasts like ECMWF AIFS ENS and Google DeepMind’s FGN. Ensemble forecasts like GEFS and ECMWF IFS ENS have been producing traces from perturbed initial conditions for decades – and more recently most major AI weather forecasts are ensemble models, or output a full probability distribution (my company’s HydroForecast does both). 

Illustrating the relationship between quantiles and confidence intervals. By shifting focus from the Median to the 95% Exceedance (Quantile 0.05) or 5% Exceedance (Quantile 0.95), operators can define risk-based protocols—such as triggering preventive spills—that account for the “long tail” of potential weather events rather than relying solely on the most likely outcome.

It becomes incredibly powerful then to incorporate uncertainty into decision making. One can define a decision protocol depending on organizational risk tolerance using a specific confidence interval. For example, a dam operator with a low tolerance for downstream flood risk might codify a preventive spill whenever the 90th percentile of the inflow forecast exceeds reservoir capacity, rather than waiting for the mean to cross that threshold. This effectively insures the facility against the ‘long tail’ of a chaotic atmosphere. Prioritizing safety over storage, even when the most likely outcome suggests the dam would hold, is an opportunity to manage the full spectrum of operational risk.

Prediction #4: End-to-end forecasts will lead to the next step-change in weather forecasting skill

The majority of recent AI earth system forecasts that are operationally available use a tidy shortcut: they are trained and evaluated on reanalysis – gridded, continuous modeled weather datasets that assimilate, but do not directly pass along true ground observations. To fully realize the potential for AI, model training must utilize the massive (and massively overwhelming) raw observations in what’s called “end-to-end” modeling.

The product of such training can be specialized per-station end-to-end forecasts with no reliance on numerical models whatsoever. 

 Allen, A., Markou, S., Tebbutt, W. et al. End-to-end data-driven weather prediction. Nature 641, (2025). https://doi.org/10.1038/s41586-025-08897-0

HydroForecast has always been “end-to-end” in that it utilizes raw streamflow observations, satellite imagery, and most recently snowpack survey observations. It uses a wide range of meteorological models ranging from NWPs to AI, but no end-to-end weather forecasts as they are not operationally available. 

The next major advancement in weather forecasting will be driven by the development of end-to-end weather models. This fundamental improvement will, in turn, lead to significantly greater skill in subsequent processes, including hydrology forecasting.

Conclusion

There is a wave of innovation occurring in a field where incremental improvement was the status quo. Organizations ready to take advantage of these new signals in their decision making will be able to more effectively mitigate risk and operate efficiently in an environment of weather extremes, changing grid dynamics, and a growing demand for power.

So much is happening that it’s hard to keep up no matter how involved in the industry you are. The pace of innovation represents a turning point in how we turn earth data into decisions. This flurry of research and development signals an appetite to actually operationalize these AI advancements and I’m excited to see how it continues to improve hydropower decision making. If you find this stuff interesting, tune into the podcast, Weathering, and let me know if there’s a specific trend or research paper you think we should discuss. And always feel free to reach out – [email protected].

Citations:

1. NVIDIA Earth-2 Team, “NVIDIA Earth-2 Powers Regional AI Weather Forecasting in the United Arab Emirates,” NVIDIA Technical Blog (March 19, 2025).
2.  Allen, A., et al., “End-to-end data-driven weather prediction,” Nature, vol. 641 (March 20, 2025). (Introducing Aardvark Weather).
3. Vonich, P. T., and Hakim, G. J., “Testing the Limit of Atmospheric Predictability with a Machine Learning Weather Model,” arXiv preprint arXiv:2504.20238 (April 2025).
4.  Microsoft Research AI for Science, “Aurora: A Foundation Model for the Earth System,” Nature (May 2025).
5.  Google DeepMind, “Skillful joint probabilistic weather forecasting from marginals,” arXiv preprint arXiv:2506.10772 (June 2025).
6. ECMWF, “AIFS ENS becomes operational,” ECMWF Newsletter, no. 185, pp. 20-24 (Operational Launch: July 1, 2025).
7. Google Research (NeuralGCM Team), “Advancing seasonal prediction of tropical cyclone activity with a hybrid AI-physics climate model,” Environmental Research Letters (August 2025).