Search

Previous research has highlighted the need for standardized experimental settings in building control studies, and recent work on real-world reinforcement learning (RL) has introduced challenges that remain largely unexplored. Building on these insights, we propose a curated set of nine real-world challenges specifically aimed at RL control in grid-interactive buildings (GIBs). We suggest that studies in this field adopt this challenge-based perspective alongside standardized environments to ensure repeatable results. While advanced control strategies like model predictive control (MPC) and RL each offer unique advantages, both face limitations that hinder real-world deployment, and direct, unbiased comparisons between them are scarce. By framing evaluations around concrete challenges, it becomes possible to systematically test controller performance under diverse conditions and facilitate fairer comparisons. To demonstrate this approach, we focus on the offline learning challenge using CityLearn, an OpenAI Gym environment tailored for RL in demand response scenarios, in which energy storage across a collection of heterogeneous buildings is manipulated to shape the aggregated electricity demand profile. Using this setup, we examine how different levels of domain knowledge and RL algorithm complexity influence outcomes, revealing that the sequence of operations (SOOs) generated by a rule-based controller (RBC)—which provides training logs for offline RL agents—directly affects performance on four energy flexibility metrics. Prolonged offline training using an optimized RBC yields sustained performance gains, whereas training on logs from a simplified RBC can deteriorate agent effectiveness over time. Additionally, sharing information between agents does not appear to affect results significantly. Overall, we call for stronger interdisciplinary collaboration to tackle these real-world challenges and unlock the full potential of GIB control systems.