Advanced control strategies have been applied to manage variable power output in standalone microgrids, reducing the risk of frequency instability and power-quality deterioration. Nevertheless, a key difficulty arises: these computationally intensive techniques, when used without stability checks, may fail to cope with the fast dynamics of real-world conditions, particularly under communication delays. This research proposes a versatile real-time solution for frequency regulation that relies on an artificial neural network (ANN) confined to stable operating regions. The proposed method combines stabilizing PID controllers—derived via small-signal analysis—with an automated optimization of ANN weights and reinforcement learning (RL)-guided constraint selection. Initially, stabilizing PID controllers are developed using the stability boundary locus approach and the Mikhailov criterion, with explicit consideration of communication delays. Subsequently, controller parameters are fine-tuned in real time via an automated search for optimal coefficient sets, using a constrained ANN to keep frequency deviations within a predefined parameter space. The framework is further strengthened by an RL-based tuning mechanism trained on an interpolated stability boundary curve, guaranteeing both stability and enhanced performance. This unique integration of ANN, RL, and sophisticated PID tuning represents a significant advancement in addressing frequency control challenges in isolated AC microgrids. Experimental results demonstrate that the proposed approach surpasses conventional techniques thanks to its reduced parameter search domain. Specifically, it achieves greater reductions in both transient and steady-state frequency deviations compared to semi-constrained and unconstrained alternatives. The enhanced performance metrics and stability assessments confirm that the method delivers superior system behavior and robustness under varying operating conditions.
The technical challenges associated with renewable energy sources (RESs) require the development of standards for frequency regulation and properly designed control systems, all aimed at achieving optimal operational performance [1]. These standards are incorporated into microgrid (MG) architectures, which have been developed to tackle three primary issues: reliability, environmental concerns, and economic considerations [2]. In MGs, the main generation units—such as diesel engine generators (DEGs), photovoltaic (PV) arrays, wind turbine generators (WTGs), and fuel cells (FCs)—are deployed and interfaced with the grid on the consumer side, collectively known as the distributed generation (DG) system [3]. Local control of RESs through this system plays a vital role, as it helps minimize transmission losses and improve overall power system reliability [4]. From the utility viewpoint, an MG functions as an independent entity capable of operating in two modes: grid-connected and isolated [5]. In grid-connected mode, the main goals typically involve providing ancillary services to the main grid and optimizing internal energy use, while isolated MGs focus on powering remote loads or enhancing power quality in weak grids via deliberate disconnection [6].
In isolated mode, achieving balance between active and reactive power generation and demand becomes particularly challenging owing to the reduced inertia of microgrids (MGs) relative to the conventional main grid [7]. A common approach to overcoming this issue is to incorporate backup storage solutions, such as electrochemical or mechanical energy storage systems (ESSs), into the MG framework [8]. Without such measures, real-time applications would become impractical due to inherent instability [9]. Precise modeling remains a major obstacle due to the pronounced time-varying and nonlinear behavior of ESSs, especially lithium-ion batteries [10-12]. Meanwhile, ensuring stability requires tight regulation of two key parameters—voltage and frequency—through robust control strategies, which are essential given the intermittent nature of renewable energy sources (RESs) and the unpredictable nature of load fluctuations [13]. To achieve this, three main architectures—centralized, decentralized, and distributed—have been developed for frequency deviation control (FDC) systems within a hierarchical control framework [14]. However, tackling complex multi-objective regulation and optimization problems demands intelligent management systems, energy scheduling mechanisms, and adaptive control units equipped with advanced algorithms [15-18]. To improve robustness, intelligent PID-like controllers have been introduced for various islanded MG types through meta-heuristic optimization techniques, including genetic algorithm (GA) [19], particle swarm optimization (PSO) [20], fuzzy logic [3, 21, 22], black hole algorithm (BHA) [23], and artificial neural networks (ANN) [24-26], all focused on optimally tuning controller parameters for superior performance. For greater robustness, the application of modern adaptive and constrained population extremal optimization, combined with PID controllers, can markedly enhance the effectiveness and reliability of FDC [27, 28], resulting in more stable and efficient management of these intricate systems.
Previous studies and related works have largely neglected the effects of communication time delays on control parameters. In distributed control approaches, transmission delays pose a critical issue that can impair controller performance and threaten system stability in practical deployments [29, 30]. In particular, such delays can modify key dynamic properties, including oscillation modes, critical eigenvalues, and the overall closed-loop system response [31]. Therefore, a thorough stability analysis is vital when incorporating communication time delays into FDC system design, as understanding the interplay between delay and control parameters is indispensable [32]. Incorporating delay terms into the system’s characteristic equation complicates stability analysis by shifting dynamics and altering stability margins, thereby requiring sophisticated methods such as frequency-domain techniques. At present, the dominant methods for assessing MG stability in the presence of time delays are frequency-domain [29, 33] and time-domain [34, 35] analyses, which have evolved alongside advancements in MG modeling [36]. While time-domain approaches (e.g., Lyapunov-based methods) are better suited for controller synthesis because they handle nonlinear, stochastic, and time-varying systems with delays [37], they typically provide only sufficient stability conditions, yielding conservative parameter designs and high computational demands in stiff power systems [38]. Conversely, frequency-domain analysis relies on small-signal stability tools, which are limited to linear or locally linearized nonlinear systems around an equilibrium point, remain valid only near hyperbolic equilibria, and address solely small disturbances [39].
Although earlier research has considered delays in MGs to some extent [40-42], these efforts have not focused on actively enforcing “dynamic adaptation” constraints or on enabling real-time FDC operation in challenging MG settings. Inspired by this research gap and the proven effectiveness of real-time controller parameter tuning reported in [43], we introduce a novel, intelligent, and robust PID-based FDC system tailored for AC MGs subject to communication delays. Accordingly, the primary contribution of this paper is the development of a method that utilizes a constrained ANN-based structure and an RL-based training approach to perform online tuning of controller parameters, incorporating the following key steps to avoid performance degradation in FDC and support its real-world implementation:
• Compute stabilizing PID controllers using the stability boundary locus (SBL) method and Mikhailov criterion, accounting for communication delays.
• Interpolate the stability boundary curve and develop an integrated RL-based technique that leverages this curve to automatically constrain the search space during training, while preserving both stability and performance.
• Construct a fully intelligent controller tuning system by optimizing the combination of ANN weights and constraints produced and chosen via the integrated RL technique, thereby identifying an optimal range of controller gains in light of output variations.
The proposed control strategy delivers several important advancements that substantially improve FDC in isolated AC microgrids:
• Time-Delay-Resistant Algorithm: The algorithm guarantees stable functioning and avoids controller failures or system instability in practical scenarios where delays are unavoidable.
• Stochastic RL and SDDPG Integration: We employ both stochastic reinforcement learning and SDDPG (stochastic deep deterministic policy gradient) to globally optimize the weights of an ANN-based PID controller tuner, increasing algorithmic adaptability.
• Automated Search Space Restriction: The approach automatically narrows the search space through interpolation of the stability boundary curve, facilitating more efficient discovery of tuned parameters during optimization.
• Self-Tuning Capability: A self-tuning mechanism is introduced to boost robustness, speed up disturbance recovery, and strengthen frequency stability, enabling the controller to dynamically adjust parameters in response to variations in system delays or disturbances.
• Integrated Performance and Stability Validation: An applied reward-based search mechanism achieves an effective trade-off between controller stability and performance.
Furthermore, this approach features a sophisticated reward function that strikes an optimal balance between operational stability and overall performance, thereby guiding the reinforcement learning agent’s training predominantly toward secure, stable zones of microgrid behavior. In contrast to earlier hand-crafted and partially constrained techniques [44, 45], the proposed framework fully automates the enforcement of operational limits by interpolating stability boundaries. When compared with standard meta-heuristic optimization algorithms [3, 19-26], the real-time adaptive tuning strategy developed here successfully overcomes the challenges posed by communication latencies. As a result, it not only streamlines neural network training but also significantly improves system robustness, reduces restoration times, and enhances frequency regulation. This is achieved via an autonomous tuning mechanism that narrows the exploration space considerably while remaining fully compatible with the demands of the frequency deviation control (FDC) framework.
Figure 1 shows the configuration of the isolated AC microgrid experiencing communication delays. It integrates wind turbine generators (WTGs), photovoltaic (PV) arrays, a conventional diesel engine generator (DEG), a fuel cell (FC) unit, and an energy storage system (ESS) elements—specifically battery energy storage system (BESS) and flywheel energy storage system (FESS)—all linked via power electronic converters that ensure electrical isolation and appropriate voltage levels. The nominal rated capacities of the main components—RESs (combined WTGs and PV), FC, DEG, and ESS—are 130 kW, 70 kW, 160 kW, and 90 kW, respectively.
The fuel cell system encompasses a DC/AC inverter, fuel supply modules, and connection hardware. Despite its inherent complexity, frequency-domain analysis can be adequately performed with a reduced third-order dynamic model [46]. The DEG primarily serves to deliver a reserved power capacity that functions as spinning reserve within the secondary frequency control (SFC) framework.
To mitigate the negative dynamic consequences arising from the highly variable and uncertain output of renewable energy sources, two practical control configurations for the SFC loop are evaluated. In the first case, only the DEG participates in frequency regulation without additional support. In the second case, the DEG operates in coordination with the FC unit, creating a more capable control structure that effectively counters large frequency excursions and thereby reduces mechanical and electrical stress on both the DEG and the energy storage units.
The energy storage system consists of two distinct technologies: FESS and BESS. The flywheel energy storage system uses a high-speed rotating mass to enable very fast charging and discharging of electrical energy. In contrast, the battery energy storage system stores energy chemically and helps maintain power balance while suppressing frequency oscillations. Accurate state estimation and control of batteries remain difficult tasks, largely because they require precise, high-fidelity models and reliable estimates of internal parameters [47]. Lithium-ion batteries, in particular, exhibit highly nonlinear behavior due to complex internal electrochemical interactions, underscoring the need for advanced modeling approaches to enable reliable performance predictions in practical battery applications [48]. In the present analysis, a simplified linear approximation has been employed for convenience. However, adoption of a higher-fidelity model would improve the fidelity and applicability of the resulting control strategy.
Figure 2 provides a simplified block diagram of the frequency response dynamics for the delayed isolated AC microgrid. Owing to the stochastic and uncontrollable nature of the power injections from PV, WTG, and the connected load, these elements are not included in the FDC controller synthesis. They are instead modeled as an aggregate disturbance input, given by:
(1)
Where 𝑃𝐿, , and 𝑃𝑊 𝑇𝐺 represent the power consumed by the load and generated by the PV and WTG units, respectively, and 𝑃𝑡𝑜𝑡, stands for the total disturbance power. The total load demand signal 𝑃𝑡𝑜𝑡 serves as the reference input for deriving the characteristic equation that will be analyzed in the subsequent section.
Taking into account the system’s inherent damping and inertia, and applying the classical swing equation, the aggregated inertia and damping dynamics of the microgrid can be described as:
(2)
Where 𝐻 is the equivalent inertia constant, 𝛥𝑓 denotes the frequency deviation, 𝐷 is the load damping coefficient, and 𝑠 is the Laplace operator. Moreover, the net power imbalance 𝛥𝑃 is computed as the difference between generation and load, expressed as:
(3)
With 𝑃𝑡𝑜𝑡,𝑔 = 𝑃𝐷𝐸𝐺 + 𝑃𝐹𝐶 + 𝑃𝐵𝐸𝑆𝑆 + 𝑃𝐹𝐸𝑆𝑆, where 𝑃𝐷𝐸𝐺, 𝑃𝐹𝐶, 𝑃𝐵𝐸𝑆𝑆, and 𝑃𝐹𝐸𝑆𝑆 correspond to the power outputs from the diesel generator and the fuel cell, battery, and flywheel storage units, respectively.
Figure 1. General diagram of delayed isolated AC MG.
Building upon this concise introduction to the isolated AC microgrid and its frequency deviation control (FDC) needs in the presence of communication delays, the discussion now advances to the stability analysis. As depicted in Figure 3, the methodology for the intelligent FDC system designates these initial activities as the analysis phase. The subsequent subsections will elaborate on the remaining elements of this phase, followed by the design phase, with particular emphasis on incorporating the ANN-based tuner, reward mechanisms, and the SDDPG algorithm—all aimed at maximizing cumulative rewards and achieving optimal performance. Lastly, the implementation phase will be outlined, including simulation-based validation to evaluate the outcomes and demonstrate superiority over conventional FDC approaches.
Figure 2. Frequency response diagram of the AC MG.
Figure 3. Overall methodology flowchart of the proposed intelligent FDC system.
To perform stability analysis of the AC microgrid and determine appropriate stabilizing controller parameters, the characteristic equation (s), which constitute the denominator of the closed-loop transfer function
(4)
Where 𝐾𝑝, 𝐾𝑖, and denote the proportional, integral, and derivative gains, respectively, is utilized to develop the FDC scheme for regulating the FC and DEG units. By aggregating the frequency-domain blocks shown in Figure 2 and applying appropriate mathematical reductions, the entire system can be represented by a unified transfer function G(s), whose denominator takes the form.
(5)
Where τ represents the communication delay, and P(s) and Q(s) are real-coefficient polynomials of degrees 9 and 7, respectively,
(6)
Remark 1. The inclusion of the additional polynomial term(s) accounting for the delayed elements significantly increases the complexity of the closed-loop characteristic equation. The delay parameter τ introduces an additional dimension to the stability problem, influencing both the dynamic response and the stability margins. This renders the characteristic equation transcendental, complicating analytical treatment and solution procedures and requiring frequency-domain methods. Although the present study assumes a fixed and known delay τ, many practical situations involve uncertain or variable delays that require estimation, further challenging controller synthesis and underscoring the need for continued investigation in this area.
For the required stability investigations, the characteristic Eq. 5 is reformulated by substituting 𝑠 = 𝑗𝜔 and 𝑒−𝑗𝜔𝜏 = cos(𝜔𝜏) − 𝑗 sin(𝜔𝜏), with 𝜔 denoting the angular frequency. This substitution yields a complex polynomial equation structured as
(7)
Where:
And ω̄ = ωτ. Setting both the real and imaginary parts of Eq.7 to zero allows the derivation of two controller parameters expressed in terms of 𝐾𝑑, 𝜏, and the frequency ω. For any fixed value of 𝐾𝑑, the following relations hold:
(8)
By respecting the predefined stability limits given as:
(9)
The corresponding stability boundary locus (SBL) curves can be plotted in the 𝐾𝑝 − plane for different combinations of 𝐾𝑑 and τ.
After concluding the analysis phase, this section elaborates on the development of the proposed real-time ANN-based tuner, as outlined in Figure 3. Emphasis is placed on the seamless integration of the ANN-based tuner, the reward mechanisms, and the SDDPG algorithm, with the overarching goal of optimizing performance and achieving the highest possible cumulative rewards.
Here, we describe the structure and functionality of the proposed real-time ANN-based tuner, shown in Figure 4, operating under stabilizing controller parameters. The presented ANN-based tuner functions as a PID-resembling controller, designed to maintain frequency stability while delivering optimal tuning for an underlying PID controller, specifically within a stability region where 𝐾𝑑 = 0. The stability characteristics of this tuner are governed by the (non)linear properties of the chosen activation function and the trained weight values within the ANN. For instance, smooth nonlinear activation functions like 𝑠𝑖𝑔𝑚𝑜𝑖𝑑, 𝑡𝑎𝑛𝑠𝑖𝑔, 𝑅𝐵𝐹, and similar ones offer inherent disturbance rejection capabilities, unlike purely linear options such as 𝑝𝑢𝑟𝑒𝑙𝑖𝑛 or 𝑅𝑒𝐿𝑈. Given that negative coefficients are generally unsuitable for PID control applications, the activation function must enforce a non-negative output range. The custom activation function introduced in [26] is a modified 𝑡𝑎𝑛𝑠𝑖𝑔 variant, expressed as
(10)
Where x(t) and y(t) represent the input and output of the activation function at time 𝑡, 𝑚𝜇 is a vector of per-neuron constraints that restrict the output signal to the interval [0,𝑚𝜇∕2], and 𝜃𝜇 is the matrix of actor ANN weights. These weights act as adjustable linear amplifiers, scaling the incoming signals and controlling the relative impact of each output on the overall tuning process. To preserve closed-loop stability, the relationships among 𝐾𝑝, 𝐾𝑑, and 𝐾𝑖 must remain confined within the predefined stability region; any violation would compromise system stability. Moreover, the tuner incorporates bounds on both the activation function outputs and the weight values, with performance rigorously verified against the stability boundaries to ensure reliable and stable tuning behavior.
As illustrated in Figure 4, the proposed ANN-based tuner integrates the three core PID components—proportional, integral, and derivative—as inputs to the input layer neurons.
• Proportional action: This component responds directly to the instantaneous frequency deviation. It exhibits reduced sensitivity to disturbances and requires fewer tuning variables, yet it may fall short of fully covering the required controller adjustments.
• Integral action: This element accounts for the accumulated historical frequency deviation, enabling a gradual and smooth tuning process. It contributes to overall stability but results in slower response times, particularly when operating independently.
• Derivative action: Acting as predictive control, this part anticipates erroneous trends in frequency deviation and responds preemptively. Although it can amplify noise and potentially degrade tuning stability, it substantially accelerates the tuner’s dynamic response.
The architecture includes tuning outputs specifically for the proportional and integral gains of the PID controller. Given the specific control requirements and dynamics of the studied microgrid system, the derivative component has been omitted in this implementation. However, it can be readily added to the control framework if needed.
Figure 4. Architecture of the proposed real-time ANN-based tuner.
A novel reward mechanism is introduced, grounded in a stability verification procedure that evaluates the tuner’s outputs, 𝐾𝑝(𝑡) and 𝐾𝑖(𝑡), against an interpolated stability boundary function 𝑔 defined over the stability region in the (𝐾𝑝, 𝐾𝑖)-plane relative to the SBL curve. Concurrently, the objective is to achieve superior control performance; thus, when (𝑡) lies within the stable region, a positive reward is assigned to the controller, whereas excursions beyond this region trigger penalties that increase with the distance from the boundary. Broadly, the reward or penalty is determined by the deviation between the current gain pair 𝐾(𝑡) and the corresponding boundary point 𝐾𝐼(𝑡) = 𝑔(𝐾𝑃(𝑡)), where 𝐾𝑃 and 𝐾𝐼 denote the nearest points on the SBL curve at time 𝑡, and 𝑔 represents the interpolated stability boundary function. Accordingly, the overall reward signal 𝑟𝑡𝑜𝑡 integrates both a stabilizing reward component 𝑟𝑠 and a performance-oriented reward component 𝑟𝑝, which can be formulated as follows:
· To enforce stability, the stabilizing reward component is defined as
(11)
· To maintain high controller performance, a supplementary reward component incorporates criteria derived from the angular function introduced in [49] and expressed as
(12)
Where 𝜙 signifies the error angle and 
denotes the integral of the absolute frequency deviation error.
In conjunction with ANN, we propose the two-stage stochastic RL optimization approach method as follows: In the first stage, the actor ANN at state 𝑠𝑡, which is denoted as (𝑠𝑡 ∣ 𝜃𝜇), is initialized with random weight values where 𝜃𝜇 is an array of non-negative random values that are updated across a specified number of episodes 𝑁. Subsequently, the reward signal (𝑎𝑡,) is observed at each action 𝑎𝑡 = (𝐾𝑝(𝑡),𝐾𝑖(𝑡)) and state 𝑠𝑡 = (𝛥𝑓, , ). Finally, the weights 𝜃𝜇 of the agent with the best total reward will be chosen as
(13)
Where 𝑟𝑡𝑜𝑡_𝑚𝑎𝑥 = argmax(𝑟𝑡(𝑠𝑡,𝑎𝑡)).
The second phase employs the SDDPG algorithm—originally presented in [50] and subsequently adapted for control-oriented applications as detailed in [43, 51] —to refine neural network parameters and achieve superior control performance in the microgrid environment. The core optimization objective is to minimize the discrepancy between the target action-value function and the critic network’s reward estimate. The target action-value function is given by
(14)
Where 𝜃𝜇′ and 𝜃𝑄′ represent the target actor and critic networks, respectively. The critic network’s predicted reward for the state-action pair is denoted as 𝑠𝑡, ∣ 𝜃𝑄. Within this setup, the action 𝑎𝑡 at timestep 𝑡 for microgrid state 𝑠𝑡 is produced by 𝑠 ∣ 𝜃𝜇. The critic loss function 𝐿𝑄 is computed as
(15)
Concurrently, the actor loss function 𝐿𝜇 is minimized to adjust the actor network weights 𝜃𝜇 according to
(16)
This process refines the policy μ(𝑠 | 𝜃𝜇) to enhance overall control efficacy. The SDDPG method is tailored to maximize the total reward 𝑟𝑡𝑜𝑡_𝑚𝑎𝑥 by identifying parameter configurations that deliver the greatest cumulative reward throughout training.
The maximum attainable reward, denoted 𝑟𝑡𝑜𝑡_𝑚𝑎𝑥 = max_i 𝑟𝑖,𝑡𝑜𝑡(𝑠𝑖, 𝑎𝑖), is estimated by estimating rewards via the critic network’s output 𝑄(𝑠𝑡, 𝑎𝑡) (as in Eq. 14) and contrasting these estimates against actual observed rewards. The critic weights 𝜃𝑄 govern the accuracy of reward predictions for given states 𝑠𝑡 and actions 𝑎𝑡, thereby influencing the effectiveness of the actor’s policy evaluation. Weight updates for the actor network (𝜃𝜇) proceed via gradient descent, with periodic evaluation of the ANN-based PID controller following each adjustment. To boost prediction reliability, the optimal future reward estimate is preserved as 𝑄′(𝑠𝑖+1, μ′(𝑠𝑖+1 | 𝜃𝜇′) | 𝜃𝑄′). For a comprehensive description of the procedure, consult Algorithm 1 and the flowchart in Figure 5, which depict the complete reinforcement learning workflow. This phased optimization approach is essential for enabling real-time adaptability in the control system while preserving computational tractability.
Algorithm 1. SDDPG algorithm with stochastic RL optimization.
Remark 2. Although Bayesian optimization performs well when tuning static parameters, reinforcement learning was preferred in this work for its strengths in real-time adaptation, ongoing learning from interactions, and the ability to manage sophisticated, high-dimensional control challenges typical of microgrids. RL is particularly advantageous because it can respond continuously to dynamic variations in operating conditions and effectively balance competing objectives—such as robustness and performance—making it a more suitable choice than Bayesian methods, which tend to remain static and pursue only one primary goal.
Figure 5. Flowchart of the proposed two-stage optimization process.
Remark 3. Constrained versus unconstrained neural networks differ fundamentally in how they trade off stability against operational freedom. The constrained variant relies on a specially designed activation function (Eq. 10) to restrict outputs to a safe, predefined stability interval. By comparison, an unconstrained network gains greater expressive power from its activation function. Still, it lacks a built-in mechanism to enforce stability, which undermines its trustworthiness when dependable operation is essential.
Remark 4. The combination of a multi-stage optimization procedure and a constrained ANN tuner achieves a practical compromise between low computational overhead and genuine real-time capability, thereby satisfying the requirement for online control. The opening RL stage builds an efficient baseline with limited processing demands, after which the SDDPG phase sharpens control quality. In addition, the ANN tuner continuously updates PID gains on the fly without requiring periodic full re-optimization, making the overall scheme well-suited for time-critical microgrid applications.
The microgrid model is defined by the block diagram in Figure 2 and the parameter set provided in Table 1. Applying these values to the previously obtained expression (7) yields the following characteristic equation:
(17)
Substituting 𝑠 = 𝑗𝜔 and replacing the delay term with 𝑒−𝑗𝜔𝜏 = cos(𝜔𝜏) − 𝑗 sin(𝜔𝜏), transforms the characteristic equation Δ(s) into the frequency-domain form given by (7), expressed in terms of angular frequency 𝜔 and delay 𝜏, where
And
With 𝜔̄ = 𝜔𝜏. Fixing 𝐾𝑑 at a constant value allows computation of the remaining parameters via Eq. 8 using the calculated coefficients 𝐴𝑖 and 𝐵𝑖 (for 𝑖 = 1,2,3,4). With the stability limits specified in Eq. 9, the resulting stability boundary locus (SBL) curves are plotted in the 𝐾 𝑝, plane for selected values of 𝐾𝑑 and 𝜏, as presented in Figures 6 and 7. Moreover, Figure 8 displays the time-domain frequency deviation trajectories corresponding to stable, critically stable, and unstable regions (Figure 7(a) for 𝐾𝑑 = 0) across several communication delay values.
Table 1. Parameter values of the AC MG.
Parameter | Value | Parameter | Value |
𝐷 | 0.015 pu/Hz | 𝑇𝑔 | 0.08 s |
2𝐻 | 0.1667 pu s | 𝑇𝑡 | 0.4 s |
𝑇𝐹 | 0.1 s | 𝑇𝐼𝐶 | 0.004 s |
𝑇𝐵 | 0.1 s | 𝑇𝐼𝑁 | 0.04 s |
𝑇𝐹𝐶 | 0.26 s | 𝑅 | 3 Hz/pu |
𝑇𝑊 𝑇𝐺 | 1.5 s | 𝑇𝑃𝑉 | 1.8 s |
Figure 6. SBL curve of PID controller of the AC MG for τ=1.
Figure 7. SBL curve of PID controller of the AC MG for K_d=0.
Figure 8. Frequency deviation in different zones of Figure 8(a) for K_d=0.
Figure 9 shows the Mikhailov stability plots for PID control with K_d = 0 and K_d = 0.
Figure 9. Mikhailov stability plots of PID controller for K_d=0.
The intelligent frequency deviation control (FDC) strategy is implemented and tested in the MATLAB/Simulink environment using the microgrid model from Figure 2. The total power mismatch 𝑃tot,d—equivalent to the net effect of renewable generation minus load demand—is shown in Figure 10. Abrupt changes take place at t = 40, 70, 110, 140, and 180 seconds, combined with persistent random variations across the entire profile.
According to Eq.10, the parameter 𝑚𝜇 imposes an upper bound on the output of each neuron in the ANN, thereby constraining the feasible tuning range. This value can either be set manually to create a partially restricted search region [45] or computed intelligently from the interpolated stability boundary function. When chosen manually, an inappropriate setting may push the PID gains into an unstable zone, resulting in system instability (Figure 11b)). In contrast, Figure 11(a) demonstrates that automatically determining 𝑚𝜇 through the stabilizing reward framework consistently maintains stability. This intelligent method confines adjustments of the proportional and integral gains strictly within the validated stable domain. For comparison, Figures. 11(c) and 11(d) show the outcome of unconstrained tuning, where the computed gains lie outside the stable region and cause unstable frequency behavior.
A performance comparison is provided in Figure 12, Tables 2 and 3, highlighting the fully intelligent, stability-constrained algorithm’s ability to effectively solve the FDC task. Evaluation relies on mean absolute error (MAE) and integral absolute error (IAE) indices, which indicate clear outperformance relative to a classical fixed-gain PID controller (𝐾𝑝 = 3.712, 𝐾𝑖 = 1.391, 𝐾𝑑 = 0), with gains selected via heuristic optimization to provide a fair baseline. The noticeably smaller excursions of frequency deviation 𝛥𝑓 from the nominal value confirm the improved behavior of the constrained ANN-PID approach when 𝐾𝑑 = 0. Although including a small derivative gain yields a modest additional benefit, the improvement is limited and supports the common engineering preference for derivative-free PID structures in noisy microgrid environments to avoid excessive noise amplification.
Figure 10. Aggregate power from RESs and load demand (total disturbance input signal).
Figure 11. Performance validation of the proposed stability-enforcing ANN-based tuner.
To examine the temporal evolution of the controller gains, Figure 13 illustrates the online adjustment of 𝐾𝑝 and 𝐾𝑖 across three distinct apprtuning strategies: fully constrained, semi-constrained, and unconstrained. The semi-constrained and unconstrained cases are drawn from prior studies [43, 44], respectively. In Figure 13(a), the outcomes of the carefully engineered fully intelligent ANN tuner are shown; all constraint bounds remain securely within the stable region, thereby guaranteeing stable operation of the AC microgrid under test. This tuner dynamically tailors each gain to match the prevailing operating conditions and consistently outperforms the semi-constrained, unconstrained, and baseline fixed PID controllers across multiple delay conditions (Tables 2 and 3).
Figure 13(b) shows a case in which the constraint limits were manually assigned inappropriately, resulting in them lying outside the acceptable stability region. Consequently, the initial phase of the simulation exhibits unstable tuning behavior, and the constrained values quickly saturate at their upper limits, remaining fixed thereafter. This behavior arises because the activation function defined in (Eq. 10) imposes a finite output range even for unbounded inputs.
Finally, Figure 13(c) depicts an unstable tuning process resulting from the adoption of an unconstrained activation function, namely the purelin function, expressed as
(18)
where, once again, x(t) and y(t) denote the input and output of the activation function at time t, and 𝜃𝜇 represents the weight matrix of the actor ANN. All simulations discussed here were conducted with a communication delay of 𝜏 = 2 s. As shown in Tables 2 and 3, the intelligently constrained tuning strategy maintains strong performance even when subjected to delays exceeding this value.
Figure 12. Frequency deviation responses of the AC MG under (semi-)constrained and unconstrained ANN-PID controllers compared with the conventional PID for 𝜏 = 2 and 𝐾𝑑 = 0.
Figure 13. Real-time parameter tuning trajectories for 𝜏 = 2.
Table 2. Comparison of different controllers based on the MAE metric.
Delay | Constrained | Ref. [44] | Ref. [43] | PID |
𝜏 = 2 | 0.00078 | 2.116 | 7.233 | 0.021 |
𝜏 = 3 | 0.00142 | 10.452 | 9.935 | 11.61 |
𝜏 = 4 | 0.00204 | 11.855 | 11.981 | 13.881 |
Table 3. Comparison of different controllers based on the IAE metric.
Delay | Constrained | Ref. [44] | Ref. [43] | PID |
𝜏 = 2 | 2.4 | 421 | 1448 | 4.354 |
𝜏 = 3 | 2.95 | 2094 | 1989 | 2326 |
𝜏 = 4 | 4.199 | 2376 | 2402 | 2781 |
The developed intelligent controller has demonstrated strong effectiveness in frequency deviation control (FDC) by leveraging reinforcement learning to autonomously adapt to disturbances in the AC microgrid, thereby improving stability margins and overall performance. Beyond its superior self-adaptation and resilience against unexpected perturbations, the principal advantages of this controller include the following:
• In contrast to conventional intelligent PID-style controllers that rely on meta-heuristic optimization techniques (Table 4), the proposed real-time tuning approach successfully mitigates the adverse effects of communication delays.
• Relative to semi-constrained [45] and unconstrained [43] techniques, the present method achieves markedly better stability by confining the parameter adjustments to a rigorously defined stable subspace. This restriction not only safeguards system stability but also substantially narrows the search domain, enabling faster, more reliable convergence during tuning.
Although the proposed framework offers a smart, computationally efficient control solution that extends beyond PID controllers and can be adapted to tune other classical regulators, certain limitations remain that warrant future attention. The relatively high computational load may constrain its deployment in real-time scenarios involving very large-scale systems, underscoring the need to improve prediction accuracy under rapidly changing conditions. Further strengthening the robustness of the RL framework—through refined reward design and incorporation of digital protection mechanisms—will be essential. Moreover, extending the approach to accommodate uncertain and time-varying delays while preserving a constrained parameter search grounded in microgrid stability boundaries represents a key area for ongoing research. Addressing these issues will be critical to expanding the practical utility and reliability of this methodology across complex modern power systems.
Table 4. Intelligent PID-type controllers for FDC.
Paper | Method | Controller | Components | Delay | Real-Time |
GA | PID/PI | WTG, DEG, STP, AE, ESS | ✗ | ✗ | |
PSO | FOPID | PV, WTG, DEG, FC, ESS | ✓ | ✗ | |
Fuzzy+RL | PID | PV, WTG, DEG, AE, FC, ESS | ✗ | ✗ | |
Fuzzy+BHA | FOPID | WTG, DEG, (B-F)ESS, EV | ✗ | ✗ | |
Fuzzy+HSA | PI | WTG, DEG, EV | ✗ | ✗ | |
BHA | PID | PV, WTG, DEG, (B-F)ESS, EV | ✓ | ✗ | |
ANN | PI | PV, WTG, DEG, FC, ESS | ✗ | ✗ | |
ANN+PSO | PID | WTG, DEG, EV | ✗ | ✗ | |
ANN+RL | PI | PV, WTG, FC, BESS | ✗ | ✓ | |
Proposed | Constr. ANN+RL | PI | PV, WTG, DEG, FC, BESS, FESS | ✓ | ✓ |
This research addresses the challenge of frequency deviation control in isolated AC microgrids by proposing a constrained artificial neural network (ANN) operating strictly within validated stability regions. The developed framework combines stabilizing PID controllers—derived from small-signal stability analysis—with an automated tuning process that optimizes ANN weights and enforces RL-guided constraints. Compared to semi-constrained [45] and unconstrained [43] approaches, the proposed method achieves substantially lower transient overshoots and steady-state errors. By incorporating reinforcement learning alongside an interpolated stability boundary locus, the technique enables flexible, real-time adaptation that maintains strong performance even under varying operating conditions and in the presence of communication delays.
Despite these advantages, the current implementation assumes a fixed communication delay, which may not fully capture the time-varying nature observed in practical deployments. Furthermore, the adoption of sophisticated RL algorithms increases computational requirements and raises potential safety considerations, posing challenges for seamless deployment in large-scale, high-complexity power networks. Resolving these constraints will be essential to enhancing the method’s scalability and real-world viability.
Future investigations will prioritize the design of adaptive mechanisms to handle uncertain or fluctuating delays, improvements in computational performance, and refinements to the RL framework through enhanced reward structures and integration with digital protection schemes. It is expected that the presented intelligent tuning strategy, when paired with emerging advanced control methodologies, will gain widespread acceptance in domains demanding reliable, high-performance regulation solutions.
AE: Aqua electrolyzer
ANN: Artificial neural network
BESS: Battery ESS
BHA: Black hole algorithm
CRB: Complex root boundary
DDPG: Deep Deterministic Policy Gradient
DEG: Diesel engine generators
DG: Distributed generation
ESS: Energy storage system
EV: Electric vehicle
FC: Fuel cell
FDC: Frequency deviation control
FESS: Flywheel ESS
FOPID: Fractional-order PID
GA: Genetic algorithm
HSA: Harmony search algorithm
IAE: Integral absolute error
MAE: Mean absolute error
MG: Microgrid
PID: Proportional–integral–derivative
PSO: Particle swarm optimization
PV: Photovoltaic
RES: Renewable energy source
RL: Reinforcement learning
SBL: Stability boundary locus
SDDPG: Simplified DDPG
SFC: Secondary frequency control
STP: Solar thermal power
WTG: Wind turbine generator
𝛼: Learning rate
𝛽: Smooth factor
𝛥𝑓: Frequency deviation
(𝑠): Characteristic equation
𝛾: Discount factor
𝜇,: Actor and critic neural networks
𝜙: Error angle
𝜏: Communication delay
𝜃𝜇,: Actor and critic ANN weights
𝜑: Solar irradiation
𝑎 𝑡,: Action and state
(𝑠): Closed-loop transfer function
𝐷,: Damping and droop coefficients
(𝑡): Interpolated stability function
(𝑠): Transfer function of PID controller
𝐻: System inertia
𝐾𝑝, 𝑑: Proportional, integral, and derivative gains
𝐿𝜇,: Actor and critic loss functions
𝑚𝜇: Limit output of neuron
𝑃∗,∗: Output power and time constant of unit ∗
𝑃𝑡𝑜𝑡, 𝑡𝑜𝑡,: Total input disturbance and power sources
𝑄′: Best critic reward prediction
𝑟𝑠,: Stabilizing and performance reward signals
𝑟𝑡,: Reward and total reward signals
𝑦𝑡: State–action-value function
This work was supported by the European Union’s Horizon Europe research and innovation program under the grant agreement No 101120657; project ENFIELD (European Lighthouse to Manifest Trustworthy and Green AI); by the Estonian Research Council through the grants PRG658 and PRG1463; and by the Estonian Centre of Excellence in Energy Efficiency, ENER (grant TK230) funded by the Estonian Ministry of Education and Research.
None
None
None
Open Access The author(s) retain copyright. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. It may be shared and adapted for non-commercial purposes with appropriate attribution, an indication of changes, and distribution of adaptations under the same license. Third-party material may be subject to separate terms identified in its credit line. View the license at https://creativecommons.org/licenses/by-nc-sa/4.0/.
Institute of Smart Energy Systems Press
