Estimation of Broadband Atmospheric Turbidity from Global Irradiance or Photovoltaic Power Data Using Deep Neural Networks

Photovoltaic systems contribute substantially to electrical grid capacity, yet the intermittent nature of solar resources and uncertainties in output make grid balancing and reserve allocation challenging. Accurately forecasting solar production begins with reliably estimating clear-sky irradiance levels. While broadband atmospheric turbidity parameters can capture reductions in clear-sky conditions, the precision of these models hinges on data capturing current and future aerosol concentrations and water vapor levels. Obtaining such information requires observations that are both temporally and spatially proximate, given the high variability in turbidity. Unfortunately, these kinds of measurements are available in near real time only at a small—and shrinking—set of locations. This study introduces an innovative approach to infer time-dependent, site-specific turbidity levels using more widely accessible data from pyranometers or monitored PV system outputs. The technique leverages a long short-term memory (LSTM) recurrent neural network to isolate the turbidity-related component embedded within global irradiance (or PV power-derived) records, overcoming the natural suppression of this signal. Designed for near real-time implementation, the method supports solar forecasting needs. Evaluation focuses on two aspects: (1) its capacity to replicate turbidity values calculated historically from beam irradiance observations during clear-sky periods; and (2) its effectiveness when supplying turbidity estimates (via persistence forecasts based on daily averages) as inputs to clear-sky models.

Background

According to the Solar Energy Industries Association (SEIA) 2020 Solar Market Insight Report [1], solar technologies accounted for the largest share of new grid capacity additions in the United States during both 2019 and 2020. In 2020 alone, photovoltaic (PV) installations represented a record 43% of all new capacity. Projections from SEIA indicate that over the following decade, solar capacity connected to the grid will grow fourfold, adding 324 GW.

This expansion holds particular relevance for Hawaii, where energy consumption greatly exceeds in-state production—roughly by a factor of 11—and imported petroleum fuels about four-fifths of electricity needs. Thanks to abundant sunshine, Hawaii emerged as a prime location for solar deployment and became the first U.S. state to achieve grid parity for PV. By 2019, solar accounted for more than half of the state’s renewable energy output, and in 2020 it supplied over 17% of total electricity generation.

PV generation is non-dispatchable because operators cannot directly control output; instead, production tracks irradiance—the incident radiant power per unit area incident on the panels. At elevated penetration levels, variability and unpredictability in irradiance create difficulties for maintaining grid balance, ensuring stability, and managing reserves. These challenges intensify in Hawaii, where each island relies on its own compact, islanded power system.

Irradiance fluctuations stem from both extraterrestrial and atmospheric influences. Outside the atmosphere, irradiance follows predictable patterns determined by Earth’s orbit and the solar cycle. Within the atmosphere, however, incoming radiation undergoes extensive modification through scattering, absorption, and re-emission processes involving air molecules, aerosols, and other particulates [2]. The core challenge in solar forecasting lies in characterizing atmospheric conditions [3, 4]. Many prediction techniques separate baseline clear-sky irradiance—computed via a clear-sky model (CSM)—from cloud-induced reductions. The overall effectiveness of these approaches depends heavily on the quality of the CSM.

In this context, clear-sky irradiance refers to the radiation level under cloud-free atmospheric conditions, where clouds exert no influence on the solar resource or resulting PV output. Under clear skies, attenuation primarily arises from aerosols (suspended particles sized roughly 1 nm to 5 μm [5]) and water vapor [6, 7]. Typical aerosols include sea salt from ocean-air exchange, smoke and soot from wildfires, volcanic ash and dust, sulfates from fossil fuel combustion, and black carbon from heavy oil burning.

Clear-sky models fall into two broad groups: detailed radiative transfer models and simpler empirical formulations [6, 8, 9]. Radiative transfer approaches generally yield superior attenuation estimates but require multispectral observations that are seldom available. Empirical models, better suited to real-time use, typically rely on nearby broadband beam irradiance data from specialized instruments [7, 10-14]. Deployment of such tracking or shadowing devices remains restricted—and is declining—owing to their high cost (around $30K USD) and ongoing maintenance/calibration demands for moving components [15, 16].

By contrast, pyranometers cost far less ($1–5K USD) and have no moving parts, reducing maintenance requirements. As a result, standardized global horizontal irradiance (GHI) data from pyranometers are abundant, including many real-time streams. Yet pyranometer readings capture global irradiance, which responds less sensitively to changes in turbidity than beam measurements do. In global data, the turbidity signal weakens because the direct and diffuse components react in opposite directions: direct irradiance decreases with increasing turbidity, while diffuse irradiance increases. This opposing behavior has historically restricted the utility of pyranometer records for turbidity assessment. Earlier efforts using archived GHI data include Polo et al. [17]’s approach to derive monthly turbidity from solar-noon GHI, Hove et al. [18]’s minimization technique comparing observed GHI to CSM predictions in Zimbabwe, and a comparable strategy by Riveros-Rosas et al. [19] for monthly estimates in Mexico.

Aim, hypothesis, and validation

This work seeks to develop a novel technique for estimating near-real-time turbidity using deep neural network (DNN) methods and more commonly available inputs, such as pyranometer readings, weather station data, and PV system monitoring records. DNNs excel at capturing complex nonlinear relationships among multiple variables. The core hypothesis is that a DNN can learn to recover the suppressed turbidity signal from global irradiance measurements, enabling broadband turbidity quantification during clear-sky periods. Broadband turbidity is expressed here via Linke’s turbidity coefficient, TL [20, 21], which indicates the equivalent number of clean, dry atmospheres needed to match the radiative extinction of the actual atmosphere.

The DNN serves as a decomposition tool, predicting diffuse irradiance from inputs from weather stations, pyranometers, or PV outputs, along with supplementary model-derived information. These inputs encompass atmospheric attenuation metrics, meteorological variables, solar positioning, and clear-sky flags. Direct irradiance is then inferred from the predicted diffuse component using an energy balance, followed by inversion of a direct irradiance model to derive turbidity. Should the approach prove effective, it would enable broader, more detailed turbidity monitoring. Pyranometer and weather station networks far outnumber beam- or diffuse-measurement stations, and distributed PV data is increasingly accessible via platforms like pvoutput.org. Applications include retrospective analysis of turbidity trends and spatial patterns, as well as near real-time provision of Linke turbidity values to CSMs that use this parameter to account for attenuation beyond Rayleigh scattering in operational solar forecasting.

Method validity is discussed in terms of resulting error, with an individual error, ei, defined as the difference between a predicted value, pi, and a corresponding “true” value, yi,

Obtained from diffuse irradiance measurements (or turbidity derived therefrom). Note that measured values approximate true conditions with inherent uncertainties; “error” here denotes only the difference between method output and reference, not measurement uncertainty.

Summary error metrics include Mean Bias Error (MBE) and Root Mean Square Error (RMSE), calculated for N observations as:

Normalized versions express these as percentages of the mean observed (or derived) value. The coefficient of determination (R²) quantifies explained variance:

Effective models exhibit low RMSE, MBE near zero, and high R². Evaluations are conducted against references that include observational uncertainties, which affect reported metrics.

Testing and validation use four datasets from three sites on Oahu, Lanai, and Maui (Figure 1). To maximize independence, the DNN is trained on only the Kala dataset, with the remaining three datasets reserved for validation.

Each dataset covers at least 500 days and includes diffuse irradiance (for DNN training/validation) and derived turbidity estimates (for overall method validation), along with inputs such as temperature, humidity, wind speed, global irradiance or PV power, solar position, clear-sky irradiance, and clear-sky Boolean flags. Datasets, sites, and instrumentation details follow (Table 1).

The Kala dataset originates from instruments at Kalaeloa Airport on southeast Oahu, provided by the National Renewable Energy Laboratory’s (NREL) Measurement and Instrument Data Center (MIDC) under the Solar Resource and Meteorological Assessment Project (SOLRMAP) [22]. Covering July 2009 to November 2012, it features diffuse irradiance from a Multi-Filter Rotating Shadowband Radiometer (RSR) Model MFR-7, global irradiance from a LICOR LI-200 pyranometer, and temperature/humidity from a Campbell Scientific CS215 sensor. Raw 3-second samples were averaged to 1-minute intervals before release [22]. Both irradiance sensors rely on unfiltered silicon photodiodes with diffusers; post-processing corrects for spectral response limitations. The resulting uncertainty approximates that of first-class thermopile pyranometers, around ± 5%.

Another set of observations, designated as Laola, consists of solar irradiance records collected from equipment installed at the 1.5 MW La Ola photovoltaic power station on Lanai Island. This information is distributed by the National Renewable Energy Laboratory (NREL) through its Measurement and Instrumentation Data Center (MIDC) within the SOLRMAP project [22]. Data collection for Laola took place between July 2009 and November 2012. The measurement devices matched those deployed for the Kala campaign; raw samples were acquired every 3 seconds and later averaged over 1 minute before release.

The final pair of datasets comes from monitoring instruments placed at the Maui Economic Development Board (MEDB) facility in Kihei, situated along Maui’s southwestern shoreline. There, researchers from the Hawaii Natural Energy Institute (HNEI) developed a dedicated photovoltaic testing platform outfitted with precision sensors to capture fine-scale solar and PV performance data. Both datasets span April 2016 to September 2018. They include diffuse horizontal irradiance obtained with a Delta-T Devices SPN1 Sunshine Pyranometer, together with air temperature and relative humidity logged by a Gill MetPak Pro meteorological station.

The Medb dataset includes total (global) irradiance readings from a Kipp & Zonen SMP21-A class pyranometer. In contrast, the Medb-PV dataset records direct-current electrical output from a dedicated measurement unit monitoring a 1.96 kW array comprising eight premium n-type monocrystalline silicon modules. The SMP21-A qualifies as a secondary standard instrument, with measurement uncertainty typically below ± 3%. This pyranometer was deliberately aligned to record irradiance in the plane of array (POA) consistent with the PV modules themselves, which face an azimuth of [specific value]° (measured clockwise from geographic north) and are inclined at [specific value]°.

The SPN1 instrument determines diffuse irradiance using a combination of seven thermopile detectors and proprietary software that applies dynamic shading patterns; its uncertainty is typically kept to ± 8%.

General description and illustrative example

An example day illustrating the inputs and outputs of the proposed approach is presented in Figure 2.

In Panel A, the global horizontal irradiance recorded by a pyranometer is plotted alongside the corresponding clear-sky model (CSM) values, assuming a fixed turbidity coefficient of 3. Periods identified as clear-sky conditions—based on the procedure —are also marked on the pyranometer measurements.

Panel B displays the clear-sky attenuation factor, computed as the ratio of the observed pyranometer irradiance (under clear-sky periods) to the CSM-predicted irradiance. This panel also includes the clearness index values, which serve as a quantitative indicator of overall atmospheric attenuation and form one of the primary inputs to the deep neural network (DNN) models .

In Panel C, diffuse irradiance estimates produced by the DNN-based separation model are compared against the actual diffuse measurements. These measured diffuse values were used for training, testing, and validation of the DNN.

Panel D presents both the direct irradiance derived from the DNN outputs and the direct irradiance calculated directly from observations. These two direct irradiance datasets are then utilized in Panel E to derive estimates of atmospheric turbidity.

Clearness index and clear-sky index

Atmospheric attenuation of solar radiation is assessed by normalizing measurements from pyranometers or PV system output using standardized index values. The foundational idea of dividing observed solar radiation by its theoretical clear-sky maximum under a completely transparent atmosphere was introduced by Black et al. [23]. This evolved into the “clearness index” (kt), defined as the ratio of measured global horizontal irradiance to extraterrestrial horizontal irradiance [24], and later the “clear-sky index” (kc), which compares measured horizontal irradiance to the irradiance expected under cloud-free conditions as simulated by a clear-sky model (CSM) [25]. More recently, Engerer et al. [26] proposed a “modified clear-sky index” that normalizes measured PV power output against the expected output under clear-sky conditions, computed using a CSM coupled with a PV performance model.

Building on these concepts, the present study adopts the term “clearness index” (kt) to denote either the ratio of measured irradiance (G) to extraterrestrial irradiance (G₀) or the ratio of measured DC PV power output (Pdc) to the corresponding extraterrestrial power output (P₀). The term “clear-sky index” (kc) refers to the ratio of measured irradiance to modeled clear-sky irradiance (Gcs), or the ratio of measured PV power to modeled clear-sky PV power output (Pcs), or:

Extraterrestrial irradiance is calculated by applying Fourier series adjustments to the solar constant—taken here as 1361.2 W/m² [6]—to account for variations caused by the eccentricity of Earth’s orbit. Solar position is determined using a fast and accurate solar geometry model (SGM) with a maximum error of 0.0027° [27], providing the solar zenith angle (θz), declination angle (δ), and hour angle (ω).

Clear-sky direct and global horizontal irradiance values are generated using the Ineichen–Perez clear-sky model [10], which incorporates the Linke turbidity factor (TL) to account for attenuation beyond simple Rayleigh scattering. TL expresses the equivalent number of clean, dry atmospheres needed to match the observed attenuation, based on the optical depth of a theoretical clean dry atmosphere [20, 21]. The Ineichen–Perez version corrects for spurious diurnal fluctuations in turbidity that appear in the original formulation under constant aerosol conditions, ensuring more stable daily turbidity estimates. For tilted surfaces, global clear-sky irradiance is computed by scaling the clear-sky diffuse component with a sky-view factor (assuming isotropic diffuse sky), adding ground-reflected irradiance using an albedo value [28], and enforcing closure. Albedo values for typical land surfaces fall between 0.1 and 0.4; an albedo of 0.1 is adopted in this analysis.

Clear-sky and extraterrestrial PV power outputs are estimated for a given system—defined by module size, type, mounting configuration, and orientation—by combining clear-sky irradiance inputs with a photovoltaic modeling framework (PVM) based on NREL’s PVWatts Version 5 [29]. The PVM incorporates temperature-induced efficiency reductions via an empirical thermal model and corrects for angle-of-incidence reflection losses using a polynomial fit [30]. Ambient weather data from on-site instruments drives the thermal calculations.

To ensure that changes in broadband turbidity produce consistent changes in the quantified atmospheric attenuation captured by the clear-sky index (kc), both Gcs and Pcs are computed under a fixed turbidity value (TL = 3).

For convenience in subsequent sections, the specific normalization model applied to irradiance and PV power data is denoted CSMre.

Detection of clear-sky periods

Under cloud-free skies, solar irradiance exhibits smooth temporal variation. Clear-sky intervals are detected by applying a sequence of threshold-based criteria that evaluate the smoothness, profile shape, and overall magnitude of observations relative to their corresponding clear-sky references within a sliding time window. This procedure adapts and reformulates the clear-sky detection algorithm originally developed by Reno et al. [31] for use with either pyranometer irradiance or PV power time series.

The initial criterion checks whether the clear-sky index falls within an acceptable range, confirming that the observed value is comparable in scale to the modeled clear-sky value. Next, slope and fluctuation statistics are computed over an 11-minute moving window. The second test examines the average slope of the observations against the average slope of the clear-sky reference over the same window. The third test compares a Variability Index (VI) [32] calculated from the measurements to the VI derived from clear-sky values. The fourth and final test compares the standard deviation of the observations in the window to that of the clear-sky counterparts. An observation is classified as clear-sky only if it satisfies all four threshold conditions. Any data point lacking a sufficient number of adjacent valid observations is automatically treated as cloud-influenced.

Starting threshold values were drawn from recommendations in Reno et al. [31] and subsequently refined through visual evaluation of the resulting variability metrics and classification outputs. Because quantitative validation of clear-sky detection accuracy is challenging, a qualitative assessment is provided in Figure 3, which displays twelve consecutive days with diverse sky conditions extracted from the Laola dataset. Observations flagged as clear-sky should exhibit markedly lower short-term variability while closely matching the magnitude and slope of the CSMre predictions. This behavior is especially evident on December 8 and 9, 2009, when predominantly clear conditions prevailed.

(For detailed interpretation of color references in this figure, refer to the online version of the article.)

DNN separation model

Early separation techniques were primarily used for low-resolution data (hourly or longer) and relied on empirical correlations among global, diffuse, and extraterrestrial irradiance. As time progressed, these models became more intricate by including additional predictor variables. Even though many hourly models can be adapted to higher resolutions [9], recent research has focused on improving 1-minute-resolution predictions. Even though machine learning methods have shown progress at this scale [33-35], separating diffuse and direct irradiance components from global radiation measurements remains challenging. More recent models report nRMSE values of 20%-35% for the diffuse component [36-39]. The approach proposed in this study uses a DNN to separate diffuse irradiance at 1-minute intervals. The model incorporates long short-term memory (LSTM), a specific type of recurrent neural network. LSTM networks are well-suited for learning long-term dependencies in time-series data, using both long-term and short-term memory feedback loops [40]. This ability allows the model to adapt to processes that change over different time scales. The DNN models were developed using MATLAB’s Deep Learning Toolbox.

Architecture of the model

The DNN model begins with an input layer followed by six LSTM layers, each containing 100 hidden units. The final layer is a fully connected regression output layer. Additionally, each LSTM layer is followed by a dropout layer with a 10% dropout rate. Dropout is a simple yet effective regularization technique that reduces overfitting by randomly omitting nodes during training, improving the model’s ability to generalize [41-43].

Training the model

Training is performed using the Adam optimizer, an adaptation of stochastic gradient descent with adaptive learning rates for each parameter [44]. The input data are standardized to address differences in scale between the input and target variables. The DNN’s structure and hyperparameters were optimized through trial-and-error. The training dataset consists of the first half of the KALA dataset, totaling approximately 140,000 data points collected over 8 months. The model training process concluded after 200 epochs. The model’s performance was then tested on the full 3-year LAOLA dataset (over 570,000 points) and a subset of LAOLA data from clear-sky conditions (over 128,000 points).

Feature selection for input data

During model development, a variety of input variables were explored, and those that did not contribute meaningfully were excluded. The remaining input features included: relative humidity, ambient temperature, clearness index, clear-sky index, clear-sky classification (as a Boolean value), and solar zenith angle (SZA). The clearness and clear-sky indices are derived from the ratio of measured to theoretical extraterrestrial irradiance. To identify the most effective combination of input variables, 13 DNN models were trained and evaluated based on their performance in predicting diffuse irradiance. Details of the input features used in each model, along with their naming conventions, are provided in Table 2. Performance was assessed by comparing the predicted and observed irradiance, with results presented in Table 3.

Across both cloud-affected and clear-sky conditions in the LAOLA dataset, all models performed reasonably well, with RMSE values ranging from 43.2 to 62 W/m² and normalized values ranging from 23.3% to 33.5%. The majority of models exhibited negative bias, indicating they tended to overestimate irradiance values. This bias was most significant in models that included environmental factors, which reduced model accuracy, particularly under clear-sky conditions. Nevertheless, all 13 models explained a significant portion of the observed variance, with R² values ranging from 0.79 to 0.9. Under clear-sky conditions, performance varied more between models. Two models, S14z and S014z, stood out with notably lower error rates and higher R² values, with rRMSE under 30%. These two models also exhibited the least bias and the highest R² scores. They were among only four models to produce positive R² values, and the only ones with R² values above 0.5. S014z slightly outperformed S14z in both the full dataset and under clear-sky conditions, making it the chosen configuration for the DNN separation model. The input features for this model include the clearness index, the clear-sky index, the clear-sky classification, and SZA. No additional training was performed; all further discussions about the DNN separation model refer to the S014z configuration.

Comparative effectiveness

The effectiveness of the deep neural network (DNN) separation approach is assessed by benchmarking it against six widely used separation techniques (Table 4). Among these, the Erbs, Louche, OH, Reindl1, and Reindl2 models rely solely on a single input—the clearness index. In contrast, the Disc model incorporates two variables: the clearness index and air mass. The Dirint model extends the Disc framework by additionally including dew-point temperature and a measure of short-term fluctuations, captured via a so-called “variability index.”

Each of these models determines the direct irradiance component, after which the diffuse irradiance is derived through the closure relation using the measured global irradiance:

Where G represents global horizontal irradiance, Gₙ is direct normal irradiance, z denotes the solar zenith angle (SZA), and G_d stands for diffuse irradiance.

Following the methodology of the previous section, performance is evaluated across the entire dataset (all sky conditions) and only under clear-sky conditions. The corresponding error metrics appear in Table 5. Among the conventional models examined, the four-predictor formulation (Dirint) delivered the strongest performance in both the complete dataset evaluation and the clear-sky subset.

Across all sky conditions, the DNN-based separation approach outperforms all conventional methods, even though those standard techniques deliver reasonably strong results overall. Specifically, the DNN model predicts diffuse irradiance with a root mean square error (RMSE) of 73.64 W/m² (equivalent to a relative RMSE of 39.79%) and a mean bias error (MBE) of 12.8 W/m² (corresponding to a relative MBE of 6.91%). The six benchmark models successfully capture a substantial portion of the observed variability in diffuse irradiance, achieving coefficient of determination (R²) values between 0.41 and 0.70. In comparison, the DNN model reduces RMSE by 44.4% to 58.4% relative to the standard approaches, lowers absolute MBE by 77.3% to 90.5%, and achieves R² values that are 0.20 to 0.49 higher.

When restricted to clear-sky conditions, the superiority of the DNN separation model becomes even more pronounced, while the conventional methods exhibit markedly weaker performance. Here, the DNN achieves RMSE reductions of 54.4% to 79% and MBE reductions of 53.7% to 94.4% compared with the standard models. Notably, none of the six conventional models yield a positive R² under clear skies, meaning they fail to explain any of the observed variance in diffuse irradiance (a similar lack of skill was previously observed in 8 out of the 11 DNN configurations evaluated in the earlier section).

Estimating turbidity

The suggested technique is designed to process inputs from either pyranometer measurements or photovoltaic (PV) power output. It estimates direct irradiance through an adapted closure relation expressed as

This estimation depends on the clear-sky index and outputs from the reference clear-sky model (CSM ref).

Subsequently, atmospheric turbidity is derived from an inversion of the Ineichen–Perez clear-sky model, using approaches described by Nou et al. [51] and Inman et al. [52]. The procedure assumes that the turbidity parameterization in the Ineichen–Perez framework remains consistent under uniform aerosol conditions, enabling the extraction of a characteristic turbidity value applicable across cloud-free conditions regardless of air mass. The inversion takes the following form:

Values for air mass (AM) and extraterrestrial irradiance (G0) are obtained directly from the CSM_ref, while the parameter b is computed based on the site’s elevation z according to:

This methodology facilitates turbidity estimation either from diffuse irradiance generated by the deep neural network (DNN) or from direct observational data (used for validation).

Attention now turns to evaluating the reliability of the turbidity estimates generated by this approach. Performance is measured by comparing discrepancies between turbidity values derived from the DNN and those calculated from observations.

High-frequency validation

To evaluate how well the technique matches turbidity values obtained from observations, various performance indicators are computed at the native temporal resolution of the input data (1-minute intervals). The influence of different input sources on estimation accuracy is examined by analyzing errors across multiple datasets. Findings are summarized in Table 6.

As anticipated, the DNN’s turbidity values show the greatest accuracy when applied to the dataset used for model training (which includes more than 47,000 clear-sky observations). In this case, the DNN-based turbidity closely matches observation-derived values, achieving an RMSE of 0.26 and explaining 55% of the variance (R² = 0.55). For the second dataset (with over 128,000 clear-sky points), the DNN estimates exhibit marginally larger errors, with an RMSE of 0.28 and an explained variance (R²) of 0.44. Results for the Medb dataset (containing over [number] clear-sky points) show error levels comparable to those of the Kala dataset. However, its R² value is the lowest among pyranometer-based datasets at 0.36. In contrast, the [fourth] dataset, which comprises more than 97,000 clear-sky observations, yields the largest errors among the four, with an RMSE of 0.35—representing roughly a 35% increase compared to the Kala results—and an extremely low R² of 0.04, indicating minimal capture of the observed turbidity variability. Across all datasets, the normalized RMSE remains under 11%, while the normalized MBE stays less than 2%

Validation at the weekly aggregation level

Key factors influencing variations in atmospheric turbidity include nearby pollution sources, wind-driven dispersion, air temperature patterns, and larger-scale atmospheric flows [53]. The analysis here examines how effectively the proposed technique captures changes in turbidity linked to mesoscale and regional circulation dynamics. This averaging timescale aligns with the primary periods of tropical atmospheric circulation, typically spanning 5–10 days [54], while also incorporating seasonal influences and providing sufficient samples per distribution to support robust statistical analysis.

In broad terms, increasing the averaging period reduces RMSE and increases R². Relative to the 1-minute results, daily aggregations lower RMSE by 3% to 17% and boost R² by 0.12 to 0.22.

Figure 4 presents a time series of weekly mean turbidity estimates, along with their 2σ spreads (defined here as the 5th–95th percentile interval). Across most weeks, turbidity values range mainly from 2 to 4, though occasional episodes show elevated levels. Both observation-based (Medb and Medb-PV) and DNN-based estimates reveal clear seasonal cycles, which appear more distinct in the Laola and Kala datasets, featuring higher turbidity during summer periods.

The weekly means derived from observations and from the DNN show strong qualitative correspondence overall. However, the 2σ ranges for observation-based estimates are systematically wider than those produced by the DNN.

Figure 5 illustrates a weekly time series of the differences between observation-derived and DNN-derived turbidity values. The mean differences stay predominantly within ±0.4, with no evidence of consistent directional bias. The 2σ spreads remain mostly within ±1 across the datasets, though discrepancies are notably larger in the Medb-PV case than in the others.

Many clear-sky models that incorporate the Linke turbidity parameter default to climatological turbidity datasets when real-time observations are unavailable, an approach that has been demonstrated to degrade model precision [52]. A widely adopted resource is the Solar Radiation Data (SoDa) Service, which provides monthly global turbidity maps at approximately 10 km resolution [55].

For context, Figure 5 also plots the differences between SoDa climatology-derived turbidity and the observation-derived values at weekly resolution, alongside the observation/DNN differences. The SoDa product exhibits a persistent positive bias relative to the site-specific observation-derived estimates at all three locations. This offset is relatively modest at the more eastern island sites on Lanai and Maui (Laola and Medb), where SoDa values average 0.21 above observations for Laola and 0.3 above for Medb. Yet the 2σ spreads for Laola tend to be broader than those for Medb, occasionally exceeding ± 2. In contrast, the bias is substantially larger for the Kala dataset on Oahu: SoDa turbidity averages 1.4 higher than observation-derived values, with weekly 2σ difference distributions frequently extending beyond −3.

Turbidity estimation using PV power output

Several factors may explain the noticeable drop in performance when the method relies on photovoltaic (PV) power measurements as input. The most significant issues likely stem from shading caused by construction activity on an adjacent property that started in late 2016, as well as soiling of the PV panels. Soiling tends to reduce PV output more severely than pyranometer readings do because the surface area of solar panels is much larger than that of a pyranometer dome, making dust or debris accumulation more impactful on power generation.

To detect periods when PV-specific problems may have degraded the turbidity estimation, differences in the clear-sky index calculated from pyranometer data versus PV power data are examined. The upper panel of Figure 6 displays the mean bias error (MBE) of these clear-sky index differences, organized into bins by calendar date and time of day (using 5-day and 30-minute bin widths). Comparably, the middle panel of Figure 6 presents the MBE based on differences between observation-derived and DNN-derived diffuse irradiance values for the Medb-PV dataset.

Shading influences become visible in the clear-sky index differences only when one instrument (pyranometer or PV array) is shadowed while the other remains exposed. Given that the pyranometer is positioned roughly 15 meters from the PV panels, such mismatches can occur. The distinct pattern and timing of the observed bias—an increase in afternoon clear-sky index MBE starting in late 2016—are attributed to shading from the nearby building construction. The strongly positive bias indicates that the PV panels were obstructed while the pyranometer continued to receive full sunlight. A marked escalation in this bias during spring 2017 corresponds to further progress in the construction work. By the following winter, once the building was completed, persistent afternoon shading of the PV system became apparent and persisted for more than 4 months. At its most severe point in December 2017, shading of the PV array began shortly after solar noon.

These same time periods and hours show correspondingly larger discrepancies between observation-based and DNN-based diffuse irradiance estimates. The largest errors in the separation model approach exceed 25 W/m² during peak PV shading.

When shading effects are absent, a consistent pattern emerges in the differences in the clear-sky index. Throughout 2016, a prevailing negative bias was observed across most date and time bins. After April 2018, however, this shifts to a positive bias that appears in nearly all bins. This change points to a gradual decline in PV power generation relative to expectations. The drop in output is likely due to the aging of system components and the progressive buildup of dust, dirt, and other deposits on the panels. The particularly rapid rise in bias during 2018 coincides with major volcanic activity on the Big Island of Hawaii, which commenced on April 18, 2018. Prevailing trade winds transported volcanic emissions, including gases and ash, across the island chain, degrading air quality on the affected islands during that period.

The bottom panel of Figure 6 displays RMSE values for diffuse irradiance and turbidity aggregated over 20-day intervals. The diffuse irradiance RMSE is determined from differences between observation-based and DNN-based (PV-powered) estimates across the entire Medb-PV dataset. Turbidity RMSE is computed similarly, but only for clear-sky conditions, using differences between observation-derived and DNN-derived turbidity. Periods affected by PV array shading exhibit elevated errors in diffuse irradiance separation, yet these do not translate into corresponding increases in turbidity estimation error—shaded measurements are flagged and treated as cloud-contaminated. In contrast, periods of heightened turbidity estimation error align with times when PV soiling is believed to be the dominant factor.

Relevance to solar power forecasting

The method presented here is intended for near-real-time operation, delivering turbidity information that can support improved solar forecasting. This section assesses the performance of the technique when its turbidity estimates serve as inputs to the Ineichen–Perez clear-sky model in a forecasting context (denoted CSM_fore).

To illustrate the influence of turbidity on photovoltaic output, synthetic clear-sky PV generation profiles were created for the MEDB site. These profiles use observed clear-sky irradiance and meteorological conditions as drivers within a PV simulation model. Each annual profile assumes a fixed turbidity level while maintaining otherwise identical inputs. Turbidity values are varied from 2 to 6 in steps of 0.5, spanning the typical range observed across the three measurement sites. Daily energy yields from these profiles are presented in Figure 7. Compared to a baseline turbidity of TL = 2, energy production at TL = 6 is reduced by 14–31%. For each 0.5 increment in turbidity, daily output changes by approximately 1.5–5%, resulting in an overall shift of roughly 3% in total annual energy production.

The evaluation compares CSM_fore performance when driven by three different turbidity sources:

·   Day-ahead forecasts based on observation-derived turbidity (CSM_foreOBS),

·   Day-ahead forecasts based on DNN-derived turbidity (CSM_foreDNN),

·   Day-ahead forecasts using the SoDa climatological turbidity (CSM_foreSoDa).

Day-ahead turbidity forecasts assume simple persistence: the daily mean turbidity from the previous day is carried forward (Figure 8). Each daily mean is computed from 1-minute irradiance or PV power observations collected during daylight hours on that day.

Forecast accuracy is evaluated by comparing predicted global horizontal irradiance from CSM_fore against observed clear-sky values. Key performance statistics appear in Table 7. For the Medb dataset, metrics are derived exclusively from time steps classified as clear-sky in both the Medb (pyranometer) and Medb-PV (power-based) records. This filtering excludes about 12% of the data but ensures a fairer comparison of the impact of switching to PV power as the input source. Across all CSM_fore configurations, R² values exceed 0.99 and are therefore omitted from the table.

Among the evaluated configurations, CSM_foreOBS (using day-ahead persistence forecasts of observation-derived turbidity) delivers the highest accuracy. Root mean square error (RMSE) values fall between 9.13 and 12.49 W/m², which translates to normalized errors of approximately 2%. Mean bias error (MBE) remains small in magnitude, ranging from 0.7 to 1.52 W/m² and corresponding to relative mean bias errors (|rMBE|) below 0.4%.

When turbidity inputs come from the DNN applied to pyranometer measurements (CSM_foreDNN pyranometer-based), a modest reduction in precision is observed. RMSE increases by 1.26 to 2.15 W/m² compared to CSM_foreOBS, although the MBE values show virtually no change and remain effectively unbiased.

Performance declines further when the DNN relies on PV power data as input (CSM_foreDNN PV-based). In this case, RMSE rises by approximately 3.85 W/m² relative to the observation-based reference (CSM_foreOBS), reflecting the cumulative impact of the additional uncertainties introduced by the PV-derived clear-sky index and subsequent turbidity estimation steps.

In summary, employing simple persistence-based day-ahead turbidity forecasts in CSM_foreOBS and CSM_foreDNN yields highly accurate clear-sky irradiance predictions, with relative root-mean-square errors (rRMSE) below 3%. These results markedly outperform the configuration that relies on the SoDa climatological turbidity input (CSM_foreSoDa). Specifically, the method-derived forecasts (CSM_foreDNN) reduce RMSE by 28%–66% and MBE by 66%–96% compared to the climatology-driven case.

This study introduces, implements, and rigorously evaluates a novel approach for estimating time-varying, site-specific atmospheric turbidity based solely on global horizontal irradiance measurements from pyranometers. The technique is designed to make optimal use of widely available instrumentation and to enable frequent, near-real-time turbidity retrieval across large areas.

Pyranometers for global irradiance are far more commonly installed than specialized direct or diffuse sensors, owing to their lower cost, simpler deployment and maintenance requirements, absence of moving parts, and longer operational lifespan. The core premise of the method is that a deep neural network (DNN) can be trained to isolate the turbidity-related signal embedded within global irradiance—or even PV system power output—despite the opposing responses of the direct and diffuse components to changes in aerosol loading. By functioning as an effective irradiance separation model, the DNN estimates the beam irradiance component, which is then inverted using a clear-sky model to derive turbidity.

The DNN receives a set of inputs that encode atmospheric attenuation under the assumption of constant turbidity within the reference clear-sky model (Ineichen–Perez): modified clearness and clear-sky indices computed from pyranometer or PV measurements, solar position data from a solar geometry model (SGM), and a binary clear-sky flag produced by a detection algorithm applicable to either pyranometer or PV data. In the case of PV systems, inputs are supplemented by outputs from a PV performance simulation tool driven by clear-sky irradiance and observed environmental conditions.

An extensive search for the best input combination involved training and testing numerous DNN architectures. Models were initially trained on approximately nine months of data collected on Oahu and subsequently validated using more than three years of independent measurements from Lanai. Model quality was assessed across the entire dataset and under clear-sky conditions using RMSE, MBE, and R² metrics, with particular emphasis on R² during cloud-free periods, as this metric best reflects the model’s ability to capture turbidity-driven variations in diffuse irradiance.

Early experiments with both DNNs and conventional separation models yielded persistently negative R² values under clear skies. Including temperature and relative humidity as additional inputs actually degraded performance, especially during clear conditions. The decisive improvement came from incorporating a simple Boolean clear-sky indicator as an input feature. With this addition, three DNN configurations achieved positive R² under clear skies. The best-performing model used a combination of the clearness index, the clear-sky index, the clear-sky flag, and the solar zenith angle (SZA).

Compared with six widely used empirical separation models, the optimized DNN delivered substantial gains: RMSE reductions of 44.4%–58.4% across all sky conditions and 54.4%–79% under clear skies. Across the full dataset and under clear conditions, the DNN reproduced observed diffuse irradiance with rRMSE below 24%, rMBE within ± 3%, and R² values of 0.90 (all conditions) and 0.61 (clear skies).

Subsequent validation targeted the accuracy of the resulting turbidity estimates across four independent datasets collected at three Hawaiian sites. At the native 1-minute resolution, DNN-derived turbidity closely matched reference values derived from rotating shadowband radiometer (RSR) observations, exhibiting very small bias (rMBE magnitude < 2%) and moderate error (rRMSE < 11%). Explained variance (R²) reached 0.44–0.55 when using horizontal pyranometer inputs, fell to 0.36 for a tilted pyranometer, and approached zero when relying on PV power-derived indices. Detailed analysis revealed that shading from nearby construction (beginning late 2016) and progressive soiling of PV panels were the primary factors degrading PV performance. Aggregating to coarser timescales markedly improved results: RMSE decreased from 0.26–0.35 (1-min) to 0.24–0.3 (daily) and 0.15–0.23 (weekly).

The final evaluation assessed the utility of the method for supplying turbidity inputs to Linke-turbidity-based clear-sky models in a day-ahead forecasting context. The highest irradiance prediction accuracy was achieved with persistence forecasts based on directly observed turbidity. Forecasts based on the proposed DNN method produced only a marginal increase in rRMSE (less than 1 percentage point) relative to observation-based forecasts, yet delivered major improvements over forecasts that used the SoDa monthly climatology.

Collectively, these findings demonstrate that the DNN successfully extracts a meaningful turbidity signal from global irradiance measurements, overcoming the natural dampening effect present in pyranometer data. Although performance declines when switching to PV power as the input source, even PV-derived turbidity forecasts still yield substantially better clear-sky model accuracy than climatological defaults. Moreover, the growing availability of pyranometer networks combined with public PV data-sharing platforms (e.g., pvoutput.org) offers the potential for dense regional sampling. Such networks could enable detailed mapping of spatiotemporal turbidity patterns and provide avenues for further enhancing estimation accuracy [56, 57].

It should be emphasized that the present results are specific to the Hawaiian environment, as both training and validation relied exclusively on local measurements. While the method shows strong promise for other tropical regions, broader geographic applicability would require retraining the DNN on representative data from those areas and conducting additional independent validation.

The author also wishes to thank HNEI’s Severine Busquet, Jonathan Kobayashi, and Richard Rocheleau for their efforts related to the MEDB testbed deployment and data collection, which ONR, United States, also funded.

None

This work was funded by the Office of Naval Research (ONR), United States, under the Asia Pacific Research Initiative for Sustainable Energy Systems (APRISES) project, Grant Award Number N00014-16-1-2116, and by the U.S. Department of Energy, United States, under the U.S.-India Collaborative for Smart Distribution System with Storage (UI-ASSIST) project, Award Number DE-IA0000025.

None

Isabella Garcia & Diego Herrera contributed to this work.

Authors and affiliations

Department of Smart Grid Technologies, Faculty of Engineering, University of Buenos Aires, Buenos Aires, Argentina
Isabella Garcia & Diego Herrera

Corresponding author

Correspondence to Isabella Garcia

Open Access The author(s) retain copyright. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. It may be shared and adapted for non-commercial purposes with appropriate attribution, an indication of changes, and distribution of adaptations under the same license. Third-party material may be subject to separate terms identified in its credit line. View the license at https://creativecommons.org/licenses/by-nc-sa/4.0/.