ASCAT is a C-band (5.255 GHz) vertically polarized scatterometer known for its high radiometric accuracy . The ASCAT instruments are onboard three Metop-series polar-orbiting satellites, Metop-A, Metop-B, and Metop-C, launched on 19 October 2006, 17 September 2012, and 7 November 2018, respectively. These satellites operate in a sun-synchronous, low Earth polar orbit at an altitude of approximately 817 km, enabling near-global coverage every 12 h. In our study area, ASCAT measurements are typically acquired during two daily time windows: from 00:00 to 05:00 UTC (ascending orbits) and from 14:00 to 19:00 UTC (descending orbits). The ASCAT backscattering coefficient has been used to retrieve SM, as it is strongly influenced by the soil's dielectric properties, which vary with moisture content .

We used the Metop ASCAT SSM CDR (H120 version 7), which provides a consistent data record of SM-related products at 12.5 km resolution , covering the entire duration of Metop satellite missions (2007–present). This dataset is derived using the latest version of the SM retrieval algorithm developed by TU Wien under the EUMETSAT H SAF program . Two key variables are used here:

Backscatter (σ40). The ASCAT instrument measures backscatter at various incidence angles. To facilitate intercomparison and reduce vegetation-related effects, the Metop ASCAT CDR provides backscatter normalized to a reference incidence angle of 40° (in dB). This normalized value, σ40, is more consistent and less sensitive to surface heterogeneity compared to raw backscatter observations.

To convert the relative SM values to volumetric SM (m³ m⁻³), we multiplied the ASCAT SM values by soil porosity, following the approach of . Porosity estimates were obtained from the Global Land Data Assimilation System (GLDAS) dataset , accessible at https://ldas.gsfc.nasa.gov/gldas/soils (last access: 31 May 2025).

In this study, we utilized the ERA5 reanalysis dataset to benefit from the comprehensive representation of the land-atmosphere system. Based on the ECMWF operational numerical weather prediction system, ERA5 incorporates a coupled land-atmosphere assimilation scheme that provides reliable datasets for our SM study. Since the original ERA5 data are available at an hourly resolution, they are used directly for our sub-daily SM retrieval.

Soil moisture (SM). ERA5 provides volumetric SM (m³ m⁻³) at 0–7, 7–28, 28–100, and 100–289 cm depths. To ensure consistency with ASCAT satellite measurements, we focus on the topmost layer of 0–7 cm, commonly referred to as surface SM. We use this variable as our primary target; owing to its spatial coherence and overall accuracy, it serves as a widely adopted benchmark in the development of machine-learning-based SM retrievals .

Soil temperature (ST). ERA5 also offers ST at the same four depth intervals. As with ERA5 SM, we used the 0–7 cm layer to align with the surface sensitivity of the satellite observations. ST is considered an important auxiliary variable in SM retrieval, given its indirect relationship with soil water content through thermal properties .

Leaf area index (LAI). Vegetation structure significantly influences ASCAT backscatter-incidence angle . Additionally, many previous studies have shown the usefulness of vegetation indices in SM retrieval . Among various vegetation indices – such as the normalized difference vegetation index (NDVI) and enhanced vegetation index (EVI) – we selected the LAI. To maintain consistency with the ECMWF assimilation framework, we use the LAI forcing from ERA5 rather than externally sourced satellite datasets . While ERA5 does not simulate dynamic LAI or assimilate LAI observations – relying instead on a prescribed monthly climatology – using this specific background was necessary to ensure strict physical consistency with our training target.

The use of ERA5-derived ST and LAI as auxiliary inputs alongside ASCAT backscatter to predict SM introduces some dependence on the ERA5 framework. However, this design is both physically and methodologically justified. Physically, ST and LAI provide crucial boundary conditions that help the model separate SM effects from vegetation attenuation and thermodynamic variability in the ASCAT signal. Methodologically, they ensure the retrieval remains consistent with the ERA5 background space, a standard requirement for data assimilation . While the product inherits ERA5's large-scale statistics, its sub-daily and event-scale variations are driven by the independent ASCAT observations, capturing dynamic information absent from the ancillary variables alone.

We used in situ SM data from the ISMN , which can be downloaded from https://ismn.earth/en/data/ (last access: 3 March 2025), to evaluate retrieval performance in 2019. The ISMN is a collaborative initiative supported by several international organizations and sustained through funding from the European Space Agency (ESA) and voluntary contributions from scientists and monitoring networks worldwide. It provides a consistent, globally accessible database of in situ SM measurements and serves as a key reference for validating both model-based and satellite-derived SM products . This ISMN dataset was completely withheld from training, which enables a robust, independent evaluation of whether the retrieval effectively leverages ASCAT observations to generate physically meaningful SM estimates beyond the ERA5 background.

Within the study area, SM data from 568 in situ sites were extracted from three major networks: the Soil Climate Analysis Network (SCAN, ); the SNOwpack TELemetry (SNOTEL, ); and the United States Climate Reference Network (USCRN, ). Although many stations report SM profiles at multiple depths (up to 2 m), only the topmost measurement – taken at 0 to 7 cm depth – was used in this study for validation to align with the depth-sensitivity of ASCAT and ERA5 data. Sites with more than 100 d of missing observations in 2019 were excluded from the analysis. The in situ measurements are reported at hourly intervals and expressed in volumetric units (m³ m⁻³).

CNNs, a class of deep learning models, were originally designed for tasks such as image and speech recognition . Due to their ability to efficiently process data with spatial or grid-like structures (i.e., images), CNNs have become increasingly popular in remote sensing applications for classification, segmentation, or retrieval ; see for a meta-analysis.

CNNs operate on input data structured as multidimensional tensors. Here, input images are represented as a tensor X of dimensions (height) × (width) × (depth), where the height and width correspond here to 100×200 pixels, representing the spatial grid of latitude and longitude cells over the CONUS. The depth corresponds to the number of input channels, such as the backscattering coefficient σ40 and other auxiliary variables. A CNN transforms these inputs through a series of layers, typically composed of convolution operations and nonlinear activation functions.

The core component of a CNN is the convolutional layer, which applies learnable kernels or filters – small matrices of weights, typically of size of 3×3× depth or 5×5× depth – across the input tensor. Each kernel performs an element-wise multiplication followed by a summation, effectively extracting localized spatial features. The result of this operation is a feature map. Multiple kernels are used per layer to learn diverse features.

Activation functions are applied after each convolution to introduce nonlinearity, which is crucial for learning complex patterns. The Rectified Linear Unit (ReLu), defined as f(x)=max⁡(0,x), is often used due to its computational efficiency and ability to mitigate vanishing gradient issues .

In addition, several architectural parameters influence the operations and outputs of convolutional layers:

Stride determines the number of pixels the kernel moves at each step. Larger strides reduce the output resolution.

While standard convolutions rely on weight sharing across the entire input, locally connected layers relax this constraint by applying distinct filters at every spatial location . This allows the network to learn location-specific patterns and achieve finer spatial specialization, driving their recent adoption in remote sensing applications .

We consider a localized CNN variant, incorporating a locally connected convolutional layer in which filters learn distinct weights for each spatial location. To preserve spatial dimensions, zero-padding is applied prior to a single 5×5 locally connected layer without weight sharing. This localized architecture has been shown to perform particularly well in extreme cases , motivating its use here for the sub-daily estimates. A detailed flowchart illustrating the proposed retrieval framework is provided in Fig. .

The input to the model comprises ASCAT-derived σ40, along with two auxiliary variables – ST and LAI – identified in Sect.  as highly relevant to the target SM variable. While ST and LAI were selected for this study, other auxiliary variables, such as the antecedent precipitation evaporation index and the amplitude of the diurnal cycle of surface temperature , may also contribute to SM retrieval. However, the selection and ranking of such variables fall outside the scope of this work.

Because the CONUS-scale dataset rendered computationally intensive automated optimization impractical , model hyperparameters were selected using a structured manual tuning approach. Consistent with methodologies established in recent remote sensing studies , parameters were initially anchored to standard literature benchmarks and systematically refined against validation metrics until further adjustments yielded negligible improvements. Ultimately, our model is trained using the Adam optimizer with an initial learning rate of 0.001 and an epsilon value of 1e-7 for numerical stability. Weight initialization is controlled using the Glorot Uniform initialization with a fixed random seed to ensure reproducibility. A batch size of 16 is used in the training. In addition, separate models are trained for ascending and descending satellite passes.

To robustly evaluate the model and actively prevent overfitting, we employed a strict temporal data splitting strategy alongside dynamic training constraints. The dataset spanning 2016 to 2018 is partitioned into 80 % for training and 20 % for validation, while the 2019 data is completely held out for independent testing and performance evaluation. During the training phase, overfitting is further mitigated through the implementation of early stopping; while the model is allowed to train for a maximum of 200 epochs, the process is halted early if the validation loss demonstrates no improvement over 5 consecutive epochs, at which point the optimal weights are restored.

To evaluate the performance of SM retrievals, we used four statistical evaluation metrics: Pearson's correlation coefficient (r, unitless), root mean square error (RMSE, m³ m⁻³), and bias (Bias, m³ m⁻³). These metrics are computed as follows: 1r=∑i=1N(ypred,i-ypred‾)(yref,i-yref‾)∑i=1N(ypred,i-ypred‾)2∑i=1N(yref,i-yref‾)22RMSE=∑i=1N(ypred,i-yref,i)2N3Bias=∑i=1N(ypred,i-yref,i)N, where ypred,i is the predicted SM, yref,i is the reference data (i.e., ERA5 or in situ SMs), N is the number of samples of SM data, and ypred‾ is the mean value of the predicted SM data. The standard deviation (SD, m³ m⁻³) of the difference between two datasets is also reported: 4SD=∑i=1N(di-d‾)2N-1 where di=ypred,i-yref,i is the difference between the predicted and reference SMs, and d‾=1N∑i=1Ndi.

Figure  illustrates the ERA5-o and CNN-lo SM estimates and the difference between ASCAT-based retrievals (CNN-o and the H120 product) with respect to ERA5-o for multiple time slots on 20 January 2019, over ascending passes. Although separate models were trained for ascending and descending orbits, both are analyzed to maximize sub-daily coverage. In this example, five ascending and five descending (not shown) passes covered the study area, with each grid cell typically receiving up to two observations per orbit type – still limited, but sufficient to explore intraday variability under certain conditions.

The CNN-derived SM estimates show strong spatial consistency with the ERA-o reference, effectively capturing similar patterns and value ranges across all observed orbits (Fig. , first and second columns). This agreement is further reflected in the relatively small differences observed in the third column, where CNN-lo deviates minimally from ERA5-o. On the other hand, H120-o shows substantially larger discrepancies relative to ERA5-o, as indicated by the prevalence of dark red and blue regions in the fourth column. It should be noted that H120 was not trained on ERA5 and is therefore more independent of it than CNN-lo. Furthermore, the absolute spatial differences between H120-o and ERA5-o would be notably reduced if H120-o were rescaled to the ERA5 climatology using a cumulative distribution function (CDF) matching technique, rather than the current scaling based on saturated moisture content. When assessed over the full set of available orbits for the year 2019, CNN-lo achieves high correlation with ERA5-o (r=0.92 for ascending and r=0.91 for descending passes), confirming its ability to capture SM dynamics at sub-daily scales. In contrast, the H120 product achieves significantly lower correlations (r=0.58 and 0.59, respectively). Because ERA5 provides the most reliable estimates of SM , achieving comparable results using only ASCAT data in our retrieval is a promising outcome. For a more independent assessment, we next compare all products against in situ data.

We evaluated three SM estimates (ERA5-o, CNN-lo, and H120-o) against measurements from 568 ISMN stations across CONUS in 2019 (see Sect. ). Each ISMN station was collocated to the nearest 0.25° grid cell to match the resolution of the gridded SM estimates. Table  presents the performance metrics, and Fig. a presents probability density functions (PDFs) of temporal correlation (r), bias (Bias, m³ m⁻³), and standard deviation (SD, m³ m⁻³). ERA5-o achieves the highest overall agreement with in situ data (median r=0.75, Bias =0.056, and SD =0.061), which is expected given that ERA5 assimilates a wide range of satellite observations and in situ measurements. ERA reanalysis has long been shown to yield higher SM accuracy than any individual satellite retrieval (see, for instance, ). The H120 product shows the lowest median correlation (r=0.59) and exhibits a long negative tail in the bias distribution, despite having the lowest median bias error (0.024 m³ m⁻³). Its higher error variability (SD) indicates reduced skill in capturing SM dynamics. CNN-lo significantly outperforms H120, achieving a median correlation of 0.65 and error levels comparable to ERA5, as shown in the distributions in Fig. a.

The spatial distribution of correlation coefficients (Fig. b) indicates that CNN-lo generally performs well, with particularly strong agreement in the western and southeastern regions. The difference map of Δr (middle panel) demonstrates that CNN-lo improves upon H120-o in a large part of the domain, as indicated by the predominance of blue points. Quantitatively, CNN-lo outperforms H120-o at 352 of 568 sites, about 62 % (green area in the last panel of Fig. b). The PDF of Δr further confirms this improvement, showing a pronounced positive tail approaching 2. This pattern reflects substantial performance gains at many sites where H120-o exhibits a low or even negative correlation. Although about 38 % of the sites show a reduced correlation, the magnitude of these degradations is generally much smaller than the observed improvements.

Evaluating performance across major CONUS land-cover classes (Fig. ) confirms trends seen in Table  and Fig. : while H120-o achieves lower biases across all classes – an artifact of local calibration – it yields higher standard deviations. Regionally, H120-o shows marginally better correlations than CNN-lo for croplands, reflecting results over the central Great Plains in Fig. b. Conversely, CNN-lo excels over complex topography (e.g., the mountainous western US), outperforming H120-o in correlation over evergreen and deciduous forests due to its ability to leverage spatial context . Despite being trained on ERA5, CNN-lo outperforms its training target over crops and grasslands in terms of correlation.

To better illustrate the comparison at the site level, Fig.  shows hourly SM time series for six randomly selected ISMN sites. All products capture the temporal dynamics and amplitude of SM reasonably well. ERA5-o consistently achieves the highest correlations (>0.8), reflecting the advantage of assimilating multiple data sources. CNN-lo closely tracks the ERA5 SM ranges and temporal patterns. The H120 product exhibits higher temporal variability than the other retrievals.

To illustrate the capacity of CNN-lo to capture short-term SM dynamics, we analyzed a day with a significant precipitation event: 20 November 2019. Figure a displays spatial maps of daily SM amplitude (ΔSM) derived from CNN-lo and ERA5-o. For reference, the full 24 h SM amplitude of ERA5 (ERA5-24h) and the corresponding daily accumulated precipitation are also represented. High ΔSM values appear in the southwest region, coinciding with a precipitation event captured in the ERA5 precipitation data.

We then examined the hourly SM time series in a representative grid cell located at [34° N, 112.75° W], shown in red in Fig. a. To independently evaluate the SM dynamics, we incorporated the MRMS gauge-corrected precipitation product, spatially aggregated to match our target grid cell. As shown in Fig. b, according to both MRMS and ERA5 precipitation datasets, the rain started around 04:00 UTC and continued throughout the day. Prior to rainfall (00:00–04:00 UTC), ERA5 SM values remained low, then gradually increased, reaching approximately 0.42 m³ m⁻³ by the end of the day.

On this day, ASCAT provided three overpasses: two during the ascending orbit (04:00–05:00 UTC) and one during the descending orbit (17:00 UTC). CNN-lo retrieved SM estimates for these overpasses, showing good agreement with ERA5. The CNN-lo daily SM amplitude (ΔSM=0.2 m³ m⁻³) is significant but lower than ERA5-24h (0.31 m³ m⁻³). The errors are due to (1) retrieval uncertainties and (2) limitations in temporal sampling.

While a single case study cannot serve as a formal demonstration of the model's overall capability, this example illustrates the model's ability to capture intraday SM dynamics during heavy precipitation events, despite sparse ASCAT observations. Crucially, the alignment with independent MRMS precipitation data confirms that the retrieved SM response is physically consistent with actual observed rainfall, rather than merely reflecting the ERA5 forcing. Capturing these dynamics is critical for hydrological applications, as rapid soil saturation can provoke flood risks. However, the spatial depiction of intraday variability remains strongly constrained by ASCAT's sampling frequency. Integrating observations from multiple instruments (e.g., SMOS, SMAP) is expected to enhance both retrieval accuracy and temporal resolution in future work.

The key question is: What level of intraday SM variation can be reliably detected from NN-based retrieval? To address this, we compared the diurnal SM amplitude (ΔSM) from CNN-lo with ERA5-o for all pixels and days in 2019 having at least two ASCAT observations. Results are shown in Fig. .

When considering all 4 419 791 samples (i.e., threshold is none), the correlation between CNN-lo and ERA5-o ΔSM is modest (r=0.21), indicating that most observed differences reflect retrieval noise rather than true diurnal variability. This is expected given ASCAT's sparse temporal sampling and the inherent difficulty of resolving small sub-daily changes. However, as shown in Sect. , to isolate such cases, we progressively applied precipitation thresholds. For days with any measurable rainfall (pt>0 mm), correlation improves slightly to 0.22. For heavier events (pt>10 mm), correlation rises to 0.32, and further increases to 0.47 when excluding cases with rain on the previous day (pt-1=0 mm). This subset (14 445 samples) primarily reflects isolated heavy rainfall events, suggesting that CNN-lo can capture SM responses when the signal-to-noise ratio is favorable.

Figure  shows the scatter plots for this subset (pt>10 mm, pt-1=0 mm), comparing CNN-lo and ERA5-o for: (1) maximum SM value (SM_max), (2) minimum SM value (SM_min), and (3) diurnal SM amplitude (ΔSM). Each point represents one single pixel-day. CNN-lo tends to underestimate SM_max and slightly overestimate SM_min – a typical behavior of statistical models that dampen extremes . Despite this, strong agreement with ERA5-o is achieved for SM_max and SM_min (r=0.82 and r=0.88, respectively). As expected, ΔSM exhibits lower correlation (r=0.47) because amplitude errors propagate from both extremes. The third panel of Fig.  indicates that the regression fit (dashed red line) is skewed by high-amplitude cases. However, the denser cluster of points in the 0–0.05 m³ m⁻³ range (i.e., yellow points) suggests that the model captures stable SM conditions well. These low-amplitude cases likely correspond to days with minimal diurnal variation, where the model performs more reliably.

We now examine the spatial and temporal structure of these intraday variations over the CONUS. Figure  summarizes key intraday SM signals for the 14 445 cases identified above: timing of SM_max and SM_min, their respective values, and the diurnal amplitude ΔSM for CNN-lo, ERA5-o, and ERA5-24h. Note that higher precipitation is concentrated in the eastern CONUS, resulting in limited sampling in western regions (white pixels).

CNN-lo and ERA5-o exhibit similar spatiotemporal patterns. Note, however, that the estimation of the SM maximum and minimum times is degraded when limited by the ASCAT time sampling. For CNN-lo and ERA5-o, in the eastern part, SM typically reaches its maximum in the late afternoon (around 17:00 UTC) and its minimum in the early morning (around 03:00 UTC), aligning with ASCAT overpass times. In contrast, ERA5-24h – benefiting from full hourly coverage – shows SM_max occurring later in the evening and SM_min earlier, indicating a more complete description of the diurnal cycle.

Maps of SM_max and SM_min are largely consistent across CNN-lo and ERA-o, supported by the high correlation reported in Fig. . However, CNN-lo underestimates ΔSM compared to ERA5-o, consistent with the tendency of statistical models to dampen variability . Interestingly, ERA5-o also exhibits lower amplitudes relative to ERA5-24h, which has a broader dynamic range due to full diurnal coverage. Therefore, the underestimation by CNN-lo likely reflects both retrieval smoothing and ASCAT's temporal sampling limitations – a constraint that could be mitigated through multi-sensor data fusion, as discussed in Sect. .