Data matching

Data Matching Skill

Overview

Successfully matching observations to simulations requires careful handling of datetime and depth coordinates. The matching must use exact values with proper rounding—no interpolation or nearest-neighbor approximation.

Observation Data Format

CSV with columns:

datetime,depth,temp,OXY_oxy
2009-01-21 12:00:00,0,0.1,16.3
2009-01-21 12:00:00,1,0.7,16.3
...

• datetime: ISO format timestamp
• depth: Measured depth in meters
• temp: Water temperature in °C
• OXY_oxy: Oxygen (not used for temperature RMSE)

Simulation Output Characteristics

GLM output:

• Time dimension: Regular hourly intervals from simulation start
• Depth dimension: Variable number of layers based on model dynamics
• Temperature: Simulated at each time step and depth layer

Exact Matching Algorithm

Step 1: Load Observations

import pandas as pd

obs_df = pd.read_csv('/root/field_temp_oxy.csv')
obs_df['datetime'] = pd.to_datetime(obs_df['datetime'])

Step 2: Round Depths

Round observation depths to nearest meter (standard practice):

obs_df['depth_rounded'] = obs_df['depth'].round(0)

Step 3: Extract Simulation Data

import netCDF4 as nc
from netCDF4 import num2date

ds = nc.Dataset('/root/output/output.nc')
temp_sim = ds.variables['temp'][:]      # [time, depth]
z_sim = ds.variables['z'][:]            # depth coordinates
time_sim = ds.variables['time'][:]      # time values

# Convert time to datetime
time_var = ds.variables['time']
dates_sim = num2date(time_sim, time_var.units)

ds.close()

Step 4: Exact Matching

def exact_match(obs_df, temp_sim, z_sim, dates_sim):
    """
    Match observations to simulation using exact datetime and rounded-depth

    Returns: aligned arrays of simulated temps, observed temps,
             and metadata for filtering
    """
    import numpy as np

    matched = {
        'sim_temp': [],
        'obs_temp': [],
        'depth': [],
        'datetime': [],
        'obs_idx': []
    }

    for idx, row in obs_df.iterrows():
        obs_date = row['datetime']
        obs_depth = row['depth_rounded']
        obs_temp = row['temp']

        # Find time index: exact datetime match
        time_idx = None
        for i, sim_date in enumerate(dates_sim):
            if sim_date == obs_date:
                time_idx = i
                break

        if time_idx is None:
            continue  # No exact datetime match

        # Find depth index: exact depth match
        depth_idx = None
        for j, sim_z in enumerate(z_sim):
            if np.isclose(sim_z, obs_depth, atol=0.01):
                depth_idx = j
                break

        if depth_idx is None:
            continue  # No exact depth match

        # Record match
        matched['sim_temp'].append(temp_sim[time_idx, depth_idx])
        matched['obs_temp'].append(obs_temp)
        matched['depth'].append(obs_depth)
        matched['datetime'].append(obs_date)
        matched['obs_idx'].append(idx)

    return matched

Quality Checks

Missing Matches

# Check what percentage of observations were matched
total_obs = len(obs_df)
matched_count = len(matched['sim_temp'])
match_fraction = matched_count / total_obs
print(f"Matched {matched_count}/{total_obs} observations ({100*match_fraction:.1f}%)")

Temporal Coverage

# Check date range of matches
import pandas as pd
match_dates = pd.DataFrame(matched['datetime'])
print(f"Match date range: {match_dates.min()} to {match_dates.max()}")

Depth Distribution

# Check which depths are represented
import numpy as np
matched_depths = np.array(matched['depth'])
unique_depths = np.unique(matched_depths)
print(f"Matched depths: {sorted(unique_depths)}")

Filtering for Metrics

After exact matching, apply semantic filters:

import numpy as np

matched_sim = np.array(matched['sim_temp'])
matched_obs = np.array(matched['obs_temp'])
matched_depths = np.array(matched['depth'])
matched_dates = np.array(matched['datetime'])

# Overall RMSE: all matches
overall_mask = np.ones(len(matched_sim), dtype=bool)

# Annual deep (depths >= 13m)
annual_deep_mask = matched_depths >= 13

# Summer deep (June-Sept, depths >= 13m)
summer_mask = np.array([d.month in [6, 7, 8, 9]
                        for d in matched_dates])
summer_deep_mask = summer_mask & (matched_depths >= 13)

# Calculate RMSE for each category
from numpy import sqrt, mean
overall_rmse = sqrt(mean((matched_sim - matched_obs)**2))
annual_deep_rmse = sqrt(mean((matched_sim[annual_deep_mask] -
                              matched_obs[annual_deep_mask])**2))
summer_deep_rmse = sqrt(mean((matched_sim[summer_deep_mask] -
                              matched_obs[summer_deep_mask])**2))

Common Pitfalls

1. Rounding inconsistency: Always round observation depths the same way
2. Timezone issues: Ensure times are in same timezone before comparison
3. Nearest-neighbor fallback: Must use exact matches only
4. Interpolation: Do NOT interpolate simulation to observation depths/times