Patient record entity resolution

Patient & Provider Entity Resolution

Overview

The same patient, provider, or facility appears many times across claims, eligibility, and clinical systems under slightly different names, addresses, and IDs. This skill links those records into a single resolved entity and produces a golden record. It implements the full entity-resolution pipeline blocking, comparison, probabilistic scoring, clustering, and survivorship using production-grade open-source tooling.

When to Use This Skill

• Building or cleaning a Master Patient Index (MPI/EMPI) or provider master.
• Deduplicating a member or provider roster before analytics or capitation payment.
• Crosswalking entities across payers/vendors that lack a shared key.
• Reducing "duplicate patient" safety risks or double-counting in reporting.
• Any general record linkage / entity resolution problem on tabular data.

Method

The core is the Fellegi–Sunter probabilistic model: for each candidate pair, combine field-level agreement/disagreement into a match weight using learned m and u probabilities, then threshold into match / possible-match / non-match.

1. Standardize fields first (names, addresses via usaddress, dates, phone, gender); normalize provider IDs and run medical-ontology-code-mapping for any coded fields.
2. Block to avoid the O(n²) all-pairs blow-up: index candidate pairs by cheap keys (e.g., soundex(last_name) + dob_year, or zip + first_initial). Critical at scale.
3. Compare within blocks using Jaro-Winkler / Levenshtein / token-set similarity per field; handle nulls explicitly.
4. Score pairs with the Fellegi-Sunter weights (Splink learns m/u via EM, so no labeled data is strictly required).
5. Cluster matched pairs into entities (connected components / graph resolution) and review the borderline band manually or with active learning.
6. Survivorship build the golden record per cluster (most-recent, most-complete, or source-priority rules) and keep full lineage back to source rows.

Example

# Probabilistic linkage with Splink (DuckDB backend)  scales to millions of rows
from splink.duckdb.linker import DuckDBLinker
import splink.duckdb.comparison_library as cl

settings = {
    "link_type": "dedupe_only",
    "blocking_rules_to_generate_predictions": [
        "l.dob = r.dob and substr(l.last_name,1,1) = substr(r.last_name,1,1)",
        "l.zip = r.zip",
    ],
    "comparisons": [
        cl.jaro_winkler_at_thresholds("first_name", [0.9, 0.7]),
        cl.jaro_winkler_at_thresholds("last_name",  [0.9, 0.7]),
        cl.exact_match("dob"),
        cl.levenshtein_at_thresholds("address", 2),
    ],
}
linker = DuckDBLinker(df, settings)
linker.estimate_u_using_random_sampling(max_pairs=1e6)
linker.estimate_parameters_using_expectation_maximisation(
    "l.dob = r.dob and l.last_name = r.last_name")
pairs = linker.predict(threshold_match_probability=0.95)
clusters = linker.cluster_pairwise_predictions_at_threshold(pairs, 0.95)

# Lightweight alternative for small data: recordlinkage
import recordlinkage as rl
idx = rl.Index(); idx.block("dob_year")
cand = idx.index(df)
cmp = rl.Compare()
cmp.string("last_name","last_name", method="jarowinkler", threshold=0.85, label="ln")
cmp.exact("dob","dob", label="dob")
features = cmp.compute(cand, df)
matches = features[features.sum(axis=1) >= 2]

Evaluation

Report precision, recall, and F1 against a labeled gold set when available; otherwise use clerical review of a sampled borderline band. Track the match-rate and duplicate-collapse rate, and always expose a tunable threshold so reviewers can trade precision vs. recall (false-merge of two real patients is a safety event bias toward precision and route ambiguous pairs to review).

Outputs

• entity_clusters.parquet source row → resolved entity_id.
• golden_records.parquet one survivorship record per entity with lineage.
• match_pairs.parquet scored pairs with per-field contributions (explainable).
• linkage_report.md blocking stats, score distribution, threshold, P/R/F1.

Healthcare Context

Optimized for patient/member and provider/NPI matching, including the hard cases: nicknames, maiden/married names, twins sharing an address, transposed DOBs, and group-vs- individual NPIs. Upstream: healthcare-data-quality-profiling. Pairs with spark-healthcare-data-pipeline for billion-row deployment.

References

• Splink (UK MoJ) https://moj-analytical-services.github.io/splink/
• Python Record Linkage Toolkit https://recordlinkage.readthedocs.io
• Fellegi, I. & Sunter, A. (1969), A Theory for Record Linkage, JASA.