Clinical nlp entity extraction

Clinical NLP Entity Extraction

Overview

Most clinical signal lives in free text. This skill converts notes into structured rows by running a clinical NER + assertion + normalization pipeline: find the entities, decide whether they are negated/hypothetical/historical, and link them to standard concept IDs. It is the text-mining workhorse that feeds cohort selection, feature engineering, and weak-supervision labeling.

When to Use This Skill

• Pulling problems, medications (with dose/route/frequency), labs, or procedures out of notes into a table.
• Building a cohort from text criteria ("patients with active CHF, not ruled out").
• Generating features for healthcare-predictive-modeling from narrative text.
• Pre-labeling candidates for snorkel-weak-supervision-labeling.
• De-noising and structuring an EHR text feed for analytics.

Pipeline

1. Sectionize detect note sections (HPI, PMH, Medications, Assessment/Plan) so context is preserved.
2. NER extract entities with a clinical model (medspaCy rules + scispaCy en_core_sci_md, or a transformer like Bio_ClinicalBERT fine-tuned for NER).
3. Assertion / context apply ConText/NegEx to tag negated, historical, hypothetical, family, uncertain. "No chest pain" must never become a positive.
4. Relation / attribute linking attach dose/route/frequency to a drug, severity to a problem, value/unit to a lab.
5. Concept normalization link spans to UMLS CUIs / SNOMED / RxNorm (hand coded output to medical-ontology-code-mapping for crosswalks).
6. Emit a tidy entity table with offsets, assertion, and concept IDs.

Example

import medspacy
from medspacy.ner import TargetRule

nlp = medspacy.load()                       # tokenizer + NER + ConText
nlp.get_pipe("medspacy_target_matcher").add([
    TargetRule("congestive heart failure", "PROBLEM"),
    TargetRule("metformin", "MEDICATION"),
])

doc = nlp("No congestive heart failure. Started metformin 500 mg BID for T2DM.")
for ent in doc.ents:
    print(ent.text, ent.label_,
          "NEGATED" if ent._.is_negated else "PRESENT",
          "HISTORICAL" if ent._.is_historical else "CURRENT")
# congestive heart failure PROBLEM NEGATED CURRENT
# metformin MEDICATION PRESENT CURRENT

# scispaCy UMLS entity linking
import spacy
from scispacy.linking import EntityLinker
nlp = spacy.load("en_core_sci_md")
nlp.add_pipe("scispacy_linker", config={"linker_name": "umls", "resolve_abbreviations": True})
doc = nlp("Patient with diabetes mellitus on insulin.")
for e in doc.ents:
    cui, score = e._.kb_ents[0]
    print(e.text, "->", cui, round(score,2))

Evaluation

Score entity extraction with strict and lenient span P/R/F1 against an annotated set, and evaluate assertion accuracy separately (negation errors are the most common and most costly failure mode). Track per-entity-type performance medications and labs usually beat problems. Validate on your own note style; public models drift on local templates.

Outputs

• entities.parquet note_id, span text, offsets, type, assertion, CUI/RxCUI, attributes.
• medications.parquet drug, dose, route, frequency, status (a structured med list).
• cohort_flags.csv per-patient boolean criteria derived from asserted entities.
• extraction_report.md entity counts, assertion distribution, sample QA.

Healthcare Context

Handles the clinical-text realities: heavy abbreviation, negation ("denies", "r/o"), templated boilerplate, and family-history confounds. De-identify (Philter/hipaa- compliance) before processing PHI. Upstream text-mining peer to clinical-text-summarization and clinical-text-search-elk.

References

• medspaCy https://github.com/medspacy/medspacy
• scispaCy https://allenai.github.io/scispacy/
• Harkema et al. (2009), ConText assertion algorithm; i2b2/n2c2 NLP challenge corpora.