Named Entity Recognition

Also Known As

TL;DR

Explanation

Named Entity Recognition (NER) is the task of scanning unstructured text, locating the contiguous spans that refer to named real-world entities, and tagging each span with a semantic type. Classic type sets include PERSON, ORGANIZATION, LOCATION, DATE, TIME, MONEY, and PERCENT, though domain-specific schemes add types like GENE, DRUG, or PRODUCT. NER answers two questions at once: where does an entity mention start and end (the boundary or span problem), and what kind of thing is it (the classification problem). A sentence like "Tim Cook joined Apple in 1998" should yield Tim Cook as PERSON, Apple as ORGANIZATION, and 1998 as DATE.

It is important to draw boundaries around what NER is not. NER does not decide that two different mentions refer to the same entity - that is coreference resolution. It does not link a mention to a canonical record in a knowledge base - that is entity linking or entity resolution. And it does not extract the relationship between entities, for example that Tim Cook works for Apple - that is relation extraction. NER is the upstream span-detection-and-typing step that those later tasks consume.

Approaches range from rule and gazetteer based systems (regexes plus curated name lists) to statistical sequence models. The dominant modern formulation treats NER as sequence labeling using a BIO or BIOES tagging scheme, where each token is tagged Begin, Inside, or Outside of an entity of a given type. Conditional random fields, BiLSTM-CRF models, and now transformer encoders such as BERT fine-tuned for token classification produce these tags. Quality is measured with span-level precision, recall, and F1, scored on exact boundary and type match rather than per-token accuracy, because a span that is partially correct still misleads downstream consumers.

The practical difficulties are boundary ambiguity (is "New York Times" a LOCATION or an ORGANIZATION, and where does the span end), type ambiguity for the same surface form ("Washington" as person, city, or state), nested and overlapping entities, and domain drift where a model trained on news fails on clinical notes. Robust pipelines pin a clear type schema, evaluate at the span level, and budget for domain-specific labeled data rather than assuming a general model transfers cleanly.

Common Misconception

Why It Matters

Common Mistakes

• Scoring per-token accuracy instead of span-level precision, recall, and F1, which masks partial-boundary failures that break downstream consumers.
• Conflating NER with entity linking or coreference and expecting a tagger to deduplicate or canonicalize mentions it never resolves.
• Relying on a model trained on one domain (such as news) for a very different one (such as clinical or legal text) without re-labeling data.
• Choosing a flat tagging scheme that cannot represent nested or overlapping entities the domain actually contains.
• Letting the type schema drift so the same surface form gets inconsistent labels across the dataset.

Avoid When

• A small fixed vocabulary of entities can be matched exactly with a curated gazetteer, making a statistical model unnecessary overhead.
• The downstream task actually needs relationships or canonical identities, in which case NER alone will not deliver and you need relation extraction or entity linking.
• The text is already structured (such as tagged fields or a database) so entities are explicit and no extraction is required.
• You cannot obtain or label representative domain data and a general model would mislabel your entities.

When To Use

• Turning free text such as articles, emails, or support tickets into structured, typed entity mentions for indexing or analytics.
• Building the upstream extraction stage of a knowledge graph or entity resolution pipeline.
• Redacting or detecting sensitive entities like names and locations in documents.
• Powering faceted or semantic search where users filter by recognized people, organizations, or places.

Code Examples

# Naive NER: a single capitalized-word heuristic with no type or boundary handling
import re

def extract_entities(text):
    # Treats every capitalized token as an entity, gives no type label,
    # and cannot join multi-token spans like 'New York Times'.
    return re.findall(r"\b[A-Z][a-z]+\b", text)

sentence = "Tim Cook joined Apple in 1998 in New York."
print(extract_entities(sentence))
# -> ['Tim', 'Cook', 'Apple', 'New', 'York']
# 'Tim Cook' is split, no PERSON/ORG/LOCATION type, '1998' is missed entirely.

# Span-level NER with typed labels using a sequence model, then span-level scoring.
import spacy

nlp = spacy.load("en_core_web_sm")  # token-classification model

def extract_entities(text):
    doc = nlp(text)
    # Each ent carries the full span (start_char..end_char) and a semantic type.
    return [(ent.text, ent.label_, ent.start_char, ent.end_char)
            for ent in doc.ents]

sentence = "Tim Cook joined Apple in 1998 in New York."
for text, label, start, end in extract_entities(sentence):
    print(f"{text!r}\t{label}\t[{start}:{end}]")
# -> 'Tim Cook'  PERSON   [0:8]
#    'Apple'     ORG      [16:21]
#    '1998'      DATE     [25:29]
#    'New York'  GPE      [33:41]

def span_f1(gold, pred):
    # Score on exact (span, type) match, not per-token overlap.
    g, p = set(gold), set(pred)
    tp = len(g & p)
    precision = tp / len(p) if p else 0.0
    recall = tp / len(g) if g else 0.0
    return 2 * precision * recall / (precision + recall) if precision + recall else 0.0

Tags

Related Terms

References

• https://web.stanford.edu/~jurafsky/slp3/8.pdf
• https://aclanthology.org/W03-0419/
• https://spacy.io/usage/linguistic-features#named-entities