Stemming & Lemmatization

Also Known As

TL;DR

Explanation

Stemming and lemmatization are two techniques for collapsing morphological variants of a word into a shared token, dramatically improving recall in a search index. Both run during analysis - typically in the same pipeline as lowercasing, stopword removal, and tokenisation - so the indexed term and the query term end up in the same normalised form. Without them, a search for 'running shoes' misses documents titled 'shoes for runners', which is almost never what users want.

Stemming is rule-based. Algorithms like Porter, Snowball, or Lancaster chop suffixes off using heuristic rules: 'running' -> 'run', 'happiness' -> 'happi', 'ponies' -> 'poni'. Stemmers are fast, language-specific, and often produce non-words - the stem is a shared token, not a dictionary entry. They over-stem ('universal' and 'university' collapse to 'univers') and under-stem ('ran' and 'run' stay separate because Porter cannot handle irregular verbs). Trade-off: high recall, some precision loss.

Lemmatization is dictionary-based. It uses a morphological lexicon plus part-of-speech context to map each token to its canonical dictionary form (lemma): 'ran' -> 'run', 'better' -> 'good', 'mice' -> 'mouse'. Results are real words and handle irregular forms correctly, but it is slower, needs POS tagging for accuracy, and depends on quality language resources (spaCy, WordNet, Stanford CoreNLP). Trade-off: higher precision, more infrastructure.

Most search engines default to stemming because it is cheap and available for many languages out of the box. Elasticsearch ships snowball, porter_stem, and language-specific analyzers; Meilisearch does light normalisation; Solr exposes both stemmers and lemmatizers via filter chains. Apply the same analyzer to indexing and querying, otherwise 'runs' at query time never matches 'run' in the index. Test with real queries: aggressive stemming ruins domain vocabulary (part numbers, brand names, code identifiers), so protect those tokens with a keyword field or a stemmer exclusion list.

Common Misconception

Why It Matters

Common Mistakes

• Applying stemming at index time but not at query time (or vice versa), so normalised tokens never match.
• Using an aggressive stemmer on domain terms like SKUs, brand names, or code identifiers, corrupting exact matches.
• Treating stemming and lemmatization as synonyms and reaching for whichever the engine defaults to without measuring recall or precision.
• Using an English stemmer on multilingual content - Porter destroys German, French, or Turkish morphology.
• Skipping a keyword sub-field for exact-match cases (facets, IDs, filters) that must not be stemmed.

Avoid When

• Exact-match fields like IDs, SKUs, email addresses, or code tokens where any normalisation breaks lookup.
• Very short controlled vocabularies (tags, enum values) where inflection does not occur.
• Case-sensitive or symbol-heavy domains such as chemistry formulas or programming identifiers.
• Languages without good stemmer or lemmatizer support in your stack - a bad analyzer is worse than none.

When To Use

• Long-form natural language content (articles, product descriptions, help docs) where users type inflected queries.
• Search recall is visibly poor - users find nothing when the answer clearly exists in a different tense or number.
• Multi-lingual search where each language field can get its own tuned analyzer.
• You can afford lemmatization infrastructure and need high precision on irregular forms like 'ran' -> 'run' or 'better' -> 'good'.

Code Examples

// Elasticsearch: mismatched analyzers - stemming at index only.
$params = [
    'index' => 'articles',
    'body'  => [
        'settings' => [
            'analysis' => [
                'analyzer' => [
                    'index_analyzer' => [
                        'tokenizer' => 'standard',
                        'filter'    => ['lowercase', 'porter_stem'],
                    ],
                    // Query analyzer forgot the stemmer:
                    'query_analyzer' => [
                        'tokenizer' => 'standard',
                        'filter'    => ['lowercase'],
                    ],
                ],
            ],
        ],
        'mappings' => [
            'properties' => [
                'body' => [
                    'type'            => 'text',
                    'analyzer'        => 'index_analyzer',
                    'search_analyzer' => 'query_analyzer',
                ],
            ],
        ],
    ],
];
// Indexed 'running' -> 'run'. Query 'running' stays 'running'. No match.

// Same analyzer on both sides + a keyword sub-field for exact match.
$params = [
    'index' => 'articles',
    'body'  => [
        'settings' => [
            'analysis' => [
                'analyzer' => [
                    'english_stem' => [
                        'tokenizer' => 'standard',
                        'filter'    => ['lowercase', 'english_stop', 'english_stemmer'],
                    ],
                ],
                'filter' => [
                    'english_stop'    => ['type' => 'stop',    'stopwords' => '_english_'],
                    'english_stemmer' => ['type' => 'stemmer', 'language'  => 'english'],
                ],
            ],
        ],
        'mappings' => [
            'properties' => [
                'body' => [
                    'type'     => 'text',
                    'analyzer' => 'english_stem', // used for index AND query
                    'fields'   => [
                        'exact' => ['type' => 'keyword'], // untouched for filters/IDs
                    ],
                ],
            ],
        ],
    ],
];
// 'running', 'runs', 'ran'* all reduce to 'run' (*ran needs a lemmatizer
// plugin or synonym rule - stemming alone won't catch irregulars).

Tags

Related Terms

References

• https://www.elastic.co/guide/en/elasticsearch/reference/current/stemming.html
• https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html
• https://snowballstem.org/algorithms/
• https://spacy.io/usage/linguistic-features#lemmatization