Competency Questions

Also Known As

TL;DR

Explanation

Competency questions are the requirements artefact of ontology and knowledge graph engineering. They are natural-language questions - 'Which suppliers ship parts to plants in the EU?', 'What drugs interact with warfarin and were approved after 2015?' - that the finished knowledge base is expected to answer correctly. Introduced as a core practice in the METHONTOLOGY and later NeOn ontology-engineering methods, they play the same role that user stories or acceptance criteria play in application development: they turn a vague ambition ('model the pharmaceutical domain') into a concrete, testable specification.

Good competency questions are written before modelling begins and are revisited whenever scope changes. Each question names the entities, relationships, and constraints it exercises, which directly reveals what classes, properties, and axioms the ontology must include. If a question cannot be expressed as a SPARQL, Cypher, or SHACL query over the current model, either the model is incomplete or the question is out of scope - both outcomes are useful. Teams typically maintain a numbered list of questions with expected answer shapes (a single value, a set, a boolean) and translate each into an executable query that becomes a regression test for the graph.

Competency questions also constrain over-modelling. A common failure mode in ontology work is to encode every distinction the domain expert can articulate, producing a sprawling model that no consumer needs. Requiring every class and property to trace back to at least one competency question keeps the ontology grounded in actual use. Conversely, when downstream applications discover a query they cannot answer, the missing capability is captured as a new competency question, extending the model in a disciplined way rather than ad hoc.

The practice fails when questions are too vague to translate ('tell me about our customers'), when they are written after the ontology to rationalise existing choices, or when they are never run as automated queries and drift out of sync with the model. Treated as living, executable specifications they anchor an ontology to real information needs and give the team a clear signal for when the model is done, complete, or over-built.

Common Misconception

Why It Matters

Common Mistakes

• Writing vague questions like 'tell me about customers' that cannot be translated into a concrete query or answer shape.
• Producing competency questions after the ontology is built, so they rationalise existing choices instead of driving design.
• Never turning questions into executable SPARQL or Cypher tests, letting the model drift out of sync with its stated requirements.
• Modelling classes and properties that trace back to no competency question, inflating the ontology with unused structure.
• Failing to specify the expected answer shape (single value, set, boolean), making it unclear when a question is satisfied.

Avoid When

• The ontology is a throwaway sketch or exploratory prototype where formal requirements would slow discovery.
• The domain is trivially small and every stakeholder already agrees on the handful of queries needed.
• There is no plan to ever query the model programmatically, making executable questions pointless.
• Requirements change so rapidly that maintaining a stable question set costs more than it saves.

When To Use

• Scoping a new ontology or knowledge graph so the model targets real information needs.
• Validating that an existing knowledge base can answer the queries its consumers depend on.
• Preventing over-modelling by requiring every class and property to trace to at least one question.
• Onboarding downstream teams by giving them a concrete, tested catalogue of what the graph can answer.

Code Examples

# Vague, untestable 'competency questions' scribbled after the fact
competency_questions = [
    "tell me about our products",
    "customer information",
    "anything interesting about orders",
]

# No expected answer shape, no query, no way to know if the ontology satisfies them.
# The team ends up modelling every attribute the domain expert mentions,
# then discovering months later that key joins are missing.
def validate_ontology(graph):
    return True  # nothing to check against

# Competency questions as executable specifications with expected answer shapes
competency_questions = [
    {
        "id": "CQ-01",
        "nl": "Which suppliers ship parts to plants in the EU?",
        "shape": "set<Supplier>",
        "sparql": """
            PREFIX : <http://example.org/supply#>
            SELECT DISTINCT ?supplier WHERE {
                ?supplier a :Supplier ; :shipsTo ?plant .
                ?plant :locatedIn/:inRegion :EU .
            }
        """,
    },
    {
        "id": "CQ-02",
        "nl": "Is part P-42 dual-sourced?",
        "shape": "boolean",
        "sparql": "ASK { ?s1 :supplies :P-42 . ?s2 :supplies :P-42 . FILTER(?s1 != ?s2) }",
    },
]

def validate_ontology(graph):
    # Each CQ becomes a regression test: the query must parse and return
    # a result of the expected shape against the reference dataset.
    for cq in competency_questions:
        result = graph.query(cq["sparql"])
        assert result is not None, f"{cq['id']} failed: {cq['nl']}"

Tags

Related Terms

References

• https://www.aaai.org/Papers/Symposia/Spring/1997/SS-97-06/SS97-06-011.pdf
• https://www.w3.org/TR/sparql11-query/
• https://neon-project.org/nw/Ontology_Engineering_Methodology.html
• https://www.obofoundry.org/principles/fp-016-maintenance.html