AI Alignment
debt(d8/e7/b7/t7)
Closest to 'silent in production until users hit it' (d9), -1. The detection_hints state automated detection is 'no' — there are no standard tools that catch alignment failures. The code pattern (system prompts without safety constraints, no output validation) is only identifiable through careful manual review. However, some misalignment surfaces during runtime testing or red-teaming, preventing a full d9. Most alignment failures manifest silently in production as subtly wrong, harmful, or data-leaking outputs.
Closest to 'cross-cutting refactor across the codebase' (e7). While the quick_fix describes writing explicit system prompts (which sounds like e3), the common_mistakes reveal that alignment is not a one-time fix: it requires re-evaluation per use-case, ongoing monitoring, red-teaming, and output validation pipelines. Retrofitting alignment constraints across all LLM integration points — system prompts, output filters, evaluation metrics, monitoring — is a cross-cutting concern that touches multiple files and systems.
Closest to 'strong gravitational pull' (b7). Alignment applies across all contexts (web, cli, queue-worker) and is tagged as a safety/ethics concern. Every LLM interaction point must account for alignment constraints. It shapes how prompts are written, how outputs are validated, how monitoring is configured, and how new features using LLMs are designed. It's a persistent architectural concern that influences every change involving model integration, though it doesn't quite define the entire system's shape.
Closest to 'serious trap — contradicts how a similar concept works elsewhere' (t7). The misconception is explicit: developers believe alignment is a theoretical AI research concern, not a practical engineering one. This causes them to write task-focused prompts without constraints, assuming the model will 'do the right thing' by default — analogous to how a well-trained human employee would. The model instead optimises only for what is stated, not what is implied, which contradicts developers' mental model of instruction-following. Assuming alignment transfers across use-cases is another serious trap.
Also Known As
TL;DR
Explanation
AI alignment addresses a fundamental problem: an AI optimised for a proxy metric can behave in ways that are technically correct yet harmful or contrary to human intent. Classic examples include reward hacking (an agent finding unintended shortcuts) and specification gaming (achieving the stated goal in an unexpected, damaging way). Alignment is split into two sub-problems: outer alignment (ensuring the training objective matches what we actually want) and inner alignment (ensuring the trained model actually optimises for that objective). Practical alignment techniques include Reinforcement Learning from Human Feedback (RLHF), Constitutional AI (CAI), debate, scalable oversight, and red-teaming. For software engineers, alignment manifests as prompt design, system prompts that constrain behaviour, output validation, and refusal classifiers. At a societal level it concerns governance frameworks and safety evaluation before deployment.
Diagram
flowchart TD
GOAL[Intended Human Goal] -->|specify| OBJ[Training Objective]
OBJ -->|train| MODEL[Trained Model]
MODEL -->|optimises| BEHAVIOUR[Actual Behaviour]
subgraph AlignmentGap
OUTER[Outer Misalignment<br/>Wrong objective specified]
INNER[Inner Misalignment<br/>Model drifts from objective]
end
GOAL -.->|gap| OUTER
OBJ -.->|gap| INNER
subgraph Techniques
RLHF[RLHF: human preference feedback]
CAI[Constitutional AI: rule-based critique]
RED[Red-teaming: adversarial probing]
GUARD[Guardrails: runtime filtering]
end
OUTER & INNER -.->|mitigated by| RLHF & CAI & RED & GUARD
style OUTER fill:#f85149,color:#fff
style INNER fill:#f85149,color:#fff
style RLHF fill:#238636,color:#fff
style CAI fill:#238636,color:#fff
Common Misconception
Why It Matters
Common Mistakes
- Writing system prompts that specify the task but omit constraints — the model optimises only for what is stated, not what is implied.
- Assuming a model aligned for one use-case transfers safely to another without re-evaluation.
- Relying solely on RLHF without red-teaming adversarial inputs that exploit fine-tuned blind spots.
- Treating alignment as a one-time step rather than an ongoing monitoring and evaluation process.
Avoid When
- Treating alignment as solved once RLHF training is complete — adversarial inputs can still bypass it.
- Applying alignment techniques designed for one domain (e.g. chat) to a different deployment context (e.g. code execution) without re-evaluation.
When To Use
- Define explicit behavioural constraints in the system prompt for every production LLM deployment.
- Red-team your prompts with adversarial inputs before launch to surface alignment failures early.
- Monitor outputs in production for drift from intended behaviour using automated evaluation metrics.
- Use Constitutional AI or structured critique loops when human feedback at scale is not feasible.