Lexing & Parsing

Also Known As

TL;DR

Explanation

Lexing (tokenisation): scans characters and groups them into meaningful tokens — T_FUNCTION, T_STRING, T_WHITESPACE. Whitespace and comments are typically discarded. Parsing: takes the token stream and builds an AST according to the language grammar (typically expressed as a context-free grammar). PHP's token_get_all() exposes the lexer output. Nikic's PHP-Parser builds the full AST. Applications: PHPStan traverses the AST for type checking, Rector modifies the AST for code transformations, php-cs-fixer analyses token structure for style checking.

Common Misconception

Why It Matters

Common Mistakes

• Parsing PHP with regex — regex cannot handle recursive structures like nested expressions.
• Not understanding that the AST represents semantics not formatting — reformatting does not change the AST.
• Token positions are byte offsets not character positions — important for multibyte PHP source.
• Modifying AST nodes without updating their parent references — causes tree inconsistency.

Code Examples

// Regex parsing of PHP — brittle and wrong:
$functions = [];
preg_match_all('/function\s+(\w+)\s*\(/', $source, $matches);
// Misses: closures, arrow functions, methods, functions in strings
// Breaks on: comments containing 'function', heredoc, nested structures

// PHP-Parser — correct AST-based analysis:
use PhpParser\ParserFactory;
use PhpParser\NodeTraverser;
use PhpParser\NodeVisitorAbstract;
use PhpParser\Node;

$parser   = (new ParserFactory)->createForNewestSupportedVersion();
$ast      = $parser->parse($sourceCode);

$traverser = new NodeTraverser();
$traverser->addVisitor(new class extends NodeVisitorAbstract {
    public function enterNode(Node $node): void {
        if ($node instanceof Node\Stmt\Function_) {
            echo 'Found function: ' . $node->name . PHP_EOL;
        }
    }
});
$traverser->traverse($ast);

References

• ↗ https://github.com/nikic/PHP-Parser

Tags

Related Terms