Parser and nodes

Parser & Node-types — operational

Companion docs: knowledge/idioms/parser-pipeline.md, knowledge/idioms/node-types-and-lists.md.

0. The three-tree pipeline — AST → Query → Plan

Every SQL statement flows through three distinct tree shapes; nodes in each header live or die at a specific stage. Confuse the stages and your patch will silently fail in a downstream walker.

SQL text
  │
  │  gram.y / scan.l / kwlist.h
  ▼
1. Raw parsetree (AST)         — parsenodes.h utility-stmt nodes
   Examples: SelectStmt,         + A_Expr / A_Const / ColumnRef /
   InsertStmt, A_Expr,             FuncCall (raw expression shapes)
   ColumnRef, FuncCall          → walker: raw_expression_tree_walker
                                  No types resolved, no relations
                                  looked up, no operators resolved.
  │
  │  parser/analyze.c           → transformStmt → transform<Foo>Stmt
  │  parser/parse_*.c           → adds rangetable, resolves names,
  │                               picks operator + cast OIDs,
  │                               applies coercions
  ▼
2. Query tree                  — parsenodes.h Query + analyzed
   Examples: Query, RangeTblEntry, expression nodes (primnodes.h)
   Var, OpExpr, Const, FuncExpr,  Examples: Var, OpExpr, Const,
   TargetEntry, FromExpr, JoinExpr  FuncExpr, Aggref, WindowFunc
                                → walker: expression_tree_walker
                                  + query_tree_walker
                                  + QTW_* flags for rtable/CTE descent
  │
  │  rewrite/                  → rules, views (Query → Query)
  │  optimizer/planner.c        → planner: Query → Plan
  ▼
3. Plan tree                   — plannodes.h
   Examples: SeqScan,            Examples: SeqScan, IndexScan, HashJoin,
   IndexScan, HashJoin,            Sort, Agg, Limit, Gather, GatherMerge
   Sort, Agg, Limit             → walker: plan_tree_walker
                                + planstate_tree_walker (executor-time)
                                  Shipped to parallel workers via
                                  nodeToString / stringToNode.

Operational rules per shape:

Shape	Key invariants
Raw parsetree	No catalog lookups (parser runs without an open transaction in some paths); A_* and ColumnRef are the pre-resolution placeholders. Don't try to compute types here.
Query tree	Types, collations, operator OIDs, cast OIDs resolved. Var carries varno+varattno into the rangetable. Subqueries are nested Querys in RangeTblEntry. Stored in pg_rewrite.ev_action for views/rules — bump catversion when changing serialized fields.
Plan tree	Materialized from Paths by createplan.c. Plan's expression trees use Vars indexed into targetlist slots, NOT rangetable slots. Shipped to workers via the out/read funcs — bumps to plan-node fields don't need catversion (not stored on disk) but DO need correct out/read coverage.

The executor-and-planner skill covers the Plan-tree side (createplan, add_path costing, ExecInit/ProcNode dispatch). This skill covers raw parsetree and Query tree work, plus the node-machinery infrastructure (gen_node_support.pl, copy/equal/out/read funcs).

Tools for working with an existing tree

• copyObject(p) — deep copy (preserves argument type via typeof_unqual macro, nodes.h:228-233). Dispatcher copyObjectImpl in copyfuncs.c:177-212, with check_stack_depth() guard (:185). By-ref Datums go through _copyConst → datumCopy; T_List is deep-copied via list_copy_deep, T_IntList/T_OidList/T_XidList shallow via list_copy. Extensible nodes dispatch to a registered nodeCopy callback (_copyExtensibleNode). Allocated in CurrentMemoryContext — caller controls the lifetime.
• equal(a, b) — deep structural compare (equalfuncs.c).
• nodeToString(p) / stringToNode(s) — Lisp-ish text serialization; load-bearing for plan cache, rule storage (pg_rewrite.ev_action), parallel-worker plan shipping. The round-trip copyObject(stringToNode(nodeToString(p))) underpins the debug_write_read_parse_plan_trees GUC.
• expression_tree_walker / _mutator — polymorphic recursion over expression trees (post-analysis: Var, OpExpr, …). raw_expression_tree_walker for the pre-analysis shape (A_Expr, ColumnRef). query_tree_walker / _mutator wrap for Query, with QTW_* flags (nodeFuncs.h:22-34) to control rtable/CTE descent. planstate_tree_walker is the executor-time analogue. Mutator contract: return NULL to delete a subtree; walker contract: callback returns true to short-circuit, false to keep descending. The macro wrappers (nodeFuncs.h:155-183) let callbacks be typed without -Wincompatible-pointer-types warnings.

When you're adding a new SQL statement

Touch four layers, in order. Build between each so you fail fast.

1. Grammar — src/backend/parser/gram.y

   • Add the keyword(s) to unreserved_keyword / col_name_keyword / etc. AND to src/include/parser/kwlist.h (alphabetical, with category).
   • Add a top-level rule producing your new *Stmt node, wire it into stmt: (search for stmt: in gram.y). Use makeNode(FooStmt) in the action, set the location with @1, fill fields from $N.
   • Match an existing simple statement as template (e.g. DropStmt, VariableSetStmt). Do NOT invent fresh helpers in gram.y if parser/parse_utilcmd.c already has one.

2. Lexer — src/backend/parser/scan.l (only if you introduced a new token shape; new keywords alone do not need scan.l changes — they go through kwlist.h).

3. Parse node — src/include/nodes/parsenodes.h (utility stmts) or primnodes.h (expressions) or plannodes.h (plan nodes)

   • First field NodeTag type; (or another node struct, for "inheritance").
   • Last field for top-level statements is typically ParseLoc location;.
   • Annotate fields if needed: pg_node_attr(...) on the struct, per-field attrs like query_jumble_ignore, equal_ignore. See nodes.h:43-125 for the full list.

4. Analyze / execute

   • Optimizable statement → add a transformFooStmt(ParseState *, FooStmt *) in parser/analyze.c and wire it into the switch (nodeTag(parseTree)) inside transformStmt (analyze.c:334-451, switch at :368). Caution (analyze.c:363-367): any change to that switch must also be reflected in stmt_requires_parse_analysis() (:469-505) and analyze_requires_snapshot() (:513-529) — three sites, one logical change.
   • Utility statement → no transform needed; just dump into a Query with CMD_UTILITY (the default path). Execution lives in src/backend/commands/ via ProcessUtility / standard_ProcessUtility (src/backend/tcop/utility.c).

5. Run gen_node_support.pl (done automatically by the build) and verify the generated copyfuncs.funcs.c, equalfuncs.funcs.c, outfuncs.funcs.c, readfuncs.funcs.c chunks for your new struct look sane.

6. Add a regression test under src/test/regress/sql/ (and a matching expected file). See .claude/skills/testing/SKILL.md.

When you're adding a new Node type (no SQL change)

From src/backend/nodes/README "Steps to Add a Node":

1. Put the struct in the right header. The file must be in gen_node_support.pl's @all_input_files list (parsenodes.h, primnodes.h, plannodes.h, pathnodes.h, execnodes.h, value.h, etc. — see gen_node_support.pl:53-77). The T_Foo tag is generated automatically into nodes/nodetags.h (do NOT hand-edit it).
2. First field is NodeTag type; — unless you "inherit" by embedding another node struct as the first field (e.g. Plan plan; in a new plan node).
3. Build. Inspect the generated copyfuncs.funcs.c / equalfuncs.funcs.c / outfuncs.funcs.c / readfuncs.funcs.c / queryjumblefuncs.funcs.c entries for your node. Add pg_node_attr(...) and per-field attrs to fix anything wrong.
4. If a field needs special treatment, common annotations:
   • pg_node_attr(custom_copy_equal) — write your own bodies in copyfuncs.c / equalfuncs.c and the generator will skip yours.
   • pg_node_attr(no_copy_equal, no_read, no_query_jumble) — opt out entirely (executor state nodes typically use nodetag_only).
   • Per-field: array_size(otherfield), copy_as(VALUE), equal_ignore, read_write_ignore, query_jumble_ignore, query_jumble_location.
5. If your node is an expression (lives in primnodes.h or has child Expr fields), add cases to every giant switch in nodeFuncs.c:
   • exprType (:42+), exprTypmod (:304+), exprCollation / exprSetCollation / exprInputCollation (:826+, :1140+, :1092+), exprLocation (:1403+) — type/typmod/collation/location introspection.
   • expression_tree_walker_impl (:2111+), expression_tree_mutator_impl (:3018+) — recursion into children.
   • raw_expression_tree_walker_impl (:4115+) — only if the node can appear in raw parsetrees.
   • set_opfuncid family (:1890+) — only if your node holds an operator OID needing resolution. Grep T_<SiblingNode> (e.g. T_OpExpr) to find every site; five-to-eight maintenance points per node, miss one and walkers silently misbehave.
6. Adding a node type renumbers existing tags. Recompile the whole tree (--enable-depend helps). No initdb needed — node numbers never go to disk. BUT if the node can appear in a stored parse tree (rule actions, view definitions), bump CATALOG_VERSION_NO in src/include/catalog/catversion.h.
7. Test with debug_copy_parse_plan_trees=on, debug_write_read_parse_plan_trees=on, debug_raw_expression_coverage_test=on (set via PG_TEST_INITDB_EXTRA_OPTS).

When you're modifying an existing Node type

• Adding a field to a node that appears in stored catalog parse trees (e.g. Query, anything in views / rules) → bump catversion. Add a pg_node_attr if the new field needs special read/write handling (e.g. read_as(...) for backward-compat when reading old serialized trees from extensions — usually not needed because catversion changes invalidate stored trees).
• Removing or reordering fields → also catversion bump if stored.
• Changing only Plan/Path internal fields (never serialized to catalogs) → no catversion bump needed. But Plan nodes ARE serialized to parallel workers, so you still need correct out/read funcs.

Pre-submit smoke

cd dev/build-debug && ninja && \
  meson test --suite regress -q

Plus the debug-tree GUCs above on a separate initdb if you touched anything tree-shape related.

Node-type reference table

The major node families and where each lives. When in doubt about which header your new field belongs in, check this table.

Header	Shape	Examples	When you'd add here
nodes/parsenodes.h	Raw + Query	SelectStmt, InsertStmt, A_Expr, ColumnRef, Query, RangeTblEntry, TargetEntry, FromExpr, JoinExpr	New SQL statement; new fields on Query (catversion bump if serialized in pg_rewrite)
nodes/primnodes.h	Expression (analyzed)	Var, Const, OpExpr, FuncExpr, Aggref, WindowFunc, SubLink, CaseExpr, CoalesceExpr	New expression node (must touch every nodeFuncs.c switch — see §"new Node type")
nodes/plannodes.h	Plan	Plan, SeqScan, IndexScan, HashJoin, Sort, Agg, Limit, Gather, Append, MergeAppend	New plan node (also wire executor under nodeXxx.c — see executor-and-planner)
nodes/pathnodes.h	Path + RelOptInfo	Path, IndexPath, JoinPath, RelOptInfo, PlannerInfo	New path shape considered by the optimizer
nodes/execnodes.h	PlanState (runtime)	PlanState, SeqScanState, EState, ExprContext, ProjectionInfo	New runtime state for an executor node
nodes/value.h	Leaf values	Integer, Float, Boolean, String, BitString	Almost never — the four primitive value carriers
nodes/pg_list.h	Lists	List, IntList, OidList, XidList	Almost never (collection infrastructure)
nodes/extensible.h	Extension nodes	ExtensibleNode, CustomScan	Custom AM / custom planner extension

nodes/nodetags.h is auto-generated by gen_node_support.pl; never hand-edit it. A new struct in any of the headers above (with the file listed in gen_node_support.pl @all_input_files) gets its T_Foo tag automatically.

Walker / mutator quick reference

Function	Where	Use for
raw_expression_tree_walker	nodes/nodeFuncs.c	Pre-analysis (A_Expr, ColumnRef, FuncCall); used by parser/parse_*.c
expression_tree_walker	nodes/nodeFuncs.c	Analyzed expressions in a Query tree
expression_tree_mutator	nodes/nodeFuncs.c	Same shape, but returns new nodes; rewriter / planner use this
query_tree_walker	nodes/nodeFuncs.c	Wraps expression_tree_walker to also walk Query subexpressions; controlled by QTW_* flags
query_tree_mutator	nodes/nodeFuncs.c	Mutator counterpart
range_table_walker / _mutator	nodes/nodeFuncs.c	Just the rangetable, with per-RTE field control flags
plan_tree_walker	nodes/nodeFuncs.c	Plan-tree (plannodes.h) recursion
planstate_tree_walker	executor/execProcnode.c	Executor-time (PlanState) recursion; used by EXPLAIN

Conventions to remember:

• Walker callback returns bool. Return true to short-circuit the whole walk; return false to continue descending.
• Mutator callback returns Node *. Return NULL to delete a subtree; return the input pointer to keep it unchanged; return a new node to replace.
• QTW_* flags (nodeFuncs.h:22-34) control rtable / CTE descent. The default skips RTE subqueries — opt in with QTW_EXAMINE_RTES_BEFORE / _AFTER when you actually want them.
• The typed macro wrappers (nodeFuncs.h:155-183) let your callback be typed (bool foo_walker(Node *n, MyCtx *ctx)) without -Wincompatible-pointer-types warnings.

Files-examined rows

file	depth	produced
src/backend/parser/README	full	SKILL §"new SQL statement"
src/backend/parser/gram.y lines 13625-13706	scanned + 1 rule	SKILL §grammar
src/backend/parser/scan.l 1-40	top	SKILL §lexer
src/backend/parser/parse_node.c 1-50	top	SKILL §analyze
src/backend/parser/analyze.c 1-80, 334-433	transformStmt full switch	SKILL §analyze
src/backend/nodes/README	full	SKILL §"new Node"
src/include/nodes/nodes.h	full	SKILL §annotations
src/include/nodes/pg_list.h 1-120, 380-485	List API + foreach	idioms
src/include/nodes/value.h	full	idioms
src/backend/nodes/gen_node_support.pl 1-140	top + file list	SKILL §generator

Cross-references

• .claude/skills/executor-and-planner/SKILL.md — Plan-tree side of the three-tree pipeline; createplan.c, ExecInitNode/ExecProcNode for new plan nodes.
• .claude/skills/catalog-conventions/SKILL.md — CATALOG_VERSION_NO bump rules when serialized parse-tree fields change.
• .claude/skills/testing/SKILL.md — debug_copy_parse_plan_trees, debug_write_read_parse_plan_trees, debug_raw_expression_coverage_test GUCs for tree-shape regressions.
• .claude/skills/coding-style/SKILL.md — node-header style, pg_node_attr placement, T_Foo enum convention.
• .claude/skills/fmgr-and-spi/SKILL.md — function-call node (FuncExpr) interaction with fmgr; SPI's parse-tree consumption.
• knowledge/idioms/parser-pipeline.md — long-form pipeline narrative.
• knowledge/idioms/node-types-and-lists.md — List / foreach patterns + node-tag invariants.
• source/src/backend/nodes/README — canonical "Steps to Add a Node" reference.
• source/src/backend/parser/README — gram.y conventions, shift/reduce conflict resolution.