Skip to content

Migrate test toolchain to the new Spago (spago.yaml + registry) #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Unisay merged 1 commit into from
Jun 23, 2026
Merged

Migrate test toolchain to the new Spago (spago.yaml + registry) #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 58 additions & 24 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,13 @@ locally:
- **PureScript**: `purs` 0.15.16, from [purescript-overlay] pinned as
`purs-bin.purs-0_15_16` (explicit pin so `nix flake update` never silently
bumps the compiler and churns goldens)
- **Spago**: 0.21.x — the *legacy* Haskell spago driven by `spago.dhall` /
`packages.dhall` (not the newer `spago.yaml`-based one), pinned as
`spago-bin.spago-0_21_0`. NB: the overlay's plain `spago` attr now resolves
to the new PureScript spago (1.x), hence the explicit legacy pin.
- **Spago**: 1.0.x — the new PureScript spago, driven by `spago.yaml` +
`spago.lock` and the PureScript Registry (it dropped Dhall support), pinned
as `spago-bin.spago-1_0_4`. The test project (`test/ps`) uses a `registry`
package-set baseline plus `extraPackages` git overrides for the Lua FFI
forks (see "Toolchain / package set" under Testing). The overlay's plain
`spago` attr also resolves to 1.x; the explicit pin keeps the version
changed only by a deliberate flake edit.

[purescript-overlay]: https://github.com/thomashoneyman/purescript-overlay

Expand Down Expand Up @@ -63,7 +66,7 @@ ghcid --command="cabal repl test:spec" --test=":main"

The test suite includes:
- **Unit tests**: Property-based testing with Hedgehog
- **Golden tests**: Compiles PureScript test modules from `test/ps/golden/Golden/*/Test.purs` to Lua and compares against golden files
- **Golden tests**: Compiles PureScript test modules from `test/ps/src/Golden/*/Test.purs` to Lua and compares against golden files
- **Evaluation tests**: Runs generated Lua code and verifies output
- **Luacheck tests**: Validates generated Lua code syntax

Expand All @@ -74,18 +77,38 @@ Golden tests require compiling PureScript sources first:
```bash
# Compile PureScript test sources (from test/ps directory)
cd test/ps
spago build -u '-g corefn'
spago build
cd ../..
```

### Resetting Golden Files
The new spago manages codegen itself and rejects `--codegen` in `--purs-args`.
CoreFn is emitted because `test/ps/spago.yaml` declares a no-op `backend`
(`cmd: "true"`): with a backend configured spago compiles with `--codegen
corefn` and the harness reads the resulting `output/**/corefn.json`. The
golden sources live under `test/ps/src/` (new spago only globs `src/` and
`test/`), with `.lua` FFI files co-located next to each `.purs`.

### Regenerating Golden Files

When a deliberate codegen/optimizer change legitimately moves the output, accept
the new structural goldens in place:

```bash
# Remove all golden files and regenerate them
./scripts/golden_reset
# Rewrites mismatching golden.ir / golden.lua with the actual output and passes
PSLUA_GOLDEN_ACCEPT=1 cabal test all --test-show-details=direct
```

This finds all files named `golden.*` in `test/ps/output` and deletes them, then runs `cabal test` to regenerate them.
Only the *structural* goldens (`golden.ir`, `golden.lua`) are auto-accepted. The
hand-verified `eval/golden.txt` oracle is **never** auto-accepted — if a change
alters runtime output, those tests still fail, which is the semantic safety net.
Review the resulting `git diff`, then run `cabal test` once more (no env var) to
confirm the accepted state is stable.

NB: `./scripts/golden_reset` deletes **all** `golden.*` files — including
`eval/golden.txt` — and lets the harness recreate them from current output. That
silently overwrites the hand-verified oracle with whatever the code emits now
(even if buggy), so prefer `PSLUA_GOLDEN_ACCEPT` and do not run `golden_reset`
unless you intend to discard the oracles.

### Code Formatting & Linting

Expand Down Expand Up @@ -265,8 +288,10 @@ Both lines should be exactly 80 characters. Helper functions go at the bottom af

Golden tests are the primary integration testing mechanism:

1. PureScript test files in `test/ps/golden/Golden/*/Test.purs`
2. Compiled to CoreFn with `spago build -u '-g corefn'`
1. PureScript test files in `test/ps/src/Golden/*/Test.purs`
2. Compiled to CoreFn with `spago build` (the no-op `backend` in
`test/ps/spago.yaml` is what makes spago emit CoreFn — see "Testing
PureScript Code")
3. Test suite reads CoreFn, compiles to IR, generates Lua
4. Compares against golden files:
- `golden.ir` - Intermediate representation
Expand All @@ -281,7 +306,7 @@ Golden tests are the primary integration testing mechanism:
bypass it.

To add a new golden test:
1. Create `test/ps/golden/Golden/NewTest/Test.purs`
1. Create `test/ps/src/Golden/NewTest/Test.purs`
2. Run `cabal test` - it will fail and create `actual.*` files
3. Review the actual files
4. Rename `actual.*` to `golden.*` if correct
Expand Down Expand Up @@ -322,18 +347,27 @@ often enough; add more when the bug spans several code paths).
spago, edit `compiler-nix-name` / `purs-bin.*` / `spago-bin.*` in
`flake.nix` (the toolchain attrs are explicitly version-pinned, so a flake
update alone never changes them).
2. PureScript package sets live in `test/ps/packages.dhall` as
`upstream-ps // upstream-lua`. The right operand wins: `upstream-lua`
(releases of `purescript-lua/purescript-lua-package-sets`) overrides core
packages with Lua forks that ship `.lua` FFI files.
3. After changing package sets or `purs`: `cd test/ps && spago build -u
'-g corefn'`, then `cabal test all`. Drop the `sha256:` annotations
when changing package set URLs — spago re-freezes them on first build.
2. PureScript dependencies live in `test/ps/spago.yaml`. The baseline is a
`workspace.packageSet.registry: <ver>` package set (pure, no-FFI packages),
and every FFI-bearing core package is overridden by its Lua fork
(`purescript-lua/purescript-lua-*`, which ships `.lua` FFI) via an
`extraPackages` git entry pinned to a tag. This replaces the old
`upstream-ps // upstream-lua` Dhall set, where the Lua forks won. The fork
list / tags / dependency lists are mirrored from the Lua package set's
`packages.dhall` (the source of truth); `spago.lock` pins the resolved
commits and is committed. Keep `purs` at 0.15.16: no registry set is built
for 0.15.16, so use the latest 0.15.15 set (a proven-compatible pairing).
3. After changing the package set or `purs`: `cd test/ps && spago build`, then
`cabal test all`. Delete `spago.lock` first if you want spago to re-resolve
the git overrides.
4. Expected churn after updates:
- `test/ps/output/*/corefn.json` are committed; their `"builtWith"`
stamp changes with the `purs` version.
- `golden.ir` files embed `.spago/<pkg>/<version>/...` source paths,
so package version bumps legitimately change goldens.
- `test/ps/output/Golden*/corefn.json` are committed; their `"builtWith"`
stamp changes with the `purs` version, and `modulePath` reflects the
`src/` source location.
- `golden.ir` files embed `.spago/p/<pkg>/<commit-hash>/...` source paths
(the new spago content-addressed layout), so a fork-tag bump that resolves
to a new commit legitimately changes goldens. `golden.lua` is path-free,
so it only moves when codegen genuinely changes.

### Known Pitfalls

Expand Down
15 changes: 8 additions & 7 deletions docs/GOLDEN_TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ shows up as a diff in these files.

- Harness: `test/Language/PureScript/Backend/Lua/Golden/Spec.hs`
- Golden primitive (vendored, customised): `test/Test/Hspec/Golden.hs`
- Test modules: `test/ps/golden/Golden/<Name>/Test.purs`
- Test modules: `test/ps/src/Golden/<Name>/Test.purs`
- Generated artifacts: `test/ps/output/Golden.<Name>.Test/`
- Reset script: `scripts/golden_reset`
- Regenerate (accept structural goldens): `PSLUA_GOLDEN_ACCEPT=1 cabal test`

## The artifacts

Expand All @@ -19,7 +19,7 @@ Each test module `Golden.<Name>.Test` maps to a directory

| File | Committed? | What it is |
|---|---|---|
| `corefn.json` | yes | CoreFn emitted by `purs` (`spago build -g corefn`). Its `builtWith` stamp tracks the `purs` version. |
| `corefn.json` | yes | CoreFn emitted by `purs`, driven by `spago build` via the no-op `backend` in `spago.yaml`. Its `builtWith` stamp tracks the `purs` version. |
| `golden.ir` | yes | Pretty-printed IR `UberModule` — **structural**, a pure function of the code. |
| `golden.lua` | yes | Generated Lua — **structural**. |
| `eval/golden.txt` | yes | Program stdout, normalised. The **semantic oracle** — hand-verified. Present only for runnable modules. |
Expand All @@ -34,8 +34,9 @@ the eval file's presence controls, beyond enabling the eval check.

`cabal test spec` (matched by `-m Goldens`) runs four groups, in order:

1. **compile** (`beforeAll_ compilePs`) — `spago build -u '-g corefn'` in
`test/ps`, producing `corefn.json` for every module.
1. **compile** (`beforeAll_ compilePs`) — `spago build` in `test/ps`, producing
`corefn.json` for every module (the `backend` in `spago.yaml` makes spago
emit CoreFn).
2. **`compiles corefn files to lua`** — for each module:
- `compileCorefn` reads CoreFn → builds the IR `UberModule` → compares the
pretty-printed IR against `golden.ir`.
Expand Down Expand Up @@ -124,9 +125,9 @@ other half of keeping failures affordable.

## Adding a new golden test

1. Create `test/ps/golden/Golden/<Name>/Test.purs` (module
1. Create `test/ps/src/Golden/<Name>/Test.purs` (module
`Golden.<Name>.Test`). For a runnable test, give it `main :: Effect Unit`.
2. `cd test/ps && spago build -u '-g corefn' && cd ../..` to emit `corefn.json`.
2. `cd test/ps && spago build && cd ../..` to emit `corefn.json`.
3. For a runnable test, create the oracle by hand:
`test/ps/output/Golden.<Name>.Test/eval/golden.txt` with the expected stdout,
plus `eval/.gitignore` containing `actual.txt`.
Expand Down
2 changes: 1 addition & 1 deletion flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
buildInputs = with pkgs; [
cachix
purs-bin.purs-0_15_16
spago-bin.spago-0_21_0
spago-bin.spago-1_0_4
lua51Packages.lua
lua51Packages.luacheck
nil
Expand Down
2 changes: 1 addition & 1 deletion lib/Language/PureScript/Backend/IR/Types.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ shift/substitute/countFreeRefs implementing the opposite convention
sibling-bound reference past its binder, DCE deleted the "unused"
binder, and codegen rendered the dangling 'Ref (Local Bind1) 1' as an
undefined Lua variable 'Bind11'. The golden test
test/ps/golden/Golden/Issue37/Test.purs and the "Let sequential (let*)
test/ps/src/Golden/Issue37/Test.purs and the "Let sequential (let*)
scoping" tests pin the convention.
-}

Expand Down
5 changes: 4 additions & 1 deletion test/Language/PureScript/Backend/Lua/Golden/Spec.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,10 @@ spec = do
putText "Compiling PureScript sources"
exitCode ←
runProcess . setWorkingDir "test/ps" . shell $
String.unwords ["spago", "build", "-u", "'-g corefn'"]
-- Spago >= 0.93 manages codegen itself and rejects `--codegen`
-- in --purs-args; the `backend` in test/ps/spago.yaml is what
-- makes it emit CoreFn (see that file for the rationale).
String.unwords ["spago", "build"]
exitCode `shouldBe` ExitSuccess
psOutputPath = $(mkRelDir "test/ps/output/")

Expand Down
2 changes: 1 addition & 1 deletion test/ps/output/Golden.Annotations.M1/corefn.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"builtWith":"0.15.16","comments":[{"LineComment":" @inline inlineMe always"},{"LineComment":" @inline inlineMeLambda always"}],"decls":[{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"bindType":"NonRec","expression":{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"argument":"v","body":{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"caseAlternatives":[{"binders":[{"annotation":{"meta":null,"sourceSpan":{"end":[6,11],"start":[6,10]}},"binderType":"LiteralBinder","literal":{"literalType":"IntLiteral","value":1}}],"expression":{"annotation":{"meta":null,"sourceSpan":{"end":[6,15],"start":[6,14]}},"type":"Literal","value":{"literalType":"IntLiteral","value":2}},"isGuarded":false},{"binders":[{"annotation":{"meta":null,"sourceSpan":{"end":[7,11],"start":[7,10]}},"binderType":"VarBinder","identifier":"x"}],"expression":{"annotation":{"meta":null,"sourceSpan":{"end":[7,15],"start":[7,14]}},"type":"Var","value":{"identifier":"x","sourcePos":[7,10]}},"isGuarded":false}],"caseExpressions":[{"annotation":{"meta":null,"sourceSpan":{"end":[6,15],"start":[6,1]}},"type":"Var","value":{"identifier":"v","sourcePos":[0,0]}}],"type":"Case"},"type":"Abs"},"identifier":"inlineMe"}],"exports":["inlineMe","dontInlineClosure","inlineMeLambda"],"foreign":["dontInlineClosure","inlineMeLambda"],"imports":[{"annotation":{"meta":null,"sourceSpan":{"end":[10,44],"start":[3,1]}},"moduleName":["Prim"]}],"moduleName":["Golden","Annotations","M1"],"modulePath":"golden/Golden/Annotations/M1.purs","reExports":{},"sourceSpan":{"end":[10,44],"start":[3,1]}}
{"builtWith":"0.15.16","comments":[{"LineComment":" @inline inlineMe always"},{"LineComment":" @inline inlineMeLambda always"}],"decls":[{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"bindType":"NonRec","expression":{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"argument":"v","body":{"annotation":{"meta":null,"sourceSpan":{"end":[5,23],"start":[5,1]}},"caseAlternatives":[{"binders":[{"annotation":{"meta":null,"sourceSpan":{"end":[6,11],"start":[6,10]}},"binderType":"LiteralBinder","literal":{"literalType":"IntLiteral","value":1}}],"expression":{"annotation":{"meta":null,"sourceSpan":{"end":[6,15],"start":[6,14]}},"type":"Literal","value":{"literalType":"IntLiteral","value":2}},"isGuarded":false},{"binders":[{"annotation":{"meta":null,"sourceSpan":{"end":[7,11],"start":[7,10]}},"binderType":"VarBinder","identifier":"x"}],"expression":{"annotation":{"meta":null,"sourceSpan":{"end":[7,15],"start":[7,14]}},"type":"Var","value":{"identifier":"x","sourcePos":[7,10]}},"isGuarded":false}],"caseExpressions":[{"annotation":{"meta":null,"sourceSpan":{"end":[6,15],"start":[6,1]}},"type":"Var","value":{"identifier":"v","sourcePos":[0,0]}}],"type":"Case"},"type":"Abs"},"identifier":"inlineMe"}],"exports":["inlineMe","dontInlineClosure","inlineMeLambda"],"foreign":["dontInlineClosure","inlineMeLambda"],"imports":[{"annotation":{"meta":null,"sourceSpan":{"end":[10,44],"start":[3,1]}},"moduleName":["Prim"]}],"moduleName":["Golden","Annotations","M1"],"modulePath":"src/Golden/Annotations/M1.purs","reExports":{},"sourceSpan":{"end":[10,44],"start":[3,1]}}
2 changes: 1 addition & 1 deletion test/ps/output/Golden.Annotations.M1/golden.ir
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ UberModule
( QName
{ qnameModuleName = ModuleName "Golden.Annotations.M1", qnameName = Name "foreign"
}, ForeignImport Nothing
( ModuleName "Golden.Annotations.M1" ) "golden/Golden/Annotations/M1.purs"
( ModuleName "Golden.Annotations.M1" ) "src/Golden/Annotations/M1.purs"
[ ( Nothing, Name "dontInlineClosure" ), ( Just Always, Name "inlineMeLambda" ) ]
)
], uberModuleForeigns = [], uberModuleExports =
Expand Down
Loading
Loading