Tags: abgoyal/sqlproxy
Tags
Eliminate per-request struct allocation in step execution Merges step execution logic into the workflow package, removing the intermediate step types (QueryStep, HTTPCallStep, ResponseStep) that were allocated and discarded on every HTTP request. The step package becomes a slim contracts-only package with interfaces and types. Step Execution (inlined into workflow): - Move query execution into execute_query.go (direct DB call, no intermediary) - Move httpcall execution into execute_httpcall.go with buildHTTPRequest helper - Move response execution into execute_response.go - Move extractSQLParams and normalizeJSONResponse into workflow package - Extract buildHTTPRequest to deduplicate initial/retry request creation Types Eliminated: - step.QueryStep, step.HTTPCallStep, step.ResponseStep (per-request allocations) - step.Result (merged into workflow.StepResult) - step.RetryConfig (use workflow.RetryConfig directly) - step.Step interface (never used polymorphically) - LoggerAdapter, LoggerInterface (passthrough wrappers) Type Unification (via aliases, eliminates field-by-field copies): - config.RateLimitConfig = workflow.RateLimitRefConfig - workflow.RateLimitContext = tmpl.RateLimitData - workflow.Logger = step.Logger - db.QueryResult = step.QueryResult ExecutionData Slimmed: - Remove DBManager, HTTPClient, Logger fields (executor uses its own fields) - Only carries per-request data: TemplateData, ExprEnv, ResponseWriter Constructors Eliminated: - NewQueryStep (7 params), NewHTTPCallStep (8 params), NewResponseStep (5 params) Tests: - Add execute_helpers_test.go (extractSQLParams, normalizeJSONResponse) - Add execute_steps_test.go (query/httpcall/response execution paths) - Coverage: executeQueryStep 100%, executeResponseStep 100%, executeHTTPCallStep 83%
Add SQL Server e2e test suite with containerized instance Adds a full SQL Server e2e test suite running the complete taskapp workflow stack against a containerized SQL Server 2022 instance in CI. SQL Server E2E: - Add testdata/taskapp-sqlserver.yaml with all SQL converted to T-SQL syntax - Add test-sqlserver CI job with SQL Server 2022 and go-httpbin containers - Use MERGE for upserts, OFFSET/FETCH for pagination, TOP for limits - Use IDENTITY(1,1), NVARCHAR, GETDATE(), DATEADD(), SCOPE_IDENTITY() - Create testdb via sqlcmd before running tests Test Script: - Extend DB_VARS skip condition to include sqlserver configs
Add self-hosted httpbin and MySQL e2e test suite
Eliminates CI flakiness from external httpbin.org dependency by
self-hosting go-httpbin as a service container. Adds a full MySQL
variant of the e2e test suite to verify the complete workflow stack
(HTTP trigger -> query -> response) against a real MySQL instance.
CI Reliability:
- Add go-httpbin service container to both test and test-mysql jobs
- Replace hardcoded httpbin.org URLs with configurable HTTPBIN_URL variable
- Use on-disk SQLite (DB_PATH=/tmp/taskapp-test.db) instead of :memory: in CI
MySQL E2E Tests:
- Add testdata/taskapp-mysql.yaml with all SQL converted to MySQL syntax
- Run full taskapp e2e suite against MySQL in test-mysql CI job
- Convert SQLite-specific syntax: AUTOINCREMENT, datetime(), INSERT OR IGNORE,
ON CONFLICT, last_insert_rowid(), || concatenation, TEXT UNIQUE
- Use MySQL equivalents: AUTO_INCREMENT, NOW()/CURDATE(), INSERT IGNORE,
ON DUPLICATE KEY UPDATE, LAST_INSERT_ID(), CONCAT(), VARCHAR(255)
Test Infrastructure:
- Make taskapp_test.sh config overridable via TASKAPP_CONFIG env var
- Make DB_VARS conditional (skipped for MySQL configs)
- Support DB_PATH env var override for on-disk SQLite testing
Cleanup:
- Bump actions/checkout and actions/setup-go to v6
Add e2e tests for condition expression functions Tests the shared expr functions (divOr, len, hasPrefix, upper, etc.) that were added to condition expressions in the previous commit. Adds a demo workflow endpoint and corresponding e2e test. E2E Tests (e2e/taskapp_test.sh): - Add test_condition_expr_functions() to verify expr functions in conditions - Test alias chaining (ready depends on has_data and has_multiple) - Test safe division with divOr() - Test string functions hasPrefix() and upper() in conditions Demo Workflow (testdata/taskapp.yaml): - Add demo_condition_functions workflow at GET /api/demo/condition-functions - Demonstrate condition aliases with len(), divOr(), hasPrefix(), upper() - Show alias chaining pattern (ready = has_data && has_multiple) Cleanup: - Add *.test to .gitignore
Add step reference validation for conditions Validate that step references in conditions (steps.foo) point to steps that exist and come before the current step. This catches forward references, self-references, and typos at validation time. Compile (compile.go): - Add stepRefFinder AST visitor to find steps.X patterns - Add ExtractStepRefs() to extract step names from expressions - Handle both dot notation (steps.foo) and bracket notation (steps["foo"]) Validation (validate.go): - Add validateStepRefs() to check step references - Detect unknown steps, forward references, and self-references - Expand aliases to check step refs hidden in alias definitions - Distinct error messages for each case Documentation (README.md): - Add "Step references" paragraph in Condition Expressions section Tests: - Add TestExtractStepRefs with 8 cases (dot, bracket, string literal, dedup) - Add TestValidate_StepReferences with 6 cases (valid, unknown, forward, alias, self, multiple)
Add AST-based alias expansion, shared expr functions, and division sa… …fety Replace regex-based condition alias expansion with AST patching for correctness. Add common functions to expr conditions from template package. Add static validation for unsafe divisions. Alias Expansion (compile.go): - Replace regex expansion with expr AST visitor pattern - Add topological sort (Kahn's algorithm) for alias dependencies - Detect and report circular dependencies at compile time - Fix false matches (property paths, string literals) - Remove CompiledCondition.Prog field (aliases now expand inline) Shared Functions (tmpl/engine.go): - Add ExprFuncs() returning functions for expr conditions - Include divOr, modOr for safe division/modulo - Include string functions: upper, lower, trim, hasPrefix, hasSuffix - Include collection helpers: len, isEmpty, first, last - Include validation: isEmail, isUUID, isURL, isIP, isNumeric - Note: contains/matches are built-in expr operators Division Safety (compile.go, validate.go): - Add ValidateDivisions() AST visitor - Reject division by literal zero at compile time - Reject dynamic divisors (require divOr for safety) - Allow division by non-zero literals (e.g., x / 2) Documentation (README.md): - Add all condition functions to Available functions table - Document division safety requirement - Note that contains/matches are operators, not functions Tests: - Add TestAliasExpansion, TestAliasChaining, TestCircularDependencyDetection - Add TestAliasInStringLiteral, TestAliasNotMatchPropertyPath - Add TestValidateDivisions, TestValidate_DivisionSafety - Add TestExprFuncs, TestExprFuncs_DivOr, TestExprFuncs_ModOr - Add TestExprFuncs_InConditions for end-to-end function usage
Use snake_case for template context variables
Changes template variable names from PascalCase to snake_case for
consistency with the existing trigger context variables (trigger.client_ip,
trigger.headers, etc.).
Template Context:
- Rename RequestID to request_id
- Rename Timestamp to timestamp
- Rename Version to version
- Rename Result to result with snake_case fields:
- Query -> query
- Success -> success
- Count -> count
- Data -> data
- Error -> error
- DurationMs -> duration_ms
Validation:
- Update valid path prefix from .Result to .result
- Update pre-query validation error message
Tests:
- Update context_test.go assertions for new field names
- Update engine_test.go template strings
- Update benchmark templates
Docs:
- Update README cache key example from .RequestID to .request_id
Add rate limit resets to fix CI E2E test failures Tests passed locally but failed in CI because faster CI execution exhausted rate limit buckets before token refill. Adds reset_rate_limits calls before tests that create data, ensuring deterministic behavior regardless of execution speed. taskapp: - test_delete_task: bucket exhausted by create/update/patch tests - test_trigger_caching: replace sleep 1 with explicit reset - test_complete_task: bucket exhausted by batch_delete test - test_public_id_round_trip: deterministic test isolation crmapp: - test_create_activity: safety margin (5 POSTs = burst limit) shopapp: - test_order_state_pending_to_paid: safety margin (14 POSTs, burst=20)
Add public IDs, template functions, and unified trigger context
Introduces encrypted public IDs to prevent database PK enumeration, adds ~70
template functions for data manipulation, and standardizes template context
under the .trigger namespace. Adds variables config section for externalized
configuration with env file support.
Public ID System:
- Add internal/publicid package with XTEA encryption and base62 encoding
- Add public_ids config section with secret_key and namespaces (name, prefix)
- Add publicID, privateID template functions for encoding/decoding
- Add isValidPublicID expression function for conditions
- Add strict 11-char length validation in base62 decoder
- Add static validation to detect public ID usage without config
Template Context Standardization:
- Move flat context (.Param, .Header, .ClientIP, .Query) to .trigger namespace
- New paths: .trigger.params, .trigger.headers, .trigger.client_ip, .trigger.query
- Add .trigger.cookies for parsed HTTP cookies
- Add template path validation against valid prefixes
- Update all examples, tests, and configs to use new paths
Template Functions (70+ new):
- Math: round, floor, ceil, trunc, abs, min, max, divOr, modOr, zeropad, pad
- Arrays: first, last, pluck, isEmpty
- Maps: dig, pick, omit, merge, keys, values, require, getOr, has
- Types: int64, float, string, bool
- Strings: truncate, substr, split, join, quote, sprintf, repeat
- Validation: isEmail, isUUID, isURL, isIP, isIPv4, isIPv6, isNumeric, matches
- IP: ipNetwork, ipPrefix, normalizeIP
- Encoding: urlEncode, urlDecode, base64Encode, base64Decode, sha256, md5, hmacSHA256
- Date/time: now, formatTime, parseTime, parseTimeOr, unixTime
- IDs: uuid, uuid4, uuidShort, shortID, nanoid
- Conditionals: coalesce, ternary, when
- Formatting: formatNumber, formatPercent, formatBytes
- HTTP helpers: header, cookie
- Safe variants: divOr, modOr, urlDecodeOr, base64DecodeOr, parseTimeOr
- Add bounded regex cache (100 patterns) for matches function
- Remove duplicate int function, keep only int64
Variables Configuration:
- Add variables section with env_file and values support
- Support ${VAR} and ${VAR:default} syntax in values
- Support {{.vars.name}} in templates for config-time expansion
- Add static template rendering for database credentials, ports, etc.
Step Computed Params:
- Add params field to steps for computed parameters via templates
- Evaluate param templates before step execution
- Check step params before trigger.params in SQL parameter extraction
- Add parseInt64 helper with precision handling for large numbers
Step Result Accessors:
- Add .steps.<name>.row shortcut for first row
- Add .steps.<name>.found, .empty, .one, .many boolean shortcuts
Condition Expressions:
- Support compound expressions with && and || operators
- Support negation with ! prefix
- Expand aliases inline within compound expressions
Validation:
- Add template path validation to catch invalid paths like .Param
- Detect publicID/privateID/isValidPublicID usage without public_ids config
- Remove dead ValidateRuntimeTemplate code from config package
Testing:
- Add comprehensive unit tests for all new template functions
- Add TestDecodeInvalidLength for public ID validation
- Add TestValidatePublicIDFunctionUsageWithoutConfig
- Update all existing tests for new .trigger.* paths
- Add tests for step computed params and condition aliases
Examples and Configs:
- Update all example configs to use new template paths
- Add public ID demonstrations to taskapp
- Add HTTPCall workflow demonstrations
- Use relative dates in test seed data
Add comprehensive E2E test suite with shared infrastructure
Expands E2E testing from a single taskapp to four complete application test
suites (taskapp, blogapp, crmapp, shopapp), each exercising different features.
Introduces shared test libraries for HTTP helpers and test runner infrastructure,
eliminating code duplication. Adds robustness improvements including cron panic
recovery and validation warnings for conditional responses.
E2E Test Infrastructure:
- Add e2e/lib/helpers.sh with HTTP wrappers (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS)
- Add assertion helpers: expect_status, expect_json, expect_contains, expect_header, etc.
- Add e2e/lib/runner.sh with shared setup/teardown, coverage support, health checks
- Refactor taskapp_test.sh to use shared libraries
- Change rate limiting test from warning to failure when not triggered
New E2E Test Applications:
- Add e2e/blogapp_test.sh testing content hierarchy, nested comments, post status
workflow, full-text search, tag relationships, pagination, caching patterns
- Add e2e/crmapp_test.sh testing API key auth, role-based access control,
multiple rate limit pools, customer/contact/deal/activity relationships
- Add e2e/shopapp_test.sh testing order state machine, inventory management,
batch price updates, product catalog, stock validation
New Test Data Configurations:
- Add testdata/blogapp.yaml with authors, posts, comments, tags schema and 20+ workflows
- Add testdata/crmapp.yaml with users, customers, contacts, deals, activities and RBAC
- Add testdata/shopapp.yaml with categories, products, orders, inventory and state machine
Makefile:
- Add individual E2E targets: test-e2e-taskapp, test-e2e-crmapp, test-e2e-shopapp, test-e2e-blogapp
- Update test-e2e to run all four test apps
- Update test-cover to collect coverage from all E2E apps
- Add validate-examples target to validate all example configs
- Add lint target for go fmt, go mod tidy, go vet
- Enhance ci target with validation, static analysis, mod tidy check
CI Workflows:
- Simplify ci.yml to use single make ci step
- Simplify release.yml to use make ci, remove verbose comments
- Remove redundant smoke test for help flag
Robustness Improvements:
- Add panic recovery in cron workflow execution with logging
- Add sqlproxy_cron_panics_total metric for panic tracking
- Improve fallback request ID generation with counter for uniqueness
- Add step-level timeout support for HTTPCall steps
Validation:
- Add warning when HTTP trigger has only conditional responses without fallback
- Remove redundant database type default (now required field)
Example Config Fixes:
- Fix 03-caching.yaml to use parameter instead of template interpolation in SQL
- Fix 08-conditions.yaml to use conditional steps instead of SQL CASE with templates
- Fix 09-advanced.yaml to remove template interpolation from SQL
- Fix 10-rest-crud.yaml to use "{}" instead of "" for empty response templates
- Fix examples/README.md table entry (groups -> blocks)
Code Quality:
- Rename http variable to httpClient to avoid shadowing package name
- Add mustCompile test helper for fail-fast compilation errors
- Add documentation comments for template function error handling
- Add comments for reserved but unused StepResult fields
- Add clarifying comment for Cache.Clear() method scope
- Promote expr-lang/expr from indirect to direct dependency
Documentation:
- Add validation warning note to README
- Document sqlproxy_cron_panics_total metric
- Update testing section with new E2E apps table
- Document shared E2E library files
- Remove obsolete taskapp_test.go entries from TESTS.md
Tests:
- Add validation tests for all-conditional response warning
- Add test for unconditional fallback response (no warning)
PreviousNext