test(billing): shadow-mode meter validation (ENG-2879)#6451
Draft
Flo4604 wants to merge 1 commit into
Draft
Conversation
Contributor
This was referenced Jun 13, 2026
This was referenced Jun 13, 2026
Member
Author
chronark
approved these changes
Jun 13, 2026
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
45f2f57 to
7fe7474
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
4dc389e to
2e1f9b4
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
7fe7474 to
ec0117c
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
3fd3e35 to
410cf69
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
952a8eb to
83862e9
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
410cf69 to
6d1d000
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
83862e9 to
30a90e0
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
6d1d000 to
bcae9fb
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
30a90e0 to
f9a8821
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
bcae9fb to
e7688d4
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
f9a8821 to
65eeadb
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
e7688d4 to
f1298d4
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
65eeadb to
85b64ed
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
f1298d4 to
e4c9ed4
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
85b64ed to
c048de0
Compare
Flo4604
force-pushed
the
eng-2879-validate-meter-query-in-shadow-mode
branch
from
e4c9ed4 to
090e8b9
Compare
Flo4604
force-pushed
the
eng-2897-implement-billing-for-active-keys
branch
from
c048de0 to
4409850
Compare
Fixture tests proving the meter query maps to known dollar bills, plus a 'dev stripe shadow' command that diffs a workspace's live ClickHouse usage against the values Stripe is holding for its meters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validates the Deploy meter query before anyone is billed (ENG-2879), the gate in front of ENG-2880. Two halves: fixture tests that prove the query maps to a known dollar bill, and a shadow command that diffs live ClickHouse usage against what Stripe actually holds.
Fixtures (pkg/clickhouse)
TestMeterUsageHandComputedBill: seeds a workload whose bill is known by arithmetic, not history — one container at exactly 1 vCPU, 2 GiB memory, 10 GiB disk steady for one hour, plus 240 MiB egress. Runs the productionGetInstanceMeterUsage, asserts the natural units (3600 CPU-s, 2 GiB-hours, 10 GiB-hours, 240 MiB), then converts through the worker's unit path and the Terraform catalog rates to a hand-computed $0.06387555. If the query drifts, the dollar figure moves and the test fails.TestMeterUsageStraddlesWindow: a container's life crosses both window edges; only intervals with both endpoints in-window are billed, so the last in-window sample contributes no delta. Documents the month-boundary rule: at most one ~15s sub-sample interval is dropped at the edge (under-count, never over-charge).These join the existing meter mechanics tests (clean series, 2-min gap drop, restart boundary, multi-workspace, window exclusion).
Shadow command (dev stripe shadow)
mise run unkey -- dev stripe shadow --workspace ws_... --customer cus_... [--month YYYY-MM]recomputes a workspace's month-to-date usage straight from ClickHouse (the same queries + unit conversions the worker uses) and diffs it per meter against the value Stripe is holding (the "last"-aggregated meter event summary = the latest MTD value the hourly push delivered). Prints a per-meter clickhouse-vs-stripe table with a verdict and exits non-zero on structural drift.Tolerance is 0.1%: the hourly push lags the live query by up to an hour, so small gaps are expected staleness, not bugs. For a zero-drift check, run the push (or trigger the restate handler) immediately before.
How to validate end to end (the operator's pre-launch check)
unkey dev seedcheckpoints for a workspace into local ClickHouse.dev stripe shadow --workspace ... --customer ...-> every meter "ok".dev stripe clock advanceto roll the month and confirm the close stamps and finalizes correctly.Verified:
bazel test //pkg/clickhouse:clickhouse_test //cmd/dev/stripe:stripe_test, full build, fmt.