JackFrostbyte/rgb-aura
1
2
Fork 0
Code Issues 24 Pull requests Projects Releases 2 Packages Wiki Activity Actions

RFC: Testing and portability validation strategy across rgb-aura's lifecycle #65

New issue
Closed
opened 2026-05-18 10:28:16 -04:00 by McJuniorstein · 1 comment
McJuniorstein commented 2026-05-18 10:28:16 -04:00
Collaborator
Copy link

Splitting this off from issue #62 as committed in §7 of comment #273.

Why this issue exists

The locked v1.1.0 direction (issue #62, comment #289) introduces three things that make a deliberate testing strategy necessary, not optional:

  1. Schema-driven runtime. A bad schema decision encodes itself into every pack any user ever writes — discovering it after release is expensive. We need a way to surface "the schema is wrong" findings as early as possible.
  2. LLM-generated content. Pack correctness depends on real LLMs producing real output against a real schema, not just on unit tests.
  3. Cross-machine portability is the v1.2.0 thesis. That thesis needs a concrete bar to clear, set before we start trying to clear it.

This issue is for agreeing on the strategy, not for executing tests. Each testing tactic that emerges becomes its own sub-issue.

Hardware in play

Owner Motherboard GPU Fans / ARGB hub Peripherals OpenRGB installed?
JackFrostbyte ASUS B650E-E RTX 5080 Thermaltake 140mm + Fractal 120mm (none in scope yet) Yes
McJuniorstein Gigabyte X870E AORUS RTX 4080 S Cooler Master ARGB hub Razer BlackWidow X (2D) Not yet

The asymmetry is deliberate — different motherboard vendors, different ARGB controllers, different LLM vendors (GPT vs Claude). Useful for the portability story precisely because of the differences.

Lifecycle: who tests what, when

v1.1.0 — Jack only. McJuniorstein's hardware deliberately stays out so the end-to-end loop isn't diluted. McJuniorstein still reviews diffs and runs the test suite locally (no hardware-touching tests).

v1.1.X — McJuniorstein enters as tester #2. This is the first portability signal. Requires McJuniorstein to install OpenRGB. Razer BlackWidow X enters schema as role: ignored — discovered and represented, not driven.

v1.2.0 — second-machine criterion gate. A concrete bar (proposed below) that has to be clearable before tagging v1.2.0.

Proposed initial mechanics (for discussion)

These are the load-bearing decisions to make in this thread. Each is a "confirm, edit, or push back" — not a finished plan.

A. Test evidence on PRs touching the schema or pack runtime

Proposal: PRs that change schema, pack loader, or validator require the author to attach (a) the validator output against the example profile, and (b) the diagnose output against an intentionally-broken example. Hardware-touching PRs add (c) a short log of running the affected animation on real hardware.

Rationale: makes "I tested it" reviewable instead of trust-based.

B. Cadence of cross-machine runs in v1.1.X

Proposal: every v1.1.X PR that changes schema, validator, diagnose, or the pack loader gets verified on McJuniorstein's machine before merge. Pure-doc / pure-test PRs don't need it.

Open question: does "verified" mean McJuniorstein runs locally and comments, or does it mean a CI job that runs against a recorded hardware fixture? (Latter is v1.2.0+ work probably.)

C. v1.2.0 second-machine criterion

Proposal: v1.2.0 cannot tag until:

  1. A fresh LLM-generated pack runs end-to-end on McJuniorstein's machine using only the public prompt package and the docs — no manual fixups.
  2. The same pack, unchanged, also runs on Jack's machine.
  3. The diagnose surface successfully guided at least one LLM through a self-correction of a malformed pack on both machines.

Rationale: "portability" is meaningless without a falsifiable test.

D. Where findings go

Proposal: each cross-machine discrepancy gets a sub-issue against the relevant milestone, never a comment buried in a PR thread. Use a label (portability or similar) so they aggregate.

What this issue does NOT cover

  • Specific test framework choices (pytest fixtures, hardware mocks, etc.) — those land in the PR that introduces them.
  • Active peripheral animation testing for the BlackWidow X — that's v1.2.0 scope per issue #62.
  • CI pipeline design — separate issue when the time comes.

Decisions wanted from JackFrostbyte

  1. A (test evidence on PRs): confirm or edit.
  2. B (cadence in v1.1.X): confirm or edit.
  3. C (v1.2.0 second-machine criterion): confirm or edit — this one is load-bearing because it defines when v1.2.0 is "done."
  4. D (where findings go): confirm or edit.
  5. When should McJuniorstein install OpenRGB? Options: (i) immediately so he can dry-run v1.1.0 PRs locally even though he's not the test target, or (ii) wait until v1.1.0 tags and v1.1.X begins.
Splitting this off from issue #62 as committed in §7 of comment #273. ## Why this issue exists The locked v1.1.0 direction (issue #62, comment #289) introduces three things that make a deliberate testing strategy necessary, not optional: 1. **Schema-driven runtime.** A bad schema decision encodes itself into every pack any user ever writes — discovering it after release is expensive. We need a way to surface "the schema is wrong" findings as early as possible. 2. **LLM-generated content.** Pack correctness depends on real LLMs producing real output against a real schema, not just on unit tests. 3. **Cross-machine portability is the v1.2.0 thesis.** That thesis needs a concrete bar to clear, set before we start trying to clear it. This issue is for **agreeing on the strategy**, not for executing tests. Each testing tactic that emerges becomes its own sub-issue. ## Hardware in play | Owner | Motherboard | GPU | Fans / ARGB hub | Peripherals | OpenRGB installed? | |----------------|---------------------|-------------|-------------------------------------|---------------------------|--------------------| | JackFrostbyte | ASUS B650E-E | RTX 5080 | Thermaltake 140mm + Fractal 120mm | (none in scope yet) | Yes | | McJuniorstein | Gigabyte X870E AORUS| RTX 4080 S | Cooler Master ARGB hub | Razer BlackWidow X (2D) | **Not yet** | The asymmetry is deliberate — different motherboard vendors, different ARGB controllers, different LLM vendors (GPT vs Claude). Useful for the portability story precisely because of the differences. ## Lifecycle: who tests what, when **v1.1.0 — Jack only.** McJuniorstein's hardware deliberately stays out so the end-to-end loop isn't diluted. McJuniorstein still reviews diffs and runs the test suite locally (no hardware-touching tests). **v1.1.X — McJuniorstein enters as tester #2.** This is the first portability signal. Requires McJuniorstein to install OpenRGB. Razer BlackWidow X enters schema as `role: ignored` — discovered and represented, not driven. **v1.2.0 — second-machine criterion gate.** A concrete bar (proposed below) that has to be clearable before tagging v1.2.0. ## Proposed initial mechanics (for discussion) These are the load-bearing decisions to make in this thread. Each is a "confirm, edit, or push back" — not a finished plan. ### A. Test evidence on PRs touching the schema or pack runtime Proposal: PRs that change schema, pack loader, or validator require the author to attach (a) the validator output against the example profile, and (b) the diagnose output against an intentionally-broken example. Hardware-touching PRs add (c) a short log of running the affected animation on real hardware. Rationale: makes "I tested it" reviewable instead of trust-based. ### B. Cadence of cross-machine runs in v1.1.X Proposal: every v1.1.X PR that changes schema, validator, diagnose, or the pack loader gets verified on McJuniorstein's machine before merge. Pure-doc / pure-test PRs don't need it. Open question: does "verified" mean McJuniorstein runs locally and comments, or does it mean a CI job that runs against a recorded hardware fixture? (Latter is v1.2.0+ work probably.) ### C. v1.2.0 second-machine criterion Proposal: v1.2.0 cannot tag until: 1. A fresh LLM-generated pack runs end-to-end on McJuniorstein's machine using only the public prompt package and the docs — no manual fixups. 2. The same pack, unchanged, also runs on Jack's machine. 3. The diagnose surface successfully guided at least one LLM through a self-correction of a malformed pack on both machines. Rationale: "portability" is meaningless without a falsifiable test. ### D. Where findings go Proposal: each cross-machine discrepancy gets a sub-issue against the relevant milestone, never a comment buried in a PR thread. Use a label (`portability` or similar) so they aggregate. ## What this issue does NOT cover - Specific test framework choices (pytest fixtures, hardware mocks, etc.) — those land in the PR that introduces them. - Active peripheral animation testing for the BlackWidow X — that's v1.2.0 scope per issue #62. - CI pipeline design — separate issue when the time comes. ## Decisions wanted from JackFrostbyte 1. **A (test evidence on PRs):** confirm or edit. 2. **B (cadence in v1.1.X):** confirm or edit. 3. **C (v1.2.0 second-machine criterion):** confirm or edit — this one is load-bearing because it defines when v1.2.0 is "done." 4. **D (where findings go):** confirm or edit. 5. **When should McJuniorstein install OpenRGB?** Options: (i) immediately so he can dry-run v1.1.0 PRs locally even though he's not the test target, or (ii) wait until v1.1.0 tags and v1.1.X begins.
JackFrostbyte commented 2026-05-23 01:14:51 -04:00
Owner
Copy link

I agree with the general testing and portability strategy proposed here.

Confirmed decisions:

  • v1.1.0 remains Jack-first for hardware-touching validation.
  • McJuniorstein can review diffs and run the non-hardware test suite locally during v1.1.0.
  • v1.1.X becomes the first cross-machine validation phase.
  • v1.2.0 keeps the second-machine portability gate as a real release criterion.
  • Cross-machine findings should become dedicated sub-issues, not buried in PR comments.
  • PRs touching schema, validator, diagnose, or pack runtime should include reviewable test evidence.

Clarification:

For v1.1.X, "verified on McJuniorstein's machine" should initially mean a local manual run with a short PR comment containing the command/output evidence. CI with recorded hardware fixtures can be deferred to a later issue after the schema and diagnose surfaces stabilize.

McJuniorstein should install OpenRGB before or during early v1.1.0, but his hardware should not become a required merge gate until v1.1.X. That way setup problems are found early without slowing the first schema/runtime work.

I agree with the general testing and portability strategy proposed here. Confirmed decisions: - v1.1.0 remains Jack-first for hardware-touching validation. - McJuniorstein can review diffs and run the non-hardware test suite locally during v1.1.0. - v1.1.X becomes the first cross-machine validation phase. - v1.2.0 keeps the second-machine portability gate as a real release criterion. - Cross-machine findings should become dedicated sub-issues, not buried in PR comments. - PRs touching schema, validator, diagnose, or pack runtime should include reviewable test evidence. Clarification: For v1.1.X, "verified on McJuniorstein's machine" should initially mean a local manual run with a short PR comment containing the command/output evidence. CI with recorded hardware fixtures can be deferred to a later issue after the schema and diagnose surfaces stabilize. McJuniorstein should install OpenRGB before or during early v1.1.0, but his hardware should not become a required merge gate until v1.1.X. That way setup problems are found early without slowing the first schema/runtime work.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
docs/issue-72-animation-pack-architecture
develop
v1.0.1
v1.0
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
JackFrostbyte/rgb-aura#65
No description provided.