53 lines
4.2 KiB
Markdown
53 lines
4.2 KiB
Markdown
# Balance benchmarks
|
||
|
||
The balance benchmark suite provides repeatable evidence for tuning World of Warcraft, CoA, and Runes of Magic classes. It runs fixed builds through versioned damage, tanking, healing, movement, stacking, and resource-endurance scenarios. The goal is to reveal balance changes and outliers, not to name a universally best class.
|
||
|
||
## Default cohort
|
||
|
||
The default comparison cohort is level 60, matching CoA's authored cap while keeping the three content sources in one cohort. It uses the `coefficient-empty` preset: no equipment, so the first cohort isolates base kits, scaling coefficients, talents, and resources. Gear-normalized cohorts should be added separately rather than mixed with this one.
|
||
|
||
Every comparison must use the same gear preset, scenario version, baseline policy ID, duration scale, and seed set. A changed level, gear preset, scenario, policy, or app version creates a new cohort and should not be compared directly with an older one.
|
||
|
||
Run the suite with:
|
||
|
||
```sh
|
||
npm run balance:benchmark
|
||
```
|
||
|
||
Use `npm run balance:smoke` for a short end-to-end check. The full command accepts comma-separated `--class`, `--role`, `--scenario`, and `--seeds` filters, plus `--level`, `--duration-scale`, and `--output`. For example:
|
||
|
||
```sh
|
||
npm run balance:benchmark -- --class priest,rom-priest --role healer --seeds 1,2,3
|
||
```
|
||
|
||
The command is the intended stable entry point for the benchmark runner. Its output directory contains:
|
||
|
||
- `matrix.json`: raw run reports, build and scenario definitions, seed metadata, validation results, ability details, resources, and event counts.
|
||
- `matrix.csv`: one row per scenario, build, and metric, with observed/null counts plus median, p10, and p90 values.
|
||
- `summary.md`: a concise role-separated comparison for review and pull-request evidence.
|
||
|
||
Keep the JSON artifact when investigating a result. The CSV and Markdown are derived views and can be regenerated from it.
|
||
|
||
## Reading results
|
||
|
||
The reported center is the median across validated seeded runs. The p10–p90 interval shows run-to-run spread; a wide interval calls for more seeds or investigation before tuning. Invalid runs are counted but excluded from distributions. An early death or other scenario stop can be a valid outcome and should not be discarded merely because the full duration was not reached.
|
||
|
||
Interpret each role as a vector of outcomes, not a single ranking:
|
||
|
||
- Damage builds emphasize DPS and priority-target damage, with deaths and mechanics failures alongside throughput.
|
||
- Tanks emphasize damage taken per second, mitigation, avoidance, threat control, minimum health, survival, and their damage contribution.
|
||
- Healers emphasize healing per second, consumed absorbs, overheal, party health, deaths, and time to resource failure.
|
||
- Movement and stack scenarios add movement interrupts and stack compliance to the relevant role vector.
|
||
|
||
No composite score is produced. Compare a build only with other builds in the same scenario version and cohort, and inspect the underlying metrics before making a balance change. A null time to resource failure means the resource was not exhausted during that run; it is not zero. Similarly, a null time to first death means no death occurred.
|
||
|
||
## Limitations
|
||
|
||
- The baseline policy is a deterministic, maintainable reference policy, not an optimal player rotation. Skilled play may outperform it, and policy changes can move results without a class-data change.
|
||
- The initial `coefficient-empty` cohort does not model production gear breakpoints or itemization. Add explicit equal-budget gear presets for that work.
|
||
- Runes of Magic coverage is primary-class-only initially. Secondary-class and dual-class interactions require their own explicitly versioned cohort when supported.
|
||
- Results cover mechanics represented by the simulator and scenarios; they do not replace live encounter playtesting, usability review, or PvP testing.
|
||
- Small seed sets can miss rare random outcomes. Treat p10 and p90 as descriptive bounds for the sampled runs, not guarantees.
|
||
- Total metrics depend on scenario duration. Compare rates or fixed-duration totals only within the same scenario definition.
|
||
- Gear-stat budgets must be equivalent, but equivalent budgets do not guarantee identical interactions with every class kit.
|