Protect the test from the development process
A model should be evaluated on evidence that did not guide its construction. If analysts repeatedly inspect the same test results and adjust the system accordingly, that test gradually becomes another development set. Keep a genuinely untouched evaluation and record who has accessed it.
Sports records make separation difficult. The same athlete can appear across matches, clubs and seasons. A random row split may place nearly identical observations on both sides. Grouping by athlete helps answer whether a system generalises to unfamiliar people. A later-season test addresses a different question: whether it travels forward in time. Both may be needed.
External validation asks whether performance holds in another organisation or competition with suitable permissions and compatible measurement. Results should describe the new setting clearly. A successful test within one well-resourced academy cannot establish reliability for every community club, region or sporting role.
Count the opportunity costs
Accuracy alone can hide the errors that matter most. If relatively few athletes reach a narrowly defined outcome, a system can appear accurate by rarely recommending anyone. Precision describes how many flagged cases meet the outcome; recall describes how many outcome-positive cases were found. Neither settles the ethical trade-off.
A false positive may lead to an extra observation session, travel or disappointment. A false negative may remove a route into support. Those costs depend on the programme. Discovery often justifies a wider net than final selection, provided follow-up capacity exists and invitations are communicated honestly.
There is another difficulty: athletes not selected may never receive the support needed to demonstrate the outcome. Their later record is not a neutral account of what could have happened. An audit should acknowledge this selective-label problem rather than treating all historical non-selections as confirmed failures.
Assumption
A high overall accuracy means the shortlist is safe to use.
Observation
The cost and distribution of missed opportunities can remain unacceptable despite a strong average.
Fairness requires a stated purpose
Compare relevant error rates, coverage and calibration across groups where lawful, justified and sufficiently supported by the sample. Consider age bands, competition levels, geography and access pathways. Sensitive characteristics require particular care: collecting them for an audit needs a valid basis, appropriate safeguards and a clear explanation.
No single fairness metric resolves every conflict. Equal invitation rates, equal error rates and comparable probability estimates can pull in different directions when outcome prevalence differs. State which harms the programme is trying to reduce and involve affected people in interpreting the trade-offs.
Small groups create uncertainty and disclosure risks. Suppressing an identifiable table may be necessary, but it must not become a reason to ignore the group. Consider confidential review, wider observation or additional carefully collected evidence. Fairlearn offers technical tools for examining disaggregated results; it cannot decide the programme’s values or legal responsibilities.
The Equality and Human Rights Commission provides a UK reference point for understanding discrimination and equality duties. Legal applicability depends on the organisation and decision. A technical fairness dashboard is not a substitute for suitable legal advice.
Observe the system before giving it authority
A prospective shadow phase can generate recommendations without allowing them to control access. Compare those recommendations with existing practice, investigate disagreements and record whether the tool changes observation coverage. Predefine evaluation questions so success is not chosen retrospectively from whichever metric improved.
Once a tool influences behaviour, the environment changes. Coaches may alter training to improve visible metrics. Analysts may preferentially collect data from highly ranked athletes. Those feedback loops can make a system appear increasingly confident while narrowing its view. Monitor the collection process as well as the output.
The European Union’s AI Act uses a risk-based regulatory structure, but a system’s obligations depend on its actual use and jurisdiction. Do not assume every sporting discovery application has the same classification. The OECD AI Principles provide a broader reference for transparency and accountability, without replacing applicable law.
Make the release decision reviewable
A validation checklist
- The intended decision, population and follow-up action are documented.
- Training and testing are separated at an appropriate athlete and time level.
- Results include uncertainty, coverage and a meaningful baseline.
- Errors are examined for unequal impact and opportunity costs.
- Affected people can request correction and meaningful review.
- An accountable owner can pause the system when evidence deteriorates.
Set monitoring and suspension criteria before launch. A change in cameras, competition rules, training population or data supplier may invalidate earlier evidence. Keep model versions and review records so a recommendation can be reconstructed later.
Validation is not a ceremony that permanently approves a tool. It is an ongoing argument about whether the evidence remains strong enough for the current use. If that argument cannot be made clearly, restrict the tool to exploratory analysis or stop using it for decisions.