This study investigated the inter-rater reliability and validity of the Fulltrack AI application, to measure ball speed under a range of cricket training conditions in comparison to a radar gun. Ball speed (km/hr) of 1081 deliveries (pace =783; spin= 298) from a range of training sessions and conditions were recorded simultaneously using a radar gun (Stalker Pro IIs) and iOS device running Fulltrack AI (v1.13.1). Statistical analyses were conducted in R Statistical Software. Reliability was assessed with standard error of measurement (SEM), coefficient of variation (CV) and intraclass correlation coefficient (ICC). Agreement was assessed using Bland Altman’s 95% limits of agreement (LOA). Validity was assessed using generalised additive models (GAMs). Pace deliveries were associated with good agreement (ICC: 0.87–0.90, CV: 2.56–3.13%), whilst spin deliveries demonstrated lower agreement (ICC: 0.72–0.76, CV: 3.08–4.33%). LOA established poor to fair levels of agreement, exceeding maximal allowable differences (>3%). GAMs identified Fulltrack AI overestimated ball speed (pace: estimate 0.72–0.77 m/s, SE =0.34–0.34; spin: estimate 1.09–1.18 m/s, SE =0.23–0.25) when compared to the radar gun. Full track AI is an ecologically valid and reliable field-based method for measuring ball speed. However, caution is warranted given the significant overestimation of ball speed in contrast with a radar gun, even after controlling for different training conditions, suggesting software could benefit from refinement.