Development of a novel hybrid cognitive model validation framework for implementation under COVID‐19 restrictions

Stone, Paul B.; Nelson, Hailey Marie; Fendley, Mary E.; Ganapathy, Subhashini

The purpose of this study was to develop a method for validation of cognitive models consistent with the remote working situation arising from COVID‐19 restrictions in place in Spring 2020. We propose a framework for structuring validation tasks and applying a scoring system to determine initial model validity. We infer an objective validity level for cognitive models requiring no in‐person observations, and minimal reliance on remote usability and observational studies. This approach has been derived from the necessity of the COVID‐19 response, however, we believe this approach can lower costs and reduce timelines to initial validation in post‐Covid‐19 studies, enabling faster progress in the development of cognitive engineering systems. A three‐stage hybrid validation framework was developed based on existing validation methods and was adapted to enable compliance with the specific limitations derived from COVID‐19 response restrictions. This validation method includes elements of argument‐based validation combined with a cognitive walkthrough analysis, and reflexivity assessments. We conducted a case study of the proposed framework on a developmental cognitive model of cardiovascular surgery to demonstrate application of a real‐world validation task. This framework can be easily and quickly implemented by a small research team and provides a structured validation method to increase confidence in assumptions as well as to provide evidence to support validity claims in the early stages of model development.