Every operationalization of a latent construct is an imperfect attempt to capture an abstract concept. Psychometrics is the science of how best to measure latent constructs as accurately and consistently as possible. Psychometrically-sound instruments can be developed for any dimension such as attitudes and perceptions, behavioral health constructs, personality traits, aptitudes and abilities, and more.
A complete process of psychometric tool development includes:
- creation of the item taxonomy
- Q-sort assessment
- cognitive pretests
- convergent and discriminate validity analyses
- item characteristics analyses
- test-retest reliability & scale internal consistency assessment
- dimension reduction methods including exploratory as well as confirmatory factor analysis
- quantify internal coherence within scales and independence between scales
- norms tracking over time
This process was executed for a military population health assessment tool. The need to assess the well-being of unit personnel has been of interest to the U.S. Air Force (USAF) for many years. The most effective prevention of adverse conditions is to stop symptoms even before they develop; failing that, at least identification of problems in early stages before they have a chance to develop into severe distress.
The U.S. Air Force Integrated Delivery System (IDS) was formed to implement these missions of primary prevention and early intervention, pooling resources from key USAF helping agencies such as family support centers, health and wellness centers, religious services, and counseling services. The overarching goal of the IDS is to improve the well-being of USAF unit members and their families, because well-being has a direct impact on the multiple facets of work efficiency, morale, unit and individual readiness to perform missions, as well as retention.
A psychometric tool was developed to support the work of the IDS with prompt tracking of psychosocial health of USAF units, with measures of negative risk factors as well as positive protective factors that have practical implications for work efficiency and effectiveness. Negative risk factors (e.g., family conflict, financial troubles) were associated with the development of serious distress spanning all areas of life including community, family, and work, whereas positive protective factors (e.g., family support, economic well-being) could facilitate or improve USAF unit members’ abilities to respond to the challenges of life and work.
Instrument development began with the construction of an item taxonomy, pretesting procedures (Q-sort; cognitive interviews), and a pilot web survey to document internal consistency and test-retest reliability, followed by two large-scale Web surveys to derive and confirm a viable factor structure. Because certain items (e.g., child care) were not relevant to all respondents, multiple imputations to replace missing values were necessary before factor analyses could proceed.
Exploratory factor analyses of data from the first web survey revealed an eight-factor solution; and data from the second web survey confirmed that the eight-factor solution could be replicated on an independent sample. Developed in accordance with psychometric principles, the final assessment tool tapped a range of psychosocial factors spanning quality of work life, community life, family functioning, and personal well-being.
Citation: Chang, LinChiat, Lucinda Z. Frost, Susan Chao, and Malcolm Ree. 2010. Instrument Development with Web Surveys and Multiple Imputations. Military Psychology 22: 7-23. <PDF>