INTRODUCTION
Mild cognitive impairment (MCI) is a transitional stage between normal aging and dementia, with 10%–15% of individuals developing dementia annually.1) Given the growing prevalence of dementia, global trends toward population aging, and the significant disruptions in daily functioning caused by dementia,2) the prevention and mitigation of abnormal cognitive decline have become a global public health challenge.3) Although no effective treatment currently exists to reverse dementia, nonpharmacological interventions, such as cognitive training and physical activity, have shown positive effects on cognitive function in individuals with MCI.4) Therefore, early detection of MCI is critical for maintaining independence and reducing the risk of progression to dementia.5)
Several traditional assessment tools for cognitive function, including the Mini-Mental State Examination and the Montreal Cognitive Assessment (MoCA), are widely used for MCI screening. However, they require in-person administration, even when administered through video conference, which has recently been shown to have acceptable reliability and agreement in older adults.6) Other paper-based or computerized alternatives, such as the Addenbrooke’s Cognitive Examination and Alzheimer’s Disease Assessment Scale–Cognitive Subscale, have been developed to improve diagnostic precision; however, they still require trained examiners. Moreover, these assessments are susceptible to practice effects, particularly those arising from repeated exposure to identical stimuli, and are typically administered infrequently (e.g., every 6–12 months). Consequently, they may be less suitable for detecting subtle early cognitive changes indicative of abnormal decline.
Recent studies suggest that smartphone-based cognitive assessments can be administered without an examiner and can incorporate varying stimuli, enabling frequent, repeated, and low-burden cognitive performance measurements in daily life.7) Previous studies have validated several smartphone-based cognitive tasks in diverse populations, including individuals with multiple sclerosis8) and fibromyalgia9) and healthy adults aged 25–65 years.10) Among these tasks, the Symbol Search Task (SST) and Dot Memory Task (DMT) were designed to assess processing speed and working memory, respectively. Validation studies have shown that the SST is moderately to strongly correlated with standard measures of processing speed, whereas the DMT shows moderate correlations with working memory measures that are comparable to those observed for traditional personal computer (PC)-based tasks.8-10)
Evidence suggests that a decline in working memory and processing speed beyond that expected with normal aging may already be present in the early MCI stages.11,12) As these changes may be subtle and difficult to detect in a single assessment, repeated assessments have been suggested as a potentially effective approach for detecting them.13) Despite these indications, the smartphone-based SST and DMT have not yet been validated for older adults. Therefore, the present study evaluated the validity and reliability of smartphone-based SST and DMT in cognitively healthy older adults and examined age-related differences in performance.
In a previous study, the smartphone-based SST and DMT included 12 and two trials per session, respectively, yielding intraclass correlation coefficients (ICCs; i.e., the reliability of a single assessment) of 0.54 and 0.39 across 70 repeated assessments (five sessions per day over 14 days).10) These results indicate that 54% and 39% of the variance in performance reflect between-person differences, respectively. When scores were averaged across five consecutive assessments, ICCs reached 0.90 (SST) and 0.75 (DMT), indicating high reliability (ICC ≥0.90). Based on these findings, the present study increased the number of trials per session to 60 for the SST and 15 for the DMT to improve single-session reliability.
MATERIALS AND METHODS
Participants
This study included 32 older and 28 younger adults. An a priori power analysis for correlation analysis was conducted using G*Power (version 3.1). To detect a medium-to-large effect size (ρ=0.50) with α=0.05 and power=0.80,14) the minimum required sample was 21 participants. Participants were recruited through websites (https://www.jikken-baito.com) and the Bunkyo Ward Silver Human Resources Center. All participants provided written informed consent before participation. Eligibility criteria included no history of major psychiatric or neurological conditions (e.g., Parkinson disease, cerebrovascular disease, or brain injury), age ≥60 years for older adults or 21–29 years for younger adults, independent daily living activities (Tokyo Metropolitan Institute of Gerontology Index of Competence [TMIG-IC] ≥10), normal level of general cognitive functioning (Japanese version of the MoCA [MoCA-J] ≥26), and proficiency in Japanese. The study protocol was reviewed and approved by the University of Tokyo Human Research Ethics Committee (Approval No. 24-390) and conducted in accordance with the Declaration of Helsinki.
Study Procedure
The study consisted of two sessions. In Session 1, participants completed a baseline self-administered questionnaire, followed by the MoCA-J. Subsequently, they completed cognitive tests in the following order: a PC version of the Symbol Digit Substitution Test, Visual Digit Span Test, Symbol Search Task (SST-PC), and Dot Memory Task (DMT-PC). Then, they completed the smartphone versions of the Symbol Search Task (SST-SP) and Dot Memory Task (DMT-SP). Practice trials were provided for all tasks and repeated as needed until participants were familiar with the procedures. Session 2, which included the SST-SP and DMT-SP, was conducted approximately 1–2 weeks later. All sessions were conducted individually in a quiet testing room by trained research staff. Session 1 lasted approximately 1.5 hours, including breaks, whereas Session 2 lasted approximately 10 min. Participants were compensated with 2,000 yen per session. Data were collected from November 2024 to April 2025.
Baseline Characteristics
The baseline questionnaire included questions regarding age, sex, education, depressive symptoms, functional ability, and global cognitive function. Depressive symptoms were assessed using the Center for Epidemiologic Studies Depression Scale (CES-D) (possible range, 0–60), with a score ≥16 indicating possible depression.15) Functional ability was assessed using the TMIG-IC (possible range, 0–13), with a score of ≤9 suggesting difficulty in independent living.16) Global cognitive function was assessed using the MoCA-J (possible range, 0–30), with cognitive impairment defined as a score ≤25. The MoCA-J assesses multiple cognitive domains including attention, executive function, memory, language, visuospatial ability, conceptual thinking, calculations, and orientation.17)
Cognitive Assessments
The Symbol Digit Substitution Test18,19) and Visual Digit Span Test20) were administered on a PC using Inquisit 6 software (Millisecond Software, Seattle, WA, USA), which included computerized adaptations of the tests. The SST-SP, SST-PC, DMT-SP, and DMT-PC10) were developed using R and JavaScript. We used a desktop computer with a 23.8-inch display (DELL XPS8920) for the PC-based tasks, and an Android smartphone (ASUS ZenFone) with a 5.5-inch display for the smartphone-based tasks.
Symbol Digit Substitution Test
The Symbol Digit Substitution Test is a widely used neuropsychological assessment tool designed to evaluate processing speed and attention. Participants were presented with a key at the top of the screen, pairing numbers 1–9 with unique symbols. Below the key, a randomized sequence of symbols appeared, and participants were given 120 seconds to match as many symbols as possible by clicking on their corresponding numbers displayed on the screen using a mouse. The task began when participants clicked "Start," and the times between successive clicks were recorded as response times. The test score was defined as the median response time for correct responses.
Visual Digit Span Test
The Visual Digit Span Test, which requires participants to recall and reproduce digit sequences, was used to assess working memory. In each trial, a sequence of digits ranging from 1 to 9 (excluding consecutive patterns such as "123") was displayed at a rate of one digit per second. After the final digit disappeared, a red dot was displayed for 1 second, which prompted participants to recall and input the recalled digit sequence into a text box. If the response was correct, the next trial increased by one digit in the sequence. If the response was incorrect, the same sequence length was repeated. When the participants made two consecutive errors in the same sequence length, the sequence length was reduced by one digit. The test score was defined as the average length of the sequences correctly recalled.20)
Symbol Search Task
The SST was designed to measure processing speed by assessing how quickly participants processed visual information and made comparisons. In each trial, participants were presented with three symbol pairs in the upper row and two in the lower row of the screen. They were instructed to identify which pair in the lower row matched one of the pairs in the upper row and select it using a mouse (SST-PC) or by touching the screen (SST-SP) as quickly as possible (Fig. 1). The task consisted of 60 trials, and participants responded at their own pace. When a pair in the lower row was selected, all symbols disappeared, and the next stimulus appeared after 200 ms. Before the main task, practice trials were repeated until adequate familiarization was confirmed. The SST score was defined as the median response time for correct answers, with shorter response times indicating a higher processing speed.
Dot Memory Task
The DMT was designed to assess working memory. It consisted of three phases: encoding, distraction, and recall. In the encoding phase, the participants were shown a 5×5 grid with three red dots placed at specific locations and asked to remember their positions. After a 3-second encoding phase, the grid and red dots disappeared, and the distraction phase began, during which participants were asked to identify and touch the "F" letters among an array of "E" letters for 8 seconds. Following the distraction phase, an empty 5×5 grid reappeared, and participants were prompted to recall the locations of the three red dots and indicate them on the grid using a mouse (DMT-PC) or by touching a screen (DMT-SP). Participants then pressed the "Done" button to complete the trial. Each participant completed 15 trials with a 1-second interval between trials (Fig. 2). Before the main task, practice trials were repeated until adequate familiarization was confirmed. The performance in each trial was defined as the error score, which was calculated as the minimum total Euclidean distance between the recalled and correct dot locations. The DMT score was defined as the total error score of 15 trials, with lower scores indicating higher recall accuracy.
Statistical Analysis
Four older adults were excluded because their MoCA-J scores were <26, leaving 28 older adults (62–83 years) and 28 younger adults (21–29 years) in the final analysis. To assess the validity of SST-SP and DMT-SP in older adults, Pearson correlation coefficients were computed for the SST-SP, DMT-SP, SST-PC, DMT-PC, and other PC-based cognitive tasks in older adults. Correlation coefficients were interpreted according to Evans’ criteria: less than 0.20 indicated very weak, between 0.20 and 0.39 indicated weak, between 0.40 and 0.59 indicated moderate, between 0.60 and 0.79 indicated strong, and greater than 0.80 indicated very strong.21) Test–retest reliability of the SST-SP and DMT-SP in older adults was assessed using ICC with a 95% confidence interval (CI) calculated from the scores obtained across the two sessions. ICC values were interpreted as poor (<0.5), moderate (0.5–0.75), good (0.75–0.9), and excellent (>0.90).22) We also assessed the split-half reliability to examine the internal consistency of the SST-SP and DMT-SP in older adults. The SST-SP and DMT-SP trials were divided into odd- and even-numbered trials, and the correlation between their scores within each task was calculated for older adults using the Spearman–Brown correction,23) with coefficients ≥0.70 acceptable, ≥0.80 good, and ≥0.90 excellent.24) Welch’s two-sample t-test was used to assess group differences in SST-SP and DMT-SP scores between older and younger adults. Statistical significance was set at p<0.05. All statistical analyses were conducted using R software (version 4.4.2).
RESULTS
Table 1 presents participant demographics, including age, sex, educational level, and MoCA-J, CES-D, and TMIG-IC scores, indicating high education levels and no signs of dementia, depression, or functional difficulties.
Table 2 shows cognitive assessment scores and Pearson correlations in Session 1 for older adults. The SST-SP was strongly correlated with SST-PC (r=0.83, p<0.001) and moderately correlated with the Symbol Digit Substitution Test (r=0.50, p=0.006). In contrast, its correlation with the Visual Digit Span Test was weak and not statistically significant (r=−0.36, p=0.060). The DMT-SP was strongly correlated with DMT-PC (r=0.86, p<0.001) and moderately correlated with the Visual Digit Span Test (r=−0.47, p=0.011). In contrast, its correlation with the Symbol Digit Substitution Test was weak and not statistically significant (r=0.27, p=0.165).
Regarding the SST-SP and DMT-SP scores in Sessions 1 and 2 among the older adults, the mean SST-SP score (i.e., reaction time) was 2,204.9±398.1 ms in Session 1 and 2,104.6±489.6 ms in Session 2. The mean DMT-SP score (i.e., total error score) was 17.4±12.5 in Session 1 and 17.0±11.6 in Session 2.
In older adults, ICC of the SST-SP between Sessions 1 and 2 was 0.760 (95% CI, 0.545–0.822; p<0.001), indicating good test–retest reliability (Table 3, Fig. 3); thus, 24% of the total variance (1 − ICC) reflected intra-individual variability across sessions. The ICC for the DMT-SP was 0.661 (95% CI, 0.385–0.828; p<0.001), indicating moderate test–retest reliability (Table 3, Fig. 3), with 34% of the total variance reflecting intra-individual variability. The mean interval between sessions was 10.1±3.0 days.
In older adults, the split-half reliability of the SST-SP was 0.934 (95% CI, 0.858–0.970; p<0.001) and 0.963 (95% CI, 0.920–0.983; p<0.001) for Sessions 1 and 2, respectively. The reliability of the DMT-SP was 0.732 (95% CI, 0.412–0.878; p=0.001) and 0.874 (95% CI, 0.723–0.942; p<0.001) for Sessions 1 and 2, respectively (Table 3).
Reaction time for the SST-SP was significantly longer in older adults (mean 2,204.9±398.1 ms) than in younger adults (mean 1,885.2±525.1 ms; p=0.008). The total error score on the DMT-SP was significantly higher in older adults (mean 17.4±12.5 a.u.) than in younger adults (mean 10.5±9.0 a.u.; p=0.017) (Table 4).
DISCUSSION
This study evaluated the validity and reliability of two smartphone-based cognitive tasks, the SST-SP and the DMT-SP, in older adults with no history of depression, dementia, or functional difficulties. Construct validity was examined through correlations with standardized cognitive tests, test–retest reliability, internal consistency (split-half analysis), and performance differences between older and younger adults. To the best of our knowledge, this is the first study to assess smartphone-based task validity and reliability in older adults.
Previous studies conducted on individuals with multiple sclerosis,8) fibromyalgia,9) or healthy adults aged 25–65 years10) validated smartphone-based cognitive tasks by examining their correlation with standardized neuropsychological measures. For instance, smartphone-based SSTs showed moderate-to-strong correlations with conventional processing speed measures (Symbol Digit Modalities Test, National Institutes of Health [NIH] Toolbox Pattern Comparison Test), whereas correlations with working memory tasks were weak, supporting their validity for evaluating processing speed. In contrast, smartphone-based DMTs showed moderate correlations with traditional working memory tasks (Operation Span, Paced Auditory Serial Addition Test, NIH Toolbox List Sorting Working Memory Test) and weaker correlations with processing speed, indicating that DMTs may be more specifically suited for assessing working memory performance.8-10)
Although smartphone-based assessments are considered promising tools for monitoring cognitive performance and detecting subtle or early cognitive decline,25) the validity of smartphone-based SST and DMT in older adults has not been specifically examined. To address this gap, we examined the validity of the SST-SP and the DMT-SP in cognitively healthy older adults. SST-SP scores showed a moderate correlation with Symbol Digit Substitution Test scores and a weak correlation with Visual Digit Span Test scores. Conversely, DMT-SP scores showed a moderate correlation with Visual Digit Span Test scores and a weak correlation with Symbol Digit Substitution Test scores. In addition, the SST-SP and DMT-SP scores were strongly correlated with their corresponding PC-based tasks (SST-PC and DMT-PC, respectively). These results suggest that the SST-SP and DMT-SP may be useful for evaluating processing speed and working memory, respectively, in older adults.
Regarding reliability, a previous study reported single-administration ICCs of 0.54 for SST and 0.39 for DMT.10) In that study, the SST consisted of 12 trials, and the DMT consisted of two trials. In contrast, in our study, the estimated single-administration ICCs were 0.77 for the SST-SP and 0.65 for the DMT-SP; the SST-SP consisted of 60 trials, and the DMT-SP consisted of 15 trials. These findings suggest that the smartphone-based tasks used in our study provide more stable and consistent single-administration estimates of cognitive performance than the previously reported smartphone-based SST and DMT tasks. Moreover, the SST-SP demonstrated good test–retest reliability, and the DMT-SP demonstrated moderate test–retest reliability. Both tasks also showed acceptable-to-good internal consistency within a single session, as indicated by the split-half reliability analyses. These results also support the reliability of the SST-SP and DMT-SP performances in a single administration.
The SST-SP demonstrated high split-half reliability (r = 0.93, 0.96), indicating that the number of trials in a single assessment may be sufficient to obtain scores with excellent reliability (≥0.90). In contrast, the split-half reliability of the DMT-SP was moderate to good (r=0.73, 0.87), suggesting that the number of trials in a single assessment of the DMT-SP may be sufficient for research purposes. In addition, the results indicated that more trials are required to achieve excellent reliability. However, increasing the number of trials may introduce participant burden, fatigue, and disengagement, particularly in older adults, which may affect the validity and feasibility. This issue should be investigated further in future studies.
For the SST-SP and DMT-SP, approximately 24% and 34% of the variance in cognitive performance scores, respectively, was attributable to intra-individual variability. Although intra-individual variability includes random variances in performance scores, previous studies have identified various factors that may contribute to systematic variability in cognitive performance, including fatigue, mood, sleep quality, attentional lapses, and contextual or environmental stressors.26,27) Additionally, greater intra-individual variability has been linked to early neural disruption and an increased risk of dementia, particularly Alzheimer’s disease.28,29) Thus, identifying the factors that drive intra-individual variation may provide deeper insights into real-world cognitive functioning and serve as potential causes of abnormal cognitive decline in older adults.
Consistent with the well-established findings on cognitive aging,30,31) significant age-related differences were observed in both smartphone-based tasks. Older adults exhibited slower processing speeds and more working memory errors than younger adults, suggesting that smartphone-based assessments can capture age-related cognitive changes. In addition, the SST-SP showed a moderate correlation with the Symbol Digit Substitution Test (r=0.50, p=0.006), and the DMT-SP showed a moderate correlation with the Visual Digit Span Test (r=−0.47, p=0.011), further supporting the convergent validity of smartphone-based tasks. Given that these conventional measures are sensitive to MCI-related changes in processing speed and working memory32,33) and that smartphone-based tasks can be administered repeatedly with varied stimuli, SST-SP and DMT-SP may be useful for detecting early signs of cognitive decline beyond normal aging. Future studies should examine this possibility in individuals with an elevated MCI or dementia risk.
This study has some limitations. First, all the participants were daily smartphone users with relatively high educational levels. Thus, the generalizability of these results to the broader older adult population, including individuals with limited digital literacy and relatively low educational levels, remains unclear. Future research should examine the influence of smartphone proficiency and educational levels on task performance. Second, the older adult participants in this study were cognitively healthy, with MoCA-J scores ≥26. Further validation is needed to determine whether our smartphone-based tasks can reliably distinguish between healthy aging and cognitive impairments such as MCI. Third, the performance in Session 2 was better than that in Session 1. This difference may reflect practice/familiarization effects, as well as order-related fatigue. Although the participants completed a practice/familiarization session before the main tests in Session 1, additional practice-related gains may have occurred. Such gains may partly reflect offline learning (i.e., consolidation), whereby performance can improve between sessions even without further practice, for example, through consolidation processes occurring during the intervening days (including sleep). Moreover, in Session 1, smartphone-based tasks were administered after PC-based assessments, whereas in Session 2, only smartphone-based tasks were administered. Thus, participants may have experienced greater fatigue in Session 1, which may have contributed to their poorer performance than in Session 2. These practice-related gains and order-related fatigue may have increased the intra-individual variability between Sessions 1 and 2, thereby contributing to an underestimation of test–retest reliability. Given the possibility of offline learning effects, performance at the initial assessment in repeated-measures designs should be cautiously interpreted.
In summary, we confirmed the validity and reliability of smartphone-based cognitive tasks to assess processing speed and working memory in cognitively healthy older adults. These tasks may allow self-administered, repeated, momentary, and ecological assessments, thereby offering new strategies for assessing cognitive performance and/or ability in older adults in both research and clinical settings. Future studies are required to explore whether these tests are useful for the early MCI detection.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation
