If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Epidemiology, Erasmus MC, Rotterdam, the NetherlandsDepartment of Radiology & Nuclear Medicine, Erasmus MC, Rotterdam, the NetherlandsDepartment of Neurology, Erasmus MC, Rotterdam, the Netherlands
Assessment of stroke severity according to the NIHSS on the basis of medical records is feasible and reliable in population-based cohorts of patients with stroke.
•
Our findings facilitate more individualised risk estimates in observational studies that lack prospective in-person stroke severity ascertainment.
•
Our findings extend results from prior research among clinical populations in the U.S. to a wider population outside of specialised stroke centres.
Abstract
Objectives
Stroke severity is an important prognostic indicator of morbidity and mortality, but often not recorded outside of specialised stroke centres. We aimed to develop a scoring rule and validate standardised assessment of the National Institutes of Health Stroke Scale (NIHSS) from medical records.
Methods
We developed a standardised assessment of the NIHSS from medical records. Four trained raters independently assessed the charts of 100 patients with first-ever stroke, randomly selected from the population-based Rotterdam Study cohort. Interrater agreement was determined using the intraclass correlation coefficient (ICC), and Fleiss’ kappa for major versus minor stroke. We validated the scoring method against 29 prospective, clinical NIHSS ratings, using Kendall's tau and Cohen's kappa.
Results
Of 100 included patients with stroke (mean age 80 years, 62% women), 71 (71%) were admitted to hospital and 9 (9%) were seen in outpatient clinic, whereas 20 (20%) were treated exclusively by their general practitioner or nursing home physician. Interrater agreement for retrospective, chart-based NIHSS ratings was excellent when assessed continuously (ICC: 0.90), and for minor versus major stroke (for NIHSS>3: κ=0.79, NIHSS>5: κ=0.78). Interrater agreement was good both for hospital-based and out-of-hospital settings (ICC: 0.97 and 0.75 respectively). Overall, assessment from medical records was in excellent agreement with prospective NIHSS ratings (τ=0.83; NIHSS>3: κ=0.93, and NIHSS>5: κ=0.93). However, for severe stroke (NIHSS>10) retrospective assessment tended to underestimate severity by 1-3 points on the NIHSS, which was accompanied by a somewhat lower interrater agreement for those more severe cases (NIHSS>10: κ=0.62).
Conclusions
Assessment of stroke severity according to the NIHSS on the basis of medical records is feasible and reliable in population-based cohorts of patients with stroke. These findings facilitate more individualised risk estimates in observational studies that lack prospective ascertainment of stroke severity.
Initial stroke severity is an important prognostic factor for outcome after acute stroke, in terms of functional dependency, post-stroke dementia, and mortality.
Whilst the majority of stroke patients who are admitted to the hospital with stroke have substantial neurological deficits, over half of all strokes that occur in the population qualify as minor stroke,
Medical attention seeking after transient ischemic attack and minor stroke before and after the UK Face, Arm, Speech, Time (FAST) public education campaign: results from the Oxford Vascular Study.
Given the important differences between minor and major stroke in prognosis and management, it is important to distinguish patients on the basis of stroke severity, but assessment outside of specialised stroke centres is often hampered by lack of detailed recordings in medical records. This limits individualised risk estimation in for example population-based studies, which by their unselected nature include a substantial portion of patients with stroke in nursing home or community hospital care.
For assessment of stroke severity, the National Institutes of Health Stroke Scale (NIHSS) is the most commonly used validated scale.
Two U.S. studies investigated the accuracy of NIHSS assessment from medical records in specialised acute stroke care facilities, blinded to prospective NIHSS scores. Among 39 clinical trial participants with stroke and 32 consecutive patients in a specialized stroke center, NIHSS assessment from medical records by neurologists and trained researchers showed good interrater agreement (intraclass correlation coefficient (ICC) 0.82 and r2=0.98) and good overall agreement with clinical scores (sensitivity 0.72 and r2=0.94).
In a larger cohort of 451 patients in specialized acute stroke care facilities, record-based scores showed high agreement with clinical scores (r2=0.99),
but chart assessors were not blinded to the clinical scores. In line with the hospital setting, each of these studies included patients mostly with major stroke. Accuracy of rating medical records has been reported lower among those with minor events,
who comprise half of all patients with stroke in the population. Moreover, the degree of completeness for NIHSS items in medical charts varied from 97% in clinical trial participants to 29% in routine teaching hospital care, and may well be lower outside of specialised stroke centres, questioning the feasibility of retrospective chart-based NIHSS scoring in 3 populations.
The aim of this study was to develop a scoring rule and validate standardised assessment of the NIHSS from medical records, that allows to distinguish between minor and major events in population-based cohorts.
Methods
Study population
This study was embedded within the Rotterdam Study, an ongoing population-based study of determinants and occurrence of disease in persons aged 40 years and older. The study comprises 17,931 individuals living in the Ommoord suburb of Rotterdam, the Netherlands. The design of the Rotterdam Study has been described in detail previously.
For assessment of interrater reliability, the current study includes a random sample of 100 participants with first-ever stroke, either ischaemic, primary intraparenchymal haemorrhagic or undetermined, since the fourth examination cycle of the study (i.e., from 2002 onwards). One researcher (JJC) hand-picked a selection of 100 participants with stroke without using a specific selection process, blinded for the event description and severity. For validation of our assessment form against prospective NIHSS ratings, we further selected all 29 stroke cases in the Rotterdam Study, for whom the NIHSS was reported in medical records since 2014.
Stroke ascertainment
Stroke was defined according to the World Health Organization criteria as a syndrome of rapidly developing clinical signs of focal (or global) disturbance of cerebral function, with symptoms lasting 24 hours or longer or leading to death, with no apparent cause other than of vascular origin. History of stroke at baseline was assessed during baseline interview and verified by review of medical records. In addition to follow-up examinations every 3 to 6 years, participants were continuously monitored for incident stroke through linkage of the study database with files from general practitioners (GP) and nursing home physicians, which included discharge letters from any hospital admission. Potential strokes were reviewed by research physicians and an experienced vascular neurologist adjudicated the final diagnosis, as described in detail previously.
Two clinician-scientists extracted terminology commonly used to describe neurological deficits in 29 medical charts of patients with stroke, randomly selected from across hospital care, primary care and nursing home care. We developed a scoring rule for NIHSS grading from medical records on the basis of the derived terminology, supplemented by earlier reports and recently proposed guidelines for record-based NIHSS assessment in the MR-CLEAN registry.
The resulting assessment incorporates elaborate descriptions of each NIHSS item to ensure reproducible ratings despite variation in the original clinical recordings. NIHSS elements that were not described in the charts were marked as missing, yet coded as normal for the analysis, with three exceptions. First, comatose patients (Glasgow Coma Score < 9) received the maximum NIHSS score.
Second, patients with a severe left-hemispheric stroke were always considered to have sensory and visual field defects (1 point each on the NIHSS), if they had all of the following symptoms: aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm and paresis of the right leg. Third, patients with a severe right-hemispheric stroke were always considered to have sensory disturbance, visual field defects and inattention (1 point each), if they had all of the following symptoms: forced gaze deviation to the right, facial palsy, paresis of the left arm and leg.
Non-documented items are automatically scored 0, with the exception of comatose patients (maximum score on all items) and large hemispheric strokes with a specific range of symptoms. Cases receive 1 point on both sensory and visual fields in case of left-hemispheric infarction with all of the following symptoms (aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm, and paresis of the right leg). Cases receive 1 point on sensory, visual fields and extinction and inattention in case of right-hemispheric infarction with all of the following symptoms (forced gaze deviation to the right, facial palsy, paresis of the left arm and leg). ¶ In case of mutism or global aphasia score 3 on item 9 and score 0 on item 10
2
Complete hemianopia
Blink-to-threat absent on one side, unspecified ‘hemianopia’
3
Bilateral hemianopia (including blindness)
Blink-to-threat on neither side
4. Facial Pal8sy
0
Normal symmetrical movements
‘Other cranial nerves intact’
1
Minor paralysis (flattened nasolabial fold, asymmetry on smiling)
‘Mild palsy’, ‘flattened nasolabial fold’ or asymmetry upon movement
2
Partial paralysis (lower face)
‘Moderate’, ‘severe’ or unspecified palsy
3
Complete paralysis (unilateral or bilateral absence of lower and upper facial movement)
Non-documented items are automatically scored 0, with the exception of comatose patients (maximum score on all items) and large hemispheric strokes with a specific range of symptoms. Cases receive 1 point on both sensory and visual fields in case of left-hemispheric infarction with all of the following symptoms (aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm, and paresis of the right leg). Cases receive 1 point on sensory, visual fields and extinction and inattention in case of right-hemispheric infarction with all of the following symptoms (forced gaze deviation to the right, facial palsy, paresis of the left arm and leg). ¶ In case of mutism or global aphasia score 3 on item 9 and score 0 on item 10
2
Severe to total sensory loss (patient is not aware of being touched at all)
Absent sensation or comatose
9. Best language
0
No aphasia; normal
Absent or unreported
1
Mild-to-moderate aphasia (some obvious changes, without significant limitation)
GCS: M6Vaphasia, only expressive/motor aphasia, ‘disturbed fluency’, ‘difficulty speaking’ suggested to be word finding difficulty, unspecified aphasia
2
Severe aphasia (fragmentary expression; inference needed, cannot identify materials)
GCS: M5V2-3/aphasia, receptive or mixed aphasia, ‘severe aphasia’, ‘no speech but good understanding’, ‘gibberish’ or ‘gobbledegook’
3
Mute, global aphasia (no usable speech or auditory comprehension)
GCS: V1, global aphasia, comatose
10. Dysarthria
0
Normal
Absent or unreported
1
Mild-to-moderate dysarthria; slurring but can be understood
‘Mild’ or ‘moderate’ or no further description ‘Sloppy/slurry speech’ of ‘difficult speech’
2
Severe dysarthria; unintelligible slurring or out of proportion to dysphasia, or is mute/anarthric.
Non-documented items are automatically scored 0, with the exception of comatose patients (maximum score on all items) and large hemispheric strokes with a specific range of symptoms. Cases receive 1 point on both sensory and visual fields in case of left-hemispheric infarction with all of the following symptoms (aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm, and paresis of the right leg). Cases receive 1 point on sensory, visual fields and extinction and inattention in case of right-hemispheric infarction with all of the following symptoms (forced gaze deviation to the right, facial palsy, paresis of the left arm and leg). ¶ In case of mutism or global aphasia score 3 on item 9 and score 0 on item 10
2
Profound hemi-inattention or extinction to more than one modality
‘Neglect’, comatose
GCS = Glasgow Coma Score; MRC = Medical Research Council scale from 0 (paralysis) to 5 (normal strength)
¥ If multiple MRC scores are described for a single limb exam, use the lowest MRC for that limb.
Non-documented items are automatically scored 0, with the exception of comatose patients (maximum score on all items) and large hemispheric strokes with a specific range of symptoms. Cases receive 1 point on both sensory and visual fields in case of left-hemispheric infarction with all of the following symptoms (aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm, and paresis of the right leg). Cases receive 1 point on sensory, visual fields and extinction and inattention in case of right-hemispheric infarction with all of the following symptoms (forced gaze deviation to the right, facial palsy, paresis of the left arm and leg). ¶ In case of mutism or global aphasia score 3 on item 9 and score 0 on item 10
Four medically trained raters with various level of clinical experience (one medical student, two medical doctors and one consultant neurologist), were all experienced or trained in clinical administration of the NIHSS, by either online training, in person training on patients or a combination of both.
All raters independently applied the NIHSS scoring rule to the medical records of 100 patients with stroke. In addition, we validated the scoring rule against medical charts of all 29 stroke cases in the Rotterdam Study that contained information on prospective NIHSS scores, after masking the NIHSS scores in the charts.
Statistical analyses
We assessed NIHSS scores continuously, as well as dichotomised at commonly used cut-offs for minor versus major stroke (i.e., NIHSS of 3 and 5). We determined interrater reliability for all 4 raters using ICC, and for the dichotomised measures using Fleiss’ kappa. We determined agreement with medical records for the continuous NIHSS rating by Cohen's kappa and for continuous measures with Kendall's tau.
We computed scatter plots and Bland-Altman plots to visually inspect validation and interrater agreement. In sensitivity analyses, we assessed interrater agreement separately for in-hospital and out-of-hospital stroke. All analyses were performed in SPSS IBM 27.0 (IBM, Armonk, NY).
Results
Population characteristics are tabulated in Table 2. Among 129 patients with first-ever stroke, 81 had ischaemic stroke and 14 haemorrhagic stroke, whereas stroke type was undetermined for the remaining 52 patients. Of the random sample of 100 participants for interrater agreement, 29 were seen exclusively in out-of-hospital care (i.e., by their GP or nursing home physician; n = 20) or in outpatient clinics (n = 9). For scoring validation, NIHSS ratings from routine practice were exclusively available in hospital setting (Table 2).
Interrater agreement between the four raters was excellent for the continuous NIHSS (ICC: 0.90), as well as for identification of major versus minor stroke (for NIHSS>3: κ=0.79; NIHSS>5: κ=0.78; Table 3). However, differences tended to increase with higher stroke severity (Fig. 1), resulting in somewhat lower agreement for the most severe strokes (NIHSS>10: 0.62). Agreement was good both for in-hospital compared to out-of-hospital assessment (ICC: 0.97 [95%CI: 0.96-0.98] and 0.75 [95%CI: 0.57-0.88], respectively). For all trained raters, comparisons with the consultant neurologist ratings are depicted in Fig. 1, with other comparisons presented in Supplementary Fig. 1.
Table 3Interrater agreement and validation for differentiating major from minor stroke.
NIHSS
Interrater agreement (n = 100)
Validation against prospective clinical rating (n = 29)
> 3
0.79
0.93
> 5
0.78
0.93
> 10
0.62
0.93
Total score
0.90
0.83
Interrater agreement is expressed as the intraclass correlation coefficient for total score, and as Fleiss’ kappa for dichotomised NIHSS scores. Agreement for validation is expressed as Kendall's tau for the total score, and as Cohen's kappa for dichotomised NIHSS score.
Fig. 1Interrater agreement for NIHSS grading based on retrospective chart-based ratings. The agreement between ratings by a trained medical student (rater 1), and two clinical research fellows with varying experience in clinical neurology (raters 2 and 3), compared to those by an experienced consultant neurologist (rater 4), depicted on a Bland-Altman plot.
Not a single medical chart was complete for all NIHSS items in the 100 patients of our sample. Items that most often went unreported included extinction/inattention (79%), ataxia (50%), and sensory deficits (51%). Agreement was broadly similar across different NIHSS items (κ=0.65-0.80), with the exception of limb ataxia (κ=0.43). Comparisons for each item are presented in Supplementary Table 1.
Agreement with prospective assessment
Among 29 cases with NIHSS reported in medical charts, median record-based NIHSS was 5 (interquartile range [IQR]: 2-13). NIHSS scores based on medical records were in excellent agreement with prospective NIHSS assessment (R2 = 0.95 and τ=0.83; Fig. 2. a), although for severe strokes compared to prospective NIHSS assessment, retrospective chart-based assessment tended to underestimate severity by a few points on the NIHSS (Fig. 2. b). Differentiation between major and minor stroke was highly consistent between prospective and retrospective chart assessment (Table 3).
Fig. 2Agreement between prospective clinical and retrospective chart based NIHSS grading. The agreement between prospective clinical NIHSS (NIHSS-P) and retrospective chart based NIHSS (NIHSS-R) expressed as the correlation between the two ratings (a), and depicted as a Bland-Altman plot (b).
Charts were complete for all of the NIHSS items in 6 out of 29 cases (20.7%). Frequently unreported items again were extinction/inattention (65.5%), ataxia (58.6%), and sensory deficits (34.5%).
Discussion
In this population-based study of patients with first stroke, we found that retrospective chart based use of the NIHSS correlates closely to prospective NIHSS assessment, and is able to distinguish minor from major stroke with excellent accuracy and high interrater agreement across different care settings. These findings facilitate more individualised risk estimates in observational studies that lack prospective in-person stroke severity ascertainment.
Our findings extend results from prior research in hospital-based settings in the U.S. to a wider, population outside of specialised stroke centres.
Feasibility of record-based NIHSS assessment is particularly valuable in large cohorts that lack the routine, deep phenotyping done in hospital-based cohorts. This applies to the majority of studies of stroke aetiology and prognosis in population-based studies. Our results support the use of record-based NIHSS assessment in such studies, notably to increase informativeness of findings to specific patient populations. Importantly, high interrater agreement for medical charts review can be achieved with limited clinical experience, as long as a basic, formal training in clinical NIHSS assessment is provided.
In routine practice, reporting of specific symptoms in medical charts varies between physicians, necessitating standardised interpretation before NIHSS scores can reliably be derived. We provide a detailed framework to facilitate retrospective chart-based coding. In our study, NIHSS items that were missing most often included sensory deficits, ataxia and extinction/inattention. Although items are often unreported because they are absent in the patient, some elements may mistakenly go unreported during urgent clinical assessment. This may have contributed to the slight underestimation of the retrospective chart-based scores for more severe stroke cases, in line with earlier findings.
This limitation of record-based assessment can be alleviated in part by adding points for visual field defects, sensory and inattention as proposed in our scoring rule, as these symptoms are often present and underreported in large hemispheric stroke.
Some limitations need to be acknowledged when interpreting our findings. First, we were not able to directly validate our scoring method for out-of-hospital assessment, as NIHSS scores are generally not reported by GPs or nursing home physicians. However, interrater agreement was high for out-of-hospital cases, and anecdotally our impressions of stroke severity based on GP and nursing home charts generally matched subsequent inpatient NIHSS for those who were admitted. Second, despite high accuracy and good use for distinguishing major from minor events, record-based NIHSS assessment is probably not sufficiently sensitive to determine change in NIHSS scores and individual NIHSS items, in which improvements of a few points can be very relevant to determine clinical recovery. Third, more attention in the training phase to the scoring of comatose and severely impaired patients might have prevented some of the discrepancy in NIHSS scoring for the most severe strokes in our study. Finally, feasibility of retrospective chart-based NIHSS assessment may differ by way of routine medical assessment and written physical examination in medical charts, potentially hampering generalizability to vastly different health care settings.
Conclusions
In conclusion, stroke severity assessment according to the NIHSS on the basis of medical records is feasible and reliable in population-based cohorts of patients with stroke. These findings will enhance stroke research in population-based studies, often lacking clinical ascertainment of stroke severity.
Statement of ethics
The Rotterdam Study has been approved by the Medical Ethics Committee of the Erasmus MC (registration number MEC 02.1015) and by the Dutch Ministry of Health, Welfare and Sport (Population Screening Act WBO, license number 1071272-159521-PG). The Rotterdam Study Personal Registration Data collection is filed with the Erasmus MC Data Protection Officer under registration number EMC1712001. A written informed consent was obtained from all participants. Additional ethical approval was not required for this study using secondary data.
Author contributions
Jacqueline J. Claus made a substantial intellectual contribution to the data acquisition, analysis, interpretation and drafting the manuscript. Brian B.P. Berghout made a substantial intellectual contribution to data acquisition, interpretation and revising the manuscript critically for important intellectual content. M. Kamran Ikram made a substantial intellectual contribution to the concept and design of the study, data acquisition, interpretation and revising the manuscript critically for important intellectual content. Frank J. Wolters made a substantial intellectual contribution to the concept and design of the study, data acquisition, interpretation and revising the manuscript critically for important intellectual content. He had full access to the data in the study and takes responsibility for data integrity and accuracy of data analysis. All authors approved the final version of the manuscript for publication.
Grant support
The Rotterdam Study is supported by the Erasmus MC and Erasmus University Rotterdam, The Netherlands Organisation for Scientific Research (NWO), The Netherlands Organisation for Health Research and Development (ZonMw), the Research Institute for Diseases in the Elderly (RIDE), the Netherlands Genomics Initiative, the Ministry of Education, Culture and Science, the Ministry of Health, Welfare and Sports, the European Commission (DG XII), and the Municipality of Rotterdam. This study is further funded by the European Union's Horizon 2020 research and innovation programme as part of the Common mechanisms and pathways in Stroke and Alzheimer's disease (CoSTREAM) project (www.costream.eu, grant agreement no. 667375). Additionally, BPB and MKI were supported by the Erasmus Medical Centre MRACE grant (grant number 386070).
None of the funding organisations or sponsors were involved in study design, in collection, analysis, and interpretation of data, in writing of the report, or in the decision to submit the article for publication.
Data availability statement
Data can be obtained upon request. Requests should be directed towards the management team of the Rotterdam Study ([email protected]), which has a protocol for approving data requests. Because of restrictions based on privacy regulations and informed consent of the participants, data cannot be made freely available in a public repository.
Declaration of Competing Interest
The Authors declare that there is no conflict of interest.
Acknowledgments
We thank all staff at the Rotterdam Study research centre, facilitating assessment of participants throughout the years, and acknowledge the support of Jolande Verkroost and Frank J.A. van Rooij as data managers.
Medical attention seeking after transient ischemic attack and minor stroke before and after the UK Face, Arm, Speech, Time (FAST) public education campaign: results from the Oxford Vascular Study.
Non-documented items are automatically scored 0, with the exception of comatose patients (maximum score on all items) and large hemispheric strokes with a specific range of symptoms. Cases receive 1 point on both sensory and visual fields in case of left-hemispheric infarction with all of the following symptoms (aphasia, forced gaze deviation to the left, facial palsy, paralysis of the right arm, and paresis of the right leg). Cases receive 1 point on sensory, visual fields and extinction and inattention in case of right-hemispheric infarction with all of the following symptoms (forced gaze deviation to the right, facial palsy, paresis of the left arm and leg). ¶ In case of mutism or global aphasia score 3 on item 9 and score 0 on item 10
00016-2&id=gr1.jpg){kind=link}
00016-2&id=gr2.jpg){kind=link}