If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
1 These authors are contributed equally to the manuscript.
Ahmed Alkhater
Correspondence
Address correspondence to Ahmed Alkhater, Student's Scientific Circle of Neurology and Neurosurgery, Neurosurgery Department, Medical University of Gdansk, Dębinki 7, 80-952 Gdansk, Poland.
Stroke is the second leading cause of death worldwide following ischemic heart disease, and the fifth in the United States. The video-sharing database, YouTube, is the second most popular visited website with more than 2 billion users, thus it's increasingly being used as a medium for delivering health information.
Aim
We aimed to evaluate the quality, reliability and audience engagement of stroke-related YouTube videos.
Methods
In October 2019 we conducted a search on YouTube using 5 keywords: stroke, brain attack, hemorrhagic stroke, ischemic stroke and transient ischemic attack. We selected the first 30 videos from each search query for further analysis. The validated DISCERN instrument was used (a score of 0-5 per question) to assess the videos by four independent raters. We then recorded qualitative data and quantitative data for each video.
Results
After sorting through 150 stroke videos, a total of 101 unique YouTube videos met our inclusion criteria. We found that the mean overall quality of YouTube videos according to DISCERN is of fair quality. Most videos (65.3%) were uploaded by hospitals, mentioned the symptoms of stroke (66.3%), had a doctor speaking (60.4%) and contained diagrams (20.8%).
Conclusion
YouTube is a useful source of gathering information about treatment choices for patients and their families as the quality of YouTube videos is fair. The audience engagement suggestions in our paper may help content creators improve the appeal of YouTube videos.
The internet has become a vital and convenient source for people to search for health information. Many patients discuss what they found online about their condition with their physician.
YouTube, in particular, is a video-sharing database which is increasingly being used as a medium for delivering health information, with more than 2 billion users placing this database at the second most popular in the world.
Studies on the quality and reliability of a wide range of diseases such as glioblastoma, abdominal aortic aneurysms and chronic obstructive pulmonary disease have been analyzed.
The unknown reliability of information concerning a prevalent disease like stroke on media platforms such as YouTube raises the need for analysis of the videos and an audience engagement criterion for video producers to ensure that patients participating with the information are properly educated. This is especially true since stroke is a medical emergency and urgent treatment is essential because the sooner a person receives treatment for a stroke, the less damage is likely to happen.
More and more patients, family members and friends use YouTube as a source of educational information to look up more information or gain knowledge about their disease and how to manage it.
Our aim was to assess the quality and reliability of stroke related videos published on YouTube using the validated DISCERN instrument to evaluate the videos. We also analyzed what qualitative metrics drive audience engagement with a video so that better content may be created in the future.
Materials and methods
Search strategy and data collection
We searched using the video-sharing website YouTube for the following five keywords ‘‘Stroke,” “Brain attack,” “Hemorrhagic stroke,” “Ischemic stroke,” and “Transient ischemic attack.” The results were sorted using YouTube's default search option, ‘‘Relevance-Based Ranking.” The first 30 videos for each keyword were assessed since an internet search engine analysis concluded that more than 90% of users choose from the results listed within the first three pages or “equivalent to 30 videos” of the search results.
Permission from YouTube was not needed to conduct this study as all the data used in our paper was publicly available and no special access was required for collecting the data.
Inclusion and exclusion criteria
150 videos were recorded in the database. Duplicate, clearly non-relevant (e.g. music videos) and videos that were not in English were excluded. The remaining videos were included for further analysis.
Variables extracted
Data collected for each video included: video ID, date uploaded, views, likes, dislikes, number of comments and video duration. The video power index (VPI) was calculated [(like*100/(like + dislike))*(views/day)/100].
Scoring system
All videos were assessed independently by four medical students in the clinical years of their medical education using the validated DISCERN instrument.
All four raters had more than 5 years of experience using DISCERN to evaluate the reliability of medical information from the beginning of their medical studies. DISCERN is a grading scale designed to judge the quality and reliability of a publication as a source of information. Videos are scored based on a five-point scale ranging from poor to moderate to good. Evaluators select scores for fifteen parts of the criteria that serve as quality indicators along with selecting an overall quality rating. Each indicator represents a critical feature or standard that is considered an important characteristic of high-quality information. The DISCERN instrument is seen on Table 1.
Table 1The DISCERN Instrument.
#
Question
Rating
1
Are the aims clear?
1
2
3
4
5
2
Does it achieve its aims?
1
2
3
4
5
3
Is it relevant?
1
2
3
4
5
4
Is it clear what sources of information were used to compile the publication (other than the author or producer)?
1
2
3
4
5
5
Is it clear when the information used or reported in the publication was produced?
1
2
3
4
5
6
Is it balanced and unbiased?
1
2
3
4
5
7
Does it provide details of additional sources of support and information?
1
2
3
4
5
8
Does it refer to areas of uncertainty?
1
2
3
4
5
9
Does it describe how each treatment works?
1
2
3
4
5
10
Does it describe the benefits of each treatment?
1
2
3
4
5
11
Does it describe the risks of each treatment?
1
2
3
4
5
12
Does it describe what would happen if no treatment is used?
1
2
3
4
5
13
Does it describe how the treatment choices affect overall quality of life?
1
2
3
4
5
14
Is it clear that there may be more than 1 possible treatment choice?
1
2
3
4
5
15
Does it provide support for shared decision making?
1
2
3
4
5
16
Based on the answers to all of these questions, rate the overall quality of the publication as a source of information about treatment choices
Videos with overall DISCERN score greater than 3 were considered to be ‘‘good” quality videos. According to the DISCERN Tool, a video with a grade of 3 is ‘‘moderate” and of fair quality. Videos with a score of 3 are useful sources, but ultimately the patient would need additional sources of information. Videos with a score less than 3 were considered to be ‘‘poor” quality; these videos are not considered useful and should be avoided by patients.
Audience engagement analysis
The DISCERN and the VPI score were analyzed against the inclusion of certain qualitative video characteristics (e.g. if it included the symptoms of a stroke). The DISCERN and VPI score were also correlated to see if the most informative videos were the most popular.
Statistical methods
Standard statistical methods were used for all calculations. Descriptive statistics for continuous variables covered mean, range and standard deviation. The Mann Whitney U test was followed by multivariate logistic regression to determine the influence of specific characteristics of the videos on educational quality (the DISCERN score) and audience engagement (the VPI score). For the multivariate tests, a VPI >10 was deemed high and ≤10 was deemed low and a DISCERN score of >42 was deemed high and ≤42 was considered low. The intraclass correlation coefficient was used to ascertain inter-rater agreement. Pearson's r correlation coefficient and Spearman's rank correlation coefficient were used to draw correlations between the DISCERN score and the VPI. A p value below 0.05 was deemed significant. Google Sheets (Google LLC), MedCalc 19.1.3 (MedCalc Software, Acacialaan 22, 8400 Ostend, Belgium) and PAST (Hammer and Harper, Øyvind Hammer, Natural History Museum, University of Oslo) were used for statistical analysis and illustrations.
Results
Video contents
Out of the 150 videos taken into consideration, 101 videos met the inclusion criteria. Fig. 1 illustrates the video content on stroke. It shows that most videos: 67/101 (66.34%) mentioned the symptoms of stroke, 61/101 (60.40%) had a doctor speaking and 58/101 (57.43%) stated the results of treatment. Only a few videos: 21/101 (20.79%) contained diagrams, 21/101 (20.79%) were patient experiences and 15/101 (14.85%) talked about the risk factors for treatment.
Fig. 2 illustrates the source of the videos uploaded. Most videos 66/101 (65.35%) were uploaded by hospitals followed by educational channels 31/101 (30.69%%). None of the videos were uploaded by physicians or patients.
The following were the mean video statistics (with their ranges in parentheses): the view count was 115,875 (107–3,249,878), the number of comments were 28 (0–577), the number of likes were 482 (0–12,000) the number of dislikes were 23 (0–571), the view rate per day was 86 (0.1–2167) and the referrers to the page were 199,037(0–18700000), the duration was 527 (27–3827) seconds, the video description word count was 106 (0–601) and the days since upload was 1,595 (107–4,468).
The following were the mean channel statistics: the channel subscribers were 316,993 (277–9,600,000), the channel daily views were 41,671 (63,9–381,300) and the channel daily subscribers were 212 (0–1,667).
Video quality evaluation
The four raters had a mean total DISCERN score of 42. The mean score per individual rater was 43 (27–70), 42 (27–71), 42 (26–73) and 42 (25–71) respectively. The intraclass correlation coefficient for the absolute agreement for DISCERN was 0.9301 (95% confidence interval 0.9066 to 0.9493) between the four raters; this is regarded as an excellent reliability interpretation. (Fig. 3)
The mean score of question 16 of DISCERN–which requires a holistic judgement of the entire video–was 2.6. The mean score per individual rater was 2.7(2–5), 2.6(2–5), 2.6(2–5), 2.6(1–4) respectively.
Fig. 4 illustrates the mean DISCERN score1 to 5 between all four raters for each of the 16 DISCERN questions. The highest scores overall were indicated by (in descending order) question 1, question 2, question 6 and question 3. These questions respectively ask if the aims are clear, if it achieves its aim, if it is balanced and unbiased and if it is relevant. Questions 7 through 15 have lower scores and mainly ask about the treatment options: how the treatment works, the benefits, the risks, the consequences of no treatment, how the treatment may affect the quality of life and the possibility of more than one treatment.
Fig. 4The mean DISCERN score(1 to 5) between all four raters for each of the 16 DISCERN instrument questions. Abbreviation: Q - question.
As seen on Table 2, there was a significant positive correlation between the DISCERN score and videos that included: the time-window of treatment (β = 1.40117; P = 0.0101), explained anatomy (β = 1.44643; P = 0.0147) and talked about the results of treatment (β = 3.06816; P<0.0001). These three variables independently influenced the DISCERN score.
Table 2Multivariate analysis (logistic regression) of the DISCERN score against the qualitative aspects included in the video and the type of uploading channel (a DISCERN score of >42 was deemed high and ≤42 was considered low). Variables significant in univariate statistics (Mann Whitney U test) were entered into this analysis. Abbreviations: 95% CI, 95% Confidence Interval; Std. Error, Standard Error.
We ran a stepwise model logistic regression to determine the independent influence of the source of upload on DISCERN score, however, all the entered variables were rejected. We ran an enter model logistic regression and found that educational health channels had a higher coefficient regarding the DISCERN score (β = 1.06142; P <0.0001) in comparison to hospital and miscellaneous uploads.
Audience engagement analysis
As seen on Table 3, videos had a higher VPI score when they included radiological images (β = 2.03671; P = 0.0028) or were uploaded by an educational channel (β = 1.00764; P = 0.0265). However, videos had a lower VPI when the speaker was a doctor (β = -2.99875; P <0.0001) and included a patient experience (β = -1.63521; P = 0.0217). The above variables independently influenced the VPI.
Table 3Multivariate analysis (stepwise logistic regression) of the video power index (VPI) against the qualitative aspects included in the video and the type of uploading channel (a VPI >10 was deemed high and a VPI ≤10 was deemed low). Variables significant in univariate statistics (Mann Whitney U test) were entered into this analysis. Abbreviations: 95% CI, 95% Confidence Interval; Std. Error, Standard Error; VPI, Video Power Index.
There was no strong correlation between the DISCERN score and audience engagement (Pearson's r correlation coefficient = 0.38; Spearman's rank correlation coefficient = 0.55).
Top quality YouTube videos
12 out of the total 101 videos (11.8%) came from these four top-quality channels. As seen on Table 4, 1 video out of 101 (0.9%) was uploaded by Holland Hospital and received the highest DISCERN mean score of 69.9. 6 out of the 101 videos (5.9%) were uploaded by Premier health and had a mean DISCERN score of 42. 2 videos out of 101 (1.9%) came from IR Education with a mean DISCERN score of 58.5, 3 videos out of 101 (2.9%) came from UCLA Health with a mean DISCERN score of 54.
Table 4The top five quality stroke videos based on the DISCERN criteria.
DISCERN
Video Title
Uploader
YouTube ID
69.9
Brain Attack: The Latest in Stroke Prevention & Treatment Options
Holland hospital
_L7Gl3Rb9wM
68.5
Management Principles for Spontaneous Intracerebral Hemorrhage
Premier health
E3bfN7V53UE
67.1
SIR-RFS Webinar 5/29/2019: Clinical Care for the Ischemic Stroke Patient
IR Education
RPTmfyAMcjY
65.6
Acute Ischemic Stroke Treatment 2015 by Esteban Cheng-Ching, MD
We found that the overall video quality and accuracy of stroke videos was fair. This indicates that while YouTube is a useful source of information for patients to gain knowledge about stroke, it still has some limitations. Our findings are novel as so far, a content-quality evaluation nor an audience engagement analysis has been done for YouTube videos on stroke.
Only a total of 8 videos out of 101 scored mean DISCERN that was considered good (4 out of 5). Most of the videos did not indicate which source of information was used or any details to support their video and most of them did not mention when the information used in their publications were produced. Approximately 50% of the videos were describing how each treatment works for stroke but only a minority talked about the benefits of each treatment, advantages, disadvantages and what are the risks of each treatment and what would happen if no treatment is used. Moreover, some actually did not mention if there is any other possible treatment. All of these details are important for higher scoring in DISCERN. We propose that the DISCERN instrument be used as a guide for YouTube content creators aiming to produce high quality and trustworthy videos on health-related topics as the 16 question criteria may help provide a thorough informational guideline for a video.
We provided a list of top 5 DISCERN score videos and they can be used as a source for doctors to use for educational purpose or to refer patients so they could learn more about the disease, or as a guide for uploaders on how to make a perfect and highly reliable medical video on YouTube. These videos talked about the symptoms of strokes. They mention the time-window of stroke and how important it is for better prognosis. They talked about treatments, results of treatment and in all of these videos the narrators were doctors. They did not use animation instead they used diagrams and real images with some explanation of the brain anatomy. They were all uploaded by hospitals.
We found that the top 5 VPI scores had poor or fair reliability. This means that likes and number of views or any metric scale does not evaluate the reliability of YouTube videos related to stroke. The reason a video has a high VPI score may either due to the channel being popular or due to its entertaining features.
Audience engagement analysis
We found that videos consistently had more audience engagement (in terms of a higher VPI score) when the video was uploaded from an educational channel rather than a hospital and when a doctor was not speaking. We feel that hospital channels and physicians focus more on telling medical information rather than presenting it in an interesting way. Educational channels, however, put an emphasis on the presentation so that the video is more understandable, memorable and enjoyable to watch. Therefore, we encourage hospitals and physicians to try to present medical information in a more engaging way since the public may appreciate when a physician is speaking. Radiological images resulted in a significantly higher VPI. Thus, we encourage video content creators to include radiological images as it drives audience engagement.
Context
Although stroke is one of the leading causes of morbidity and mortality in the world, no studies have yet been done on the accuracy and trustworthiness of the videos related to stroke. However, YouTube quality analysis studies on other neurological topics, such as glioblastoma treatment and stereotactic radiosurgery, have been done and also show a low or fair quality of videos at best.
Our analysis shows that the quality of stroke videos may be improved by including additional resources in video, treatment options with the benefits and the risks and how it may affect a patient's quality of life.
The value of YouTube videos regarding the social aspects of stroke have been researched. A 2019 study aimed to identify the prevalence of the unmet needs of caregivers of stroke survivors and to evaluate YouTube as a resource for addressing these unmet needs.
They found that YouTube is capable of fulfilling the needs of caregivers of stroke survivors and it is a good source of information to raise the awareness of the issue. Moreover, they found that inadequate information delivery is the most reported unmet need which corresponds with our findings.
Another study analyzed the value of YouTube vlogs on people recovering from a stroke. The authors found that YouTube is a convenient platform for stroke survivors for sharing their experience in this area, such as the course of the disease itself and its treatment. Moreover, the vlogs uploaded by the stroke patients provided many examples of daily struggles and helpful coping strategies to deal with the debilitating consequences of a stroke. Moreover, the vloggers met with the recognition and admiration from the viewers for educating others on this subject which contributes to the recognition of YouTube not only as a medical source but also a social platform for support.
While our study did not focus on the psychological needs and recovery of stroke survivors or their caregivers, we feel that it is important to note these studies as YouTube can also be used as a way for stroke survivors and family members to better cope with the impact a stroke may have on their quality of life.
YouTube provides a convenient way to acquire different types of information yet doctors may not feel confident to refer their patients to YouTube as a source of information due to the existence of false or misleading information.
For this reason, we have included the top highest quality videos which health providers can offer to their patients who wish to learn more. This may be an excellent way to help patients understand the disease, make wise decisions and choose the best approach alongside their health provider.
Limitations
This was a cross-sectional study that included only a part of the information available on YouTube (150 videos) thus the sensitivity could have been increased if we included more videos. However, since 90% of YouTube users do not view videos further than the first 30 videos, we felt that 30 videos per keyword was a reliable sample size.
Some may consider medical students unreliable video raters compared to physicians (despite a strong intraclass correlation coefficient of 0.9301 for DISCERN between the four raters), however, the publisher of DISCERN instrument designed the tool so that it “was not dependent on specialist knowledge of a health condition or treatment.”
This claim was validated in a follow-up evaluation study of the DISCERN instrument where different authors study concluded that “DISCERN could enable both patients and healthcare professionals [emphasis added] to discriminate between the plethora of variable quality information currently available.”
However, we agree that a future study may further look at the differences between specialists and medical students’ evaluation. That being said, YouTube videos are not meant to be a replacement for a specialist's opinion or be a platform where patients self-diagnose themselves and self-start treatment. Instead, YouTube is a platform to educate the general public including stroke patients and their families with relevant information that may guide them to the hospital in time to start treatment (e.g. the principle of “time is brain”, basic treatment options and the common signs and symptoms of stroke). Thus, this study analyzed YouTube as a proper source of information for patients as the title of the paper implies.
Finally, we included only videos in English into our study, therefore videos in other languages may have a different quality of videos.
Future directions
This study may be repeated in a few years to see if there is a difference in content quality especially since new videos are uploaded frequently and YouTube database search results changes with time based on several “relevant” metrics. Further research in the form of surveys may be done to see if patients feel that they are obtaining adequate information about stroke from YouTube and what other websites they reference and consider trustworthy. The best strategies to interact with patients online (i.e. via telemedicine) may also be an area of research.
We also encourage researchers to do similar study to ours in languages other than English to assess the reliability of online medical information for people who do not speak English.
Conclusion
With YouTube being one of the most common online sources used by patients and the general population for acquiring medical information, it's of major importance for physicians and medical institutions to make sure that the information provided to the viewers is of good quality, and contains valid data. This could help improve the outcome of the care provided by physicians to their patients. The DISCERN instrument may be a helpful guide for improving the quality of the videos. In our study the overall DISCERN score for stroke videos on YouTube was “fair” quality which indicates that there is a shortage of information provided to the public concerning stroke on YouTube. We have provided a list of the highest quality videos that physicians may refer their patients to.
Disclosure
The authors have no financial conflicts of interest.
References
Underhill C.
Mckeown L.
Getting a second opinion: health information and the Internet.
30483-3&id=gr1.jpg){kind=link}
30483-3&id=gr2.jpg){kind=link}
30483-3&id=gr3.jpg){kind=link}
30483-3&id=gr4.jpg){kind=link}