Scientific Practice Regarding Visual Accessibility of Research Papers: Pitfalls and Recommendations PDF Free Download
1 / 65/65
100%
1
Scientific Practice Regarding Visual Accessibility of Research Papers:
Pitfalls and Recommendations
Anna Katharina Probst
Snr 2064537
Master's Thesis
Communication and Information Sciences
Specialisation New Media Design
Department Communication and Cognition
School of Humanities and Digital Sciences
Tilburg University, Tilburg
Supervisor: Dr. E. van Miltenburg
Second Reader: Dr. M. Shin
April 2023
2
Abstract
The extent to which graphs in research papers published in the proceedings of the ACM CHI 2020
Conference
1
meet visual accessibility guidelines will be investigated in this quantitative content
analysis. The goal is to establish the status quo of methods to make data visualisations accessible for
persons with low vision through design choices and textual input. A cross-section of 399 visualisations
from the conference papers was assessed based on their type of graph, the design, such as colour,
contrast and redundancy gain, and figure title, Alternative text (Alt-Text) and long description. Through
semi-automated colour and contrast checks and content analyses of text provision, it was established
whether graphs satisfied the medium accessibility criteria of the World Wide Web Consortium (W3C)
for accessibility. From the results a significant lack of Alt-Text (85%) and their quality and 80% missing
redundancy gain in visually providing information about the graphs in more than one way became
visible. Furthermore, the colour contrast between the bars or lines was not distinct enough from the
background in 80% of cases and from each other in 75%. Positively, the long description was present
in 90% of the sample, albeit often with room for improvement. Assessed visualisations of the CHI 2020
conference match accessibility guidelines quite limitedly. As the researcher was sighted, it cannot be
ruled out that the assessed papers may require additional adjustments besides the highlighted problems.
Conducting research with visually impaired people could help explore the accessibility difficulties
comprehensively. Significant problems emerged for design and text choices, showing that neither
relying on vision nor screen readers would be sufficient for users with low vision to comprehend the
visuals completely. These design and text problems show that the accessibility guidelines for
conferences and journals should become more prominent to enable access for visually impaired people.
Lastly, visualisation software should also transition to more accessible pre-sets to support researchers
in providing accessible documents.
Keywords: visual impairment, digital inclusion, blindness, academia, Alt-Text, disability, WCAG
1
Association for Computing Machinery (ACM) Special Interest Group (SIG) Computer–Human Interaction
(CHI) Conference on Human Factors in Computing Systems 2020.
3
Table of Content
Scientific Practice Regarding Visual Accessibility of Research Papers: Pitfalls and Recommendations ……………………..4
Accessibility Regulations ...................................................................................................................................................... 5
Scope .................................................................................................................................................................................... 6
Societal & Scientific Relevance ............................................................................................................................................ 7
Theoretical Framework .............................................................................................................................................................. 7
Key Terms ............................................................................................................................................................................ 8
Low Vision ....................................................................................................................................................................... 8
Aids for Visual Impairments ............................................................................................................................................ 8
Affected Individuals and Variations ................................................................................................................................. 9
Literature on Accessibility .................................................................................................................................................. 10
Accessibility Guidelines ..................................................................................................................................................... 11
Application of Guidelines .............................................................................................................................................. 12
Examples of Guidelines ................................................................................................................................................. 13
Applicability of Guidelines to Visuals ........................................................................................................................... 15
Accessibility Mistakes in Graph Types ............................................................................................................................... 17
Exhaustiveness and Universal Application of Accessibility Guidelines ............................................................................. 19
Limited Accessibility .......................................................................................................................................................... 21
Achieving Accessibility in Software ................................................................................................................................... 21
Methodology ............................................................................................................................................................................ 24
Data Collection ................................................................................................................................................................... 24
Data Selection ..................................................................................................................................................................... 25
Codebook ............................................................................................................................................................................ 27
Annotation .......................................................................................................................................................................... 31
Data Analysis ...................................................................................................................................................................... 32
Results ..................................................................................................................................................................................... 32
Common Mistakes in Visual Accessibility of Graph Design for People with Low Vision (SQ1) ...................................... 33
Distribution of Accessibility Issues and Their Correlation with Graph Type (SQ2) ........................................................... 37
Issues Outside the Annotation Scheme (SQ3) .................................................................................................................... 38
Extent of Increased Textual Accessibility Used to Improve Understanding (SQ4) ............................................................ 40
Observations Beyond the Sub-Questions ............................................................................................................................ 41
Discussion ................................................................................................................................................................................ 41
Limitations .......................................................................................................................................................................... 45
Recommendations ............................................................................................................................................................... 46
Practical implications ..................................................................................................................................................... 46
Further research .............................................................................................................................................................. 46
Conclusion ............................................................................................................................................................................... 48
Reference List .......................................................................................................................................................................... 50
Appendix - Codebook .............................................................................................................................................................. 59
4
Scientific Practice Regarding Visual Accessibility of Research Papers: Pitfalls and
Recommendations
The topic of visual accessibility is relevant not only because of the substantial number of people
affected (Ackland et al., 2017) but also because societal awareness about disability is increasing,
causing the provision of inclusion to become a requirement (Kletenik & Adler, 2022). Enabling
accessibility is crucial to make society inclusive and to facilitate participation for everybody and is an
essential step towards equity (Shaheen & Lohnes Watulak, 2019). Accessibility is commonly
interpreted as providing access through supportive (assistive) technology to give disabled users
autonomy, although such improvements can benefit everyone, both disabled as well as non-disabled
people (Henry et al., 2014). In 2015, the International Agency for the Prevention of Blindness estimated
a prevalence of about 253 million people worldwide with mild to severe visual impairments, and 36
million blind people (Ackland et al., 2017). This number is expected to increase due to a growing and
ageing society (WHO, 2011).
Despite many people trying to provide accessibility (Brown et al., 2018), it is not always clear
how to accomplish this and according to which standards (Diamond, 2020). As visual impairments are
increasingly prevalent in today's society (WHO, 2011), researching how to make knowledge accessible
is becoming relevant and is taken into account more often when producing research papers. Therefore,
a case study on accessibility of data visualisations in scientific texts, using the 2020 Human-Computer
Interaction (CHI) conference of the Association for Computing Machinery (ACM) proceedings (ACM
SIGCHI, 2020b) will provide insight into the extent to which visually impaired readers can access visual
and textual information provided by graphs in the articles.
Low vision will be used exemplary for visual accessibility. Low vision, as summarised by the
National Eye Institute of the United States of America (2020), describes visual problems which hinder
the execution of daily activities, such as reading and seeing one's computer. Low vision makes it more
imprecise and straining to take in visual information, potentially resulting in a lack of information or
5
errors in interpretation. Instead, documents can become visually accessible, for example, through texts
printed in a tangible font (Braille, the tactile embossed point writing system) or screen readers, which
read the digital text aloud (American Optometric Association, 2020). Primarily technical documents
often contain data visualisations that must be verbalised auditorily or made visually distinct through
image descriptions and colour contrast to be functional for users with reduced vision (W3C, 2018).
Accessibility Regulations
Aside from a general humanitarian desire for fair access, there are also legal grounds for
accessibility. Internationally, the United Nations’ “Convention on the Rights of Persons with
Disabilities (CRPD)” states that disabled people should not be excluded from education and that they
should receive necessary accommodations for their education (United Nations, 2016). This view is also
reflected in the 4th Sustainable Development Goal “ensur[ing] inclusive and equitable quality education
and promote lifelong learning opportunities for all” (United Nations Department of Economic and
Social Affairs, 2015). Despite this, thus far, the laws on accessibility differ per country, and the
enforcement of such laws is unequal (Web Accessibility Initiative (WAI), 2018).
The main accessibility guidelines (more in the Theoretical Framework) are the Web Content
Accessibility Guidelines (WCAG 2.1.; W3C, 2018) which provide success criteria that distinguish
between sufficient and different levels of insufficient accessibility, through which satisfaction of the
guidelines can be established. The WCAG specifically is considered a worldwide standard for
guidelines. As the criteria are exhaustive, many publications or conferences require some guidelines of
the WCAG, which is also applicable to the Human-Computer Interaction Conference, CHI 2020
conference. The variability of such national accessibility standards could lead to problems in the
interdisciplinary, global scientific community, highlighting the misalignment of accessibility
expectations, standards, and laws (Campoverde-Molina et al., 2021). While scientists might adjust the
accessibility of their documents based on the journal they are submitting to, different standards might
influence collaborations. Since people follow different accessibility standards, variations in their
6
behaviour are expected. The Theoretical Framework will discuss which standards exist, how they differ,
and which purpose they serve.
Scope
Accessible visualisations should conform to contemporary accessibility guidelines, but so far,
little is known about the exact definition and prevalence of visual accessibility (Schwabish et al., 2022).
Therefore, determining and quantifying the accessibility of research documents can help to identify
problems and their prevalence, as well as generate awareness. Beyond that, this will provide insight into
the fulfilment of accessibility needs on various levels of visual impairments and the effect of
accessibility guidelines on different types of visuals. To gain understanding to which degree scientific
visuals are visually accessible, the following research question will be answered. “To what extent do
papers from the Human-Computer Interaction Conference of 2020 follow best practices regarding
visual accessibility for low-vision users in designing comparative quantitative charts (like bar and
line charts) in research papers?”
As mentioned before, the CHI 2020 requires adherence to some of the WCAG 2.1 standards
which results in the researchers publishing with CHI having above average awareness of and interest in
accessibility (Bernhaupt et al., 2020; CHI 2020, 2020), making this conference a suitable focus for the
present research. Further, best practices can be established according to these standards in order to make
documents accessible, such as choosing colours according to sufficient contrast.
To gain an in-depth understanding, the scope was limited and exemplified scientific visuals by
focusing on distinct types of graphs commonly used in the scientific community, namely bar and line
charts. The graphs themselves, as well as accompanying alternative text (Alt-text) and image
descriptions, will be assessed on their visual as well as textual accessibility quality. Methodologically,
a quantitative content analysis will be applied, for which initially, the dataset will be qualitatively
evaluated based on accessibility standards and consequently, quantitatively interpreted and analysed
according to their effects and in connection to the present literature.
7
Societal & Scientific Relevance
The societal merit is the establishment of intermediate-level knowledge on which accessibility
guidelines benefit low-vision graph accessibility and determining current scientific practices in visual
accessibility. An understanding of the quality dimensions of effective visualisation of quantitative data
through visuals and text by professionals will be produced. When adhered to, the accessibility
guidelines should solve all accessibility issues a person with low vision could encounter when
inspecting the graph of a scientific paper (Allan et al., 2016). From the perspective of equal access, it
should be a focal point of science to facilitate and improve accessibility. Discrimination against the
visually impaired arises when they cannot receive and evaluate information about themselves, while
reversely, accessible accommodations also provide benefits for non-impaired people. Whenever a
scientific paper is not accessible, visually impaired people depend on someone else or may have to skip
relevant data. Such restrictions hinder the usage of scientific discoveries and inhibit academic progress.
Contrary, improved visual distinguishability can also benefit non-impaired users of bad-quality copies.
At the same time, alternative text can help understand a website when the internet is too slow to load
graphics. These insights can be transferred to everyday visuals like newspapers or advertisements.
Anything that helps to make the world more inclusive and fairer is worth studying, especially when the
adjustments are quick and easy.
Theoretical Framework
In order to evaluate current practices for visual accessibility, the analysis will specifically focus
on accessibility guidelines which affect people with low vision, who might still be able to use some
navigation, e.g., by mouse (which blind people cannot), while also requiring a screen reader for other
tasks. Such a unique overlap and degree of abilities highlight the necessity for smooth transitions of
accessibility needs. Beyond including people with permanent impairments, like people with low vision,
findings and improvements in this area could also become more standardised to seamlessly support
temporary or situational impairments of non-disabled users.
8
Key Terms
Low Vision
Low vision, the focus of this paper, is a visual impediment that cannot be helped with glasses,
lenses, or surgery (American Optometric Association, 2020). This impairment is caused by different
conditions resulting in specific vision losses, such as central or peripheral vision loss, each causing a
person to lose a distinct part of their vision. Other conditions are night blindness, which results in the
loss of vision in dim light or darkness, or a blurry or hazy vision, interfering with the clarity with which
someone can see (National Eye Institute, 2020).
Aids for Visual Impairments
Many visually impaired people use screen readers to perceive the content displayed on their
devices, such as computers or phones (American Optometric Association, 2020). Screen readers read
out what is written on the screen and can be navigated by the user through key commands such as the
tab or arrow keys since visually impaired people can often not use touch screens or computer mouses
anymore. While it is pretty ‘easy’ for the screen reader to translate written text into speech, the program
cannot describe the content of images and thereby provide the user with the relevant information (Lazar
et al., 2007). Therefor, screen readers use so-called “alternative text descriptions” (Alt–Text) which
content creators might provide to describe the images used. Initially and primarily, the description stems
from HTML (hypertext markup language), which describes images not shown due to browser problems.
Nowadays alternative text is employed to describe images for visually impaired persons.
Similarly, to Alt-Text, complex images, such as very detailed graphs, require so-called long
descriptions to describe relative sizes of columns or chart structures to visually impaired users. This
description reduces complexity, by making the relevant information, for example, the graph’s trend,
stand out and helps readers follow along (Web Accessibility Initiative (WAI), 2022). These textual
improvements also benefit blind people, while improved visuals can aid people with mild visual
impairments. Generally, but not exclusively, people with visual impairments use magnification software
9
to enlarge text and visuals. Additionally, they often use custom settings for font, colour, and word/line
spacing and listen to audio descriptions in videos.
Apart from screen readers, visually impaired people often use refreshable Braille displays that
tangibly present the text in Braille (MKB Toegankelijk, 2018). These accessibility features should be
incorporated in documents in advance, or if possible, could be adjusted to each individual’s needs by
enabling later manipulation of, e.g., colour contrast, spacing, font size or colour relevance.
Unfortunately, despite the prevalence of visual impairments, many (research) documents do not yet
provide such features (Masum Billah et al., n.d.). This lack of accessibility can partially be attributed to
missing awareness and expertise to create accessible documents, as well as potentially unclear
accessibility laws and standards, which differ between countries (Shinohara et al., 2018).
Affected Individuals and Variations
Naturally, visually impaired academics and students in the audience of the CHI 2020
Conference are also affected by these issues, as they have to compensate for inaccessible documents
(Gerber, 2003). For instance, approximately 3.4% of students in the US are visually impaired and 4%
of US - academics live with a disability (Brown et al., 2018; Diamond, 2020; Statista, 2020). Changing
accessibility requirements in favour of visually impaired scholars would also directly influence the work
(load) of any non-impaired scholars, conference organisers and visitors and journal publishers.
Moreover, there are other types of impairments which can hinder visual interpretation. The WHO
estimates that worldwide over 2.2 billion people have near or distance vision impairments (WHO, n.d.),
which can cause accessibility problems like not being able to read information. Other impairments that
might cause visual problems include colour blindness, dyslexia, intellectual or processing disorders and
low literacy (European Dyslexia Association, 2020; Roser & Ortiz-Ospina, 2018). Therefore, more than
a quarter of the global population would benefit from improved visual accessibility.
Visual problems can arise in different circumstances (Microsoft Design, 2000). Most apparent
are permanent visual issues present since birth, acquired through illness or accident or connected to
ageing. These can be divided into so-called “Legal Blindness” referring to a significant, irreparable and
10
legally distinguished reduction of the visual field or blurriness, and “Totally Blind” people who can
perceive even less visual information (WebMD, 2022). People with low vision may experience “legal
blindness” and often cannot view and read documents at an average distance, even with aids like glasses
or lenses (Cupples et al., 2012). However, they can compensate for part of their disability with
environmental modifications like computers with bigger fonts or higher contrast (American Optometric
Association, 2020).
In contrast to this, there can be temporary or situational visual impediment. In the short term, a
concussion, for example, can lead to temporary impairments, potentially causing the person to have
blurry or double vision, be sensitive to (display-) light or lose part of their field of view. On a nearly
daily basis, someone can also be situationally impaired when, e.g., blinded by (sun-) light, have trouble
focusing their eyes because of tiredness, or motion can seem to freeze or skip in strobe light (American
Optometric Association, 2020). All these situations result in deviations in one’s vision compared to an
average visual ability. However, unlike a person situationally or temporarily missing visual information,
a visually disabled person cannot ‘just’ replay a scene they missed in a movie when the situation is
better and then understand the full context (Microsoft Design, 2000). Permanently disabled people
require long-term solutions and dependable accessibility provisions.
Literature on Accessibility
The topic of accessibility has been studied from a legal perspective (Wu et al., 2021), for
example about requirements for accessibility guidelines, as well as an urban design angle (Jensen et al.,
2002) to foster participation. Yet, few academic papers have been published regarding the (visual)
accessibility of scientific graphs (Schwabish et al., 2022). In order to achieve visual accessibility, many
papers rely on auditory-focused approaches, such as verbalised instructions or sounds. Another research
approach is based on creating accessible pictures, for example, through Alt-text (Craven, 2006) or
enhancing the (screen) readability of documents by studying document structure (De Jong & Schellens,
2000). Building on both types of research, it is needed to investigate (complex) scientific graphs based
on the presence and appropriateness of textual elements like Alt-text or descriptions, as well as
11
usefulness of visual features, such as colour and contrast. Combining these two perspectives is also
relevant to the distinct needs of the low vision users.
Accessibility Guidelines
Accessibility guidelines are not yet as standardised as one might expect, especially not on an
international scale. The first breakthrough concerning accessibility was achieved through the
Rehabilitation Act of 1973 in the United States of America (U.S. Equal Employment Opportunity
Commission, 1973). Over the following decades, the strides towards inclusion and access became more
open to various disabilities, and the influence of laws grew (U.S. Department of Justice Civil Rights
Division, 1990). More recently, in 2008 and the years after, equal access laws were refreshed and began
to include the internet, which keeps requiring global initiatives towards digital accessibility (U.S. Equal
Employment Opportunity Commission (EEOC), 2008). Currently, the accessibility guidelines are
generally organised per country or slightly larger groups and are often still subject to every user's
interpretation and skill level (Ronan, 2021).
However, many countries do agree on a large set of guidelines by the World Wide Web
Consortium (W3C) called the Web Content Accessibility Guidelines (WCAG), currently used in the
2018 version 2.1 (W3C, 2018). WCAG 2.1 follows the principles that documents should be perceivable,
operable, understandable, and robust. This means that features which are presented can be processed
and handled by the user, are formulated or designed clear enough to be understood and are still logical
in case of errors. Although WCAG is nowadays used worldwide, this was not initially the case. In 2015,
a digital accessibility concept was also put forth by the European Union (EU) (European Commission,
2019). Three years later, the WCAG standard is not only a recommendation anymore, but was also
incorporated into an existing European Digital Accessibility Standard (European Commission, 2015).
Following this, countries like Australia also adopted such standards (Australian Government, 2022). As
of September 2020, EU websites must adhere to the “Accessibility requirements for ICT products and
services” (European Telecommunications Standards Institute (ETSI), 2019).
12
In order to enable people with disabilities to access products and services in the same way as
their peers, most countries have implemented equality or discrimination acts (Bunbury, 2019). While
different countries implement different strategies to accomplish such goals, it seems that by now, many
countries have adjusted their laws accordingly With the USA and especially the EU adhering to very
similar rules, other countries will likely catch up to that standard in order to remain competitive, as
depicted by the so-called “Brussels effect” (Bradford, 2012).
Recent publications generally show that many universities or research events, although mostly
mandatory, did not fulfil accessibility guidelines in recent years. Saltes (2020) describes that despite
legal regulations, accommodations for disabled employees could not be sufficiently provided at half of
the Canadian universities. The author found that this is usually due to missing consistency when
implementing policies or due to language and content struggles. Saltes also determined that the current
accessibility policies perpetuate differences between disabled and non-disabled people and urges
universities to create inclusive workplaces (Saltes, 2020). A study conducted by Acosta-Vargas et al.
(2019) assessed Latin American university websites and frequently visited websites with WCAG 2.1
standards and found that none of these pages, despite their regular usage, were accessible. In their
research, they asked legislators to improve accessibility regulations in order to elicit significant
improvement of websites and mobile application accessibility. The authors also advise that best
practices of the WCAG should be used to increase the level of accessibility in a consistent manner
worldwide. Beyond making such guidelines consistent, they also highlight that awareness of the topic
is lacking and that programmers need to become more trained to successfully implement accessibility
features in their work (Acosta-Vargas et al., 2019).
Application of Guidelines
Even though events, such as the CHI conference (ACM SIGCHI, 2020a), incorporate
accessibility standards, this might not be up to date with the newest version, or more lenient in
interpretation. In the CHI 2020 edition, the WCAG 2.0 was suggested to authors, rather than version
2.1, which had already been introduced by then. Additionally, to the excerpt of WCAG guidelines, the
13
CHI 2020 Guidelines recommend the use of the “accessibility check in Word". However, due to the
lack of concreteness of this rule and the varying precision of the Word tool, this recommendation does
not provide clear or exhaustive guidance. Based on different regulations and strictness, the current
guidelines can still result in more or less accessible documents.
Examples of Guidelines
To put the guidelines in context, some standards from specifically the CHI conference will be
compared to the worldwide applicable WCAG guidelines and how they are represented in the European
guidelines. The three example elements for visual accessibility are Alt-text, Redundancy Gain and
Colour Contrast. As described before, Alt-Text, is a common tool in web design, but also highly relevant
to accessibility. So much so that it became the first WCAG guideline (Web Accessibility Initiative
(WAI), 2019). A reader should always be able to understand non-text input such as graphics or digital
buttons like the search function, via, for example, speech output, or a braille display.
On a more visual level, any design should be lenient with a user's perception, as not to
overwhelm the user. Wickens et al. (2004) created an overview of 13 design principles, which support
the user's perception, attention, memory and mental model when conducting a task. While several of
these heuristic principles aid accessibility, particularly Redundancy Gain is relevant to graphs and is
represented in the various standards described above. Redundancy Gain requires information to be
provided in multiple forms, rather than relying on just one method (Wickens et al., 2004). This could
be a combination of colour as well as positioning, as done for traffic lights. It is clear that the light at
the top is red and means stop and that the green light at the bottom means go. This visualisation allows
traffic to function regardless of someone’s colour perception (Wickens et al., 2004). Similarly, graphs
benefit from providing information in more than one way, rather than relying on colour. Lastly, colour
can be an accessibility problem itself in terms of colour contrast, as the contrast between various colours
in a graph or between the line and the background can heavily impact the accessibility (Alcaraz Martínez
et al., 2022). Especially people with low vision might not be able to visually compensate for a lack of
contrast, and therefore miss out on information. Colour contrast is relevant to make text or visuals
14
distinguishable and increase perception (National Eye Institute, 2020). Table 1 shows these visual
accessibility elements and how they are represented in each of the accessibility guidelines.
Table 1
Comparison of elements for visual accessibility in different accessibility guidelines.
CHI 2020 Guidelines2
WCAG 2.13
Accessibility
requirements for ICT
products and services
(European Norm
301 549 V3.1.14)
Alt-Text
“All visual content
(figures, charts, etc.)
should have alternative
text that describes the
content. Typically, this
alternative text should
not just be a repetition
of the caption but
should provide
additional details so
that the figure is
understandable by
someone who cannot
see the images.”
1.1 “All non-text
content that is
presented to the user
has a text alternative
that serves the
equivalent purpose
[...]”
4.2.2 (3) “Users with
limited vision may
also benefit from non-
visual access (see
clause 4.2.1).”
Redundancy Gain
“All visual content
should use more than
just colo[u]r to convey
differences (e.g.,
patterns in bar charts,
solid vs. dashed lines
in graphs).”
1.4.1 “Colo[u]r is not
used as the only visual
means of conveying
information, indicating
an action, prompting a
response, or
distinguishing a visual
element.”
4.2.3 “Where
significant features of
the user interface are
colour-coded, the
provision of additional
methods of
distinguishing between
the features may
contribute towards
meeting this clause.“
Colour Contrast
Not explicitly
mentioned.
1.4.3 “The visual
presentation of text
and images of text has
4.22 (2) “Where
significant features of
the user interface are
2
(CHI 2020, 2020)
3
(W3C, 2018)
4
(European Telecommunications Standards Institute (ETSI), 2019)
15
a contrast ratio of at
least 4.5:1, except for
the following:
Large Text
Large-scale text and
images of large-scale
text have a contrast
ratio of at least 3:1.”
dependent on depth
perception, the
provision of additional
methods of
distinguishing between
the features may
contribute towards
meeting this clause.”
Applicability of Guidelines to Visuals
Beyond the different guidelines and elements for visual accessibility, also the visual type
influences the relevant accessibility measures. Currently, research about the accessibility of scientific
graphs, beyond alternative text, is scarce. Nonetheless, this is crucial as graphs contain a multitude of
information, such as all variables, main events and trends of data (Anscombe, 1973), summarised in a
compact visualisation, which is often not repeated in the text due to perceived redundancy (Dawson,
1992). Yet, outsourcing a significant amount of crucial information into a potentially inaccessible image
increases the exclusion of people with low vision.
Abela (2020) created a comprehensive classification of chart types, grouping them according
to their visualisation of comparisons, relationships, distributions, or compositions, as well as how the
data behaves over time or the amount of variables that are displayed in a graph. According to this
classification, the present dataset will be structured based on the visual aspects of the data. One of
Abelas’s (2020) categories were comparative charts which aim to visualise data by depicting graphs
combined with numbers, such as bar and line charts. These two graph types are commonly used in
research to compare findings of one or several variables or data changes over time, enabling a deeper
understanding of the domain (Abela, 2020). As these charts are ubiquitous, focusing on them might be
particularly profitable for both authors and readers. In contrast, other kinds of data visualisations, such
as tables, stacked column charts, pie charts, box plots, scatterplots, and heat tracking graphs, will be
16
removed because their accessibility needs are transferable from the included graphs and would not
significantly add to insights about accessibility of graphs (Goldsmith & Davenport, 1990).
Consequently, research visuals that should follow accessibility guidelines for pictures, for
example, mock-ups of a research setting (e.g., Table 2A) or other research materials, such as screenshots
of chats (e.g., Table 2B), will be excluded from the sample. The same applies to models explaining the
variables of a research (e.g., Table 2C) or stylisations of charts without actual result numbers (e.g.,
Table 2D) as they provide minimal information about the accessibility needs of elaborate graphs.
Additionally, population pyramids and the norm lines in graphs will not be considered as the pyramids
visually largely align with bar charts and norm lines will be excluded because they do not hold the same
numerical information as line graphs provide. In cases where visuals have more than one type of graph,
the part that matches the included type of comparison charts established above, will be considered
(Goldsmith & Davenport, 1990).
Both included graph types benefit from and make it possible to design them according to the
accessibility guidelines described above (Anscombe, 1973). As graphs are commonly used within the
scientific community, it is reasonable to assess how the CHI conference guidelines as well as
international accessibility standards work on that visualisation type in practice.
Table 2
Visual examples of graphs that will be excluded from the dataset.
A) Mock-ups of research settings.
B) Research materials.
Figure from (C. Wang, et al., 2020)
Gaiters: Exploring Skin Stretch Feedback on the Legs
for Enhancing Virtual Reality Experiences.
Figure from (Xiao et al., 2020)
If I Hear You Correctly: Building and
Evaluating Interview Chatbots with Active
Listening Skills.
ID: 0010-000
ID: 0003-000
17
C) Models of research variables.
D) Chart stylisations.
Figure from (J. Wang, et al., 2020)
Alexa as Coach: Leveraging Smart Speakers to Build
Social Agents that Reduce Public Speaking Anxiety.
Figure from (Henrikson et al., 2020)
Head-Coupled Kinematic Template
Matching: A Prediction Model for Ray
Pointing in VR.
ID: 0002-002
ID: 730-003
Accessibility Mistakes in Graph Types
“What are common mistakes in visual accessibility of comparative quantitative graph design
for people with low vision, and how frequent are they?” (SQ1) - In order to establish whether authors
at CHI 2020 adhered to the accessibility guidelines, it is necessary to assess potential accessibility
mistakes as well and determine which affect people with low vision and at which frequency do they
occur. Mistakes are deviations from the accessibility guidelines that can result in accessibility problems
for visually impaired users and have been assessed binarily in terms of present or not present. They can
18
arise, among others, in the context of Alt-Text, colour and redundancy gain (see above) (Alcaraz
Martínez et al., 2022; Allen et al., 1992). For example, not providing Alt-Text causes users to be unable
to understand what the graph shows through screen readers. Based on Chintalapati et al. (2022), valid
Alt-Text is expected to be present for around 15% of the graphs, while Brady et al. (2015) found ALT-
Text in up to 25% of their sample.
Regarding colour use, especially missing contrast is a crucial source of mistakes (Muth, 2022),
while another prominent issue arises from the principle of redundancy gain when not adhered to (Allen
et al., 1992). Both problems can commonly be seen in graphs, especially originally coloured line graphs,
printed in black and white, which thereby become indistinguishable due to lack of quality, contrast, and
any additional information (see Figures 1A-D). This issue can be solved, for example, by replacing
solid lines with dotted or striped versions to show different types of variables as well as increasing
colour contrast (Wickens et al., 2004). Research shows that colour is anticipated to have sufficient
contrast in around 30% of the cases (Trewin et al., 2009), while redundancy gain in charts is expected
to be present in about 20% (Durso et al., 2011). Even though the CHI authors are expected to be more
aware of accessibility, it cannot be ruled out that the accessibility of their publications could be
negatively impacted by inaccessible tools which will be illustrated with the hypotheses.
Figure 1A
Lines in a line graph with two (not very
distinguishable) colours.
Figure 1B
Lines in a line graph using two distinguishable
colours, differently patterned lines and marks
the main points of the graph with numbers.
19
Figure 1C
Black and white version of figure 1a, making it
nearly impossible to discern variable A from B.
Figure 1D
Black and white version of figure 1b with
variables still clear to determine based on the
patterned lines and specified points on the axes.
“How are the accessibility issues distributed and what do they correlate with (e.g., graph type,
co-occurrence, …)?” (SQ2) - Aside from identifying which mistakes are common, it is crucial to
establish if any accessibility issues are more common in certain graph types and therefore pose an extra
challenge or if certain errors often occur together. Visually impaired readers cannot necessarily see all
the information conveyed through a graph, for example, size, colour, text. In order to receive the same
amount of input about the content of a graph as anybody else, they might require more complex text
descriptions which basically translate the visual information into descriptive text, additionally to any
interpretation given in the text. This requires the content to serve two types of needs, namely not
disturbing other users while providing more information (Campoverde-Molina et al., 2021). Alt-Text is
a good place to store long, seemingly redundant information, which is usually not shown on screen,
while it still provides all relevant visual information through screen readers (Shinohara et al., 2018). All
these various elements increase the risk for inaccessibility.
Exhaustiveness and Universal Application of Accessibility Guidelines
“Do the graphs also show issues that fell outside the annotation scheme, but that still form a
barrier to interpretation?” (SQ3) - Further, it is important to establish if the existing guidelines and
recommendations are exhaustive enough to assure that when people adhere to them, all articles would
20
be accessible for people with low vision. As people with low vision can make use of visual
enhancements as well as textual alternatives, the amount of relevant guidelines is already extensive
(American Optometric Association, 2020). However, in order to support the processing of visual
information, additional adherence to the design principles (Wickens et al., 2004) mentioned before
could further improve the understanding. According to this theory, the proximity of elements can
support the perception of them belonging together. For viewers with low vision who might struggle to
focus on the data in front of them (National Eye Institute, 2020), this proximity is helpful to aid the
understanding, but a lack of (distinct) proximity could make it more difficult as well. Moreover, a
crucial aspect of visual accessibility and part of WCAG 2.1, which will not be included in the annotation
scope of this paper, is the provision of font size and legibility, without causing a loss of function (W3C,
2018). With regards to font size, especially in graphs, it is relevant that the quality remains
uncompromised, while zooming on elements to be able to read clearly what the graph entails. Often
visuals are inserted and become compressed, thereby becoming pixelated, making it difficult to clearly
make out words. This is particularly hindering for people with low vision(Trewin et al., 2009). These
and other accessibility issues, which are not explicitly examined in this research, but which may arise,
should be taken into account for a comprehensive accessibility strategy.
“To what extent is increased textual accessibility used to improve the understanding of the
graphs?” (SQ4) - Any reader, especially the ones not familiar with the subject, may benefit from
information, which is purposefully provided to support the understanding of the data. Therefore, textual
elements of graphs, such as long descriptions, might universally benefit in understanding the content of
the graph design (Gleason et al., 2020). Universal accessibility also relates to other disabilities or
backgrounds, which means that scientific jargon needs to be explained or abbreviations should be
described close-by (W3C, 2018). While jargon is part of scientific discourse, any clarifications, or ways
to make the main findings stand out more are helpful (Langdon et al., 2014). Consequently, accessible
design is potentially useful for all users. Therefore, it is interesting to see how much the authors of CHI
2020 already provide textual information specifically for graphs, outside the general text, to guide the
reader along.
21
Limited Accessibility
“Data in graphs cannot be accessed completely (100%), because most people opt for
summarised Alt-Text and thereby (accidentally) omit information about the data.” (H1) - A study by
Brady et al. (2015) looked into the accessibility of conference papers through HTML tagging used to
provide structure, e.g., for screen readers, and found that only about a quarter of research papers at the
CHI conferences from 2011-2014 contained tags. Assuming that accessibility and awareness has
increased in the past decade, the percentage likely has increased. Yet, complete accessibility is currently
not realistic, since an omission of data potentially may happen due to a lack of individual experience
with the accessibility issue or unawareness (Jandrey et al., 2021; Kletenik & Adler, 2022). This lack of
awareness may be caused by a cognitive bias known as the “Curse of Knowledge”, where someone
knowledgeable on a particular topic, like a teacher, might not realise another person’s difficulties with
the same topic (Kennedy, 1993). This bias could cause people not to realise that their graphs have an
incomplete Alt-Text if they can visually make up for the lacking text information. As a result, even if
accessibility is present, its scope may be limited. In order to investigate this aspect as comprehensively
as possible, any description possibilities, including figure title, and long description for an image, will
be considered in this paper, in case researchers were trying to provide information but might not have
added the information in the right place.
Achieving Accessibility in Software
“About half of the inspected visualisations (50%), will not have redundancy gain incorporated
in the graph design.” (H2) - Currently, many standard software packages do not add supportive
redundancy (e.g., patterns) or colour contrast features as default. Therefore, the widespread use of
common visualisation software such as Microsoft’s Excel, as well as statistical programmes like SPSS
and R Studio and programming software like Python lead to inaccessible graphs when using default
settings (Tables 3 and 4). Specifically, programmes used by consumers (Table 3) focus on the visual
appeal and ease of use, through which they plausibly, but not rightfully, neglect accessibility needs.
Similarly, scientific software (Table 4) which is used in areas like universities, where a higher (legal)
22
requirement for accessibility is prevalent (Acosta-Vargas et al., 2019), does also not provide accessible
default settings for their, usually educated and professional, users.
In order to understand the status quo of standard settings for various software, several
accessibility features were assessed. To control for colour aspects, the Colour Contrast Analyser (CCA)
from The Paciello Group (TPGi, 2021) was used to assess the colour ratio of the bar/line colours to each
other or the background according to WCAG 2.1. All software (Table 3 + 4), for either of their two
graphs (line and bar chart) do not provide enough colour contrast between their graph colours to make
the individual bars or lines distinct from each other. Neither of the graphs standardly provides texture
to create redundancy gain through, e.g., patterned bars or lines, additionally to colour. Furthermore,
none of the software supports the ease of interpretation through markings and numbering of the data
points on the line or next to the bars. Instead, Python does not even provide any lines in the background
to guide the reader, while R Studio standardly uses a grey background, which impairs the colour contrast
between the bar/line and the background.
When comparing the graph colour with the background, Excel is successful with one of three
variables, while Numbers succeeds only partially with one colour. Google Sheets has two of three
bars/lines with partially sufficient contrast to the background (Table 3). Within scientific software, the
success rates differ vastly. SPSS has enough contrast for two of three colours, and nearly succeeds with
the last one. Contrary, R Studio has bad contrast with all colours and Python only partially works for
two colours (Table 4). Overall, this comparison shows that the standard settings for consumer as well
as scientific software are not accessible, not only in terms of software use, but also regarding its output.
Inaccessible output might be overlooked by sighted creators, due to their bias that everyone has
the same information as them (Kennedy, 1993). Paired with extra steps users need to adjust visualisation
software to be accessible, it can be assumed that most people use the default settings, and consequently
it is likely that the majority of graphs in papers about accessibility research will not be accessible, among
others, in terms of redundancy gain.
Table 3
23
Comparison of accessibility of default settings in common visualisation software.
Programme
Type
Excel (Microsoft5)
Version 2022
(14931.20132)
Default setting
Numbers (Apple6)
Version 11.2 (7032.0,146)
Default setting
Spreadsheets (Google7)
Version April 3, 2022
Default setting
Example
Table 4
Comparison of accessibility of default settings in common visualisation Statistical programmes &
programming software.
Programme
Type
SPSS (IBM8)
Version 27 (64-bit edition)
Default setting
R Studio (PBC9)
RStudio 2022.02.1 Build
461
R version 4.0.3 (2020-10-
10)
Ggplot2 Version 3.3.5
Python (Python Software
Foundation10)
Python version: 3.7.9 64-
bit
Spyder version: 5.2.2
Matplot.lib 3.5.1
NumPy 1.19.3 s
5
https://www.microsoft.com/en-us/microsoft-365/excel
6
https://www.apple.com/numbers/
7
https://www.google.com/sheets/about/
8
https://www.ibm.com/support/pages/spss-statistics-27-now-available
9
https://ggplot2.tidyverse.org/
10
https://matplotlib.org/
24
Example
Methodology
This study uses a quantitative content analysis to study graphs in research papers with regards
to visual accessibility. That means the measured accessibility of the visualisations was quantified and
the effect of presented versus missing accommodations on the accessibility of graphs conveying
research data were interpreted. The data is cross-sectional of the 2020 edition of CHI, which enabled
understanding of the state of visualisation accessibility at this distinct point in time, when accessibility
guidelines were already widespread but not standardised and the guidelines were being reviewed and
improved regularly. The guidelines, which are currently lacking in implementation can be assessed and
provide a basis for future comparisons regarding the normalisation of accessibility.
Data Collection
The data sample consisting of graphs was taken from the research papers of the 2020 Human-
Computer Interaction conference (CHI) of the Association for Computing Machinery (ACM) for
Human Factors in Computing Systems (ACM SIGCHI, 2020b). The dataset was generated from 758
accepted CHI 2020 papers (ACM SIGCHI, 2020a), in which all figures in the papers were extracted via
Python
11
. Originally, 4352 visuals were detected, although not all of them could be downloaded, due to
11
The papers were downloaded on May 20th 2020, and the visuals were extracted on the 5th of October
2020 by Dr. Emiel van Miltenburg.
25
different formatting or broken links. The information of the 3531 remaining items was stored in a
comprehensive spreadsheet. That overview includes tags from the HTML version of the papers, together
with the URL of each image, the figure titles, and descriptions of the visuals like the alternative texts
(Alt-Text), and a long description (see Appendix - Codebook). All papers were freely accessible online.
Authors were not asked for consent, as they voluntarily entered their work in the scientific discourse,
where one can reasonably expect others to build on and criticise earlier publications.
As a highly prestigious conference (Schauerte, 2018), CHI establishes standards other
conferences compare themselves to, as well as focuses on the research regarding usage of computers
by humans, which enhances the relevance of accessibility for the research output. Due to its
multicultural contributors with various backgrounds, analysing papers from this conference provides a
comprehensive overview of standards being applied by people around the world. As the papers stem
from a conference that actively encourages accessibility and provides guidelines as well as various
accommodations (Bernhaupt et al., 2020), the papers can represent forerunners in general accessibility
of research papers. Furthermore, researchers publishing during that conference might be generally more
attuned to accessibility (CHI 2020, 2020), and their graphics are potentially more accessible than the
average sample of research papers.
Data Selection
Figure 2 provides a visualisation of the data selection process. The dataset initially had 3513
visuals, which were reduced by focusing on quantitative diagrams (1170). This choice was made since
scientific papers are more likely to portray complicated statistical, quantitative graphics than casual
visuals such as photographs and rendering (Cleveland, 1984; Franzblau & Chung, 2012). Therefore, to
obtain the most insight into how to make academia in specific accessible, the focus was placed on charts
displaying numbers and connections. Accessibility guidelines for images, e.g., of an experiment set-up,
should be based on general accessibility instructions for visuals. An explanation for and examples of
visuals that have been excluded can be found in the Theoretical Framework. A selection of comparative
charts such as line charts and bar charts that portray data developments among items constitutes the first
26
subset (598). In order to make the analysis more elaborate a smaller, representative amount of charts
was chosen. To enable a comparison between the types of graphs the 400 charts were divided between
200 line charts and 200 bar charts, the research subset. The bar charts included both horizontal (bar
charts) and vertical (column charts) graphs to include insights on accessibility such as the effect of
proximity of bars. The graphs were sampled from various papers in the dataset to avoid unnecessarily
redundant findings, based on the assumption that researchers used consistent levels of accessibility
within the same paper. In total 20 additional line and bar graphs each were reserved for a pilot study to
experiment with the coding scheme. To ensure the validity of the analysis, a diverse sample of the
dataset which encompasses charts from various papers, as well as different types of charts was chosen.
During the coding it became apparent that the extracted long description of a graph did not
match the visual it was connected to. This was resolved by excluding the visual from the dataset, as the
HTML version of the article had been removed on the ACM website (ACM SIGCHI, 2020b) as well
and could not be verified anymore, resulting in 199 graphs of that bar/chart graphs. In order to not
retroactively change the dataset, it was decided not to use a replacement graph, since the expected effect
of one additional graph on the overall findings is minimal with at most 0.5%.
Additionally, line graphs with only one line were marked as not applicable for the assessment
of colour between graph elements (see code 2.1), as there were no visually conflicting lines presented.
Figure 2
Flowchart of the data selection until sample.
27
Codebook
The codebook (simplified structure see Table 5, complete codebook with examples see
Appendix - Codebook) was mainly compiled from a deductive (“top-down-processing”) approach
(Potter & Levine-Donnerstein, 1999). It is based on comparisons of various accessibility guidelines and
literature as shown in the Theoretical Framework. To ensure that the codebook is as complete as
possible it was enriched with an inductive (“bottom-up-processing”) approach (Mayring, 2000) with
noteworthy elements
12
that emerged when sorting the corpus. The elaborate codebook allows for a
content and frequency analysis of specific categories and relationships, such as overlaps as well as
statistical analyses of the codes (Mayring, 2000). The codebook has been objectively established by
computer measurable pass/fails of colour choices and formulas assessing textual inquiries. In order to
establish the applicability of the codebook the pilot test allowed the control of potential errors before
the actual annotation. No errors were identified, but certain parameters for the interpretation of the codes
12
Some figure titles are called e.g., “Figure 1” rather than using a title that provides content information. Line
graphs do not necessarily have accentuated data points marked on the lines, making the results harder to
distinguish for visually impaired people.
28
were narrowed down further. An example of this is the determination that for line graphs with ranges
that are visualised with lighter background colours, the exact background colour underneath/next to the
data line has been assessed. Through these measures, the level of consistency and clarity of the
codebook could be established.
Table 5
The four categories of the codebook: General, Design, Text, Other.
Type of Code
Code Group
(+WCAG 2.1 Section)
Specific Code
(1) General
1. Chart Type
(/)
1.1 Bar Chart Vertical (Column)
1.2 Bar Chart Horizontal
1.3 Line Chart
(2) Design
2. Colour
(1.4.11)
2.1 Non-text Contrast Pass
2.2 Non-text Contrast Fail
3. Colour Contrast
(1.4.3)
3.1.1 Large Text Pass
3.1.2 Large Text Fail
3.2.1 Regular Text Pass
3.2.2 Regular Text Fail
4. Redundancy Gain
(1.4.1)
4.1 Redundancy Gain Pass
4.2 Redundancy Gain Fail
(3) Text
5. Figure Title
5.1 Figure Title Present
5.2 Figure Title Missing
29
(/)
6. Alt-Text
(1.1)
6.1.1 Alt-Text Present
6.1.2 Alt-Text Missing
6.2.1 Phrase Image Description Present
6.2.2 Phrase Image Description Absent
6.3 Length
6.4 Distinct from Figure Title
6.5.1 Content Provided
6.5.2 Content Missing
7. Long Description
(5.2.2)
7.1.1 Long Description Present
7.1.2 Long Description Missing
7.2.1 Title Description
7.2.2 Axes Description
7.2.3 Variable Description
7.2.4 Main Event Description
7.3 Long Description vs Figure Title
(4) Other
8. Other
(/)
8.1 Other
Based on the sorting of the data sample, the “General (1)” category, the code “1. Chart Type“
distinguishes between bar, column, or line charts. This is relevant for inferences about frequencies and
30
co-occurrences of graph types and potential problems. The “Design (2)” category focuses on how much
information people with low vision can still obtain from the graphs by looking at it. While the size of
the graph might also impact the visibility, it was excluded from this analysis since it can often be
mediated through zoom features or magnification tools. The code “2. Colour” is used to evaluate how
well colours in graphs can be discerned from each other. For example, while red and blue and black are
all different colours, those saturated colours might look very similar to people with low vision. This
could prevent readers from being able to identify which piece of data is represented where.
The code “3. Colour Contrast” allows assessment of how well a colour can be seen in front of
a certain background colour. For example, writing with yellow on a white background would be
problematic for most readers, but for people with low vision, many other colour combinations could
cause such a problem. The colours of the design codes will be determined based on the darkest element
of the colour, usually in the middle of a line, rather than the outside parts that “fizzle out”. Therefore,
the colour-picker tools are focused on the middle of a line to avoid fading and colour discrepancies due
to aliasing and which results in the most optimistic contrast assessment (Adobe, 2022). Lastly, “4.
Redundancy Gain” inspects how much information is presented in more than one way, allowing
impairments while still providing equal access. Like a traffic light providing information via colour
(green = go), position can be used to understand that light at the bottom level also means “go”, even if
a person cannot differentiate colours. In a graph this can be achieved by using e.g., distinct positions
with labelling or patterns in a column chart, rather than exclusively using colour to distinguish between
graph parts.
The “Text (3)” category assesses the accessibility of the graphics, especially but not exclusively
for screen reader users. To make documents fully accessible, visuals have to be “translated” to
descriptions of their content, fully encompassing the information non-impaired users could also make
use of. A first starting point to do so is by adding code “5. Figure Title”, in which the purpose or goal
of the figure is explained to all readers, impairment or not. If not even a short sentence about what is
depicted can be found, visually impaired readers might struggle to find the context of the visual.
31
Beyond a very rough introduction of the visual, images should come with alternative texts,
assessed with the code “6. Alt-Text”. The presence of an Alt-Text determines if a visually impaired user
can independently understand what is shown. The alternative text includes a concise, but more extensive
description than the figure title of the main elements shown in the visual (University of Washington,
2022). Key features such as variables, short summaries of the trends and findings should be provided,
and the text should be distinct from the figure title to convey additional information. While HTML does
not restrict the Alt-Text length, various screen readers used to limit reading the text after around 120
characters (University of Washington, 2022). Nowadays, that does not apply anymore, but it is
encouraged to keep Alt-Text concise to avoid overwhelming the listeners with too much detail. In an
accepted Conformance Test for a WCAG 2.0 draft, the cut-off score for English Alt-Text was 100
characters (Ridpath & Andershonis, 2005), which is what this research followed. As screen readers can
pick up when something is labelled as an Alt-Text, if done properly, it is unnecessary to add phrases
like ‘image description’ upfront, as this will become redundant for the user. Words excluded for this
purpose are: Description, Alt-Text, Illustration, Image, Figure, Picture, Graph, Screenshot.
Finally, the code “7. Long Description” inspects any information provided beyond the
alternative description. In case of complex visuals, it might be necessary to explain more of the shown
graphics to enable visually impaired readers to comprehend more details of the data. A long description
should be included when the Alt-Text cannot encompass all the information the author wants to convey.
When it is chosen to provide a long description, features commonly used in graphs should be
communicated. These include but are not limited to, descriptions of the title, axes, and variables used
in the graph as well as trends or peaks in data. Lastly, the category “Other (4)” was reserved for
potential, unexpected discoveries relevant to accessibility throughout the annotation. While a few
general observations will be discussed in the results, no recurring “other” observations have been made.
Annotation
The selected graphs were annotated in a semi-automatic manner based on the codebook. The
visual accessibility was analysed with the WCAG 2.1 compliant Colour Contrast Analyser (CCA) by
32
The Paciello Group (TPGi, 2021). The textual accessibility for example the presence or absence of
figure titles and alternative descriptions was largely established through formulas in a spreadsheet. The
code categories are considered sufficient when the “other” category is not used more than 40 times,
(~10% selected dataset) to deposit ungrouped considerations. To ensure that the code categories are
adequately broad, various (at least 3) sources of accessibility guidelines corroborate the use of each (see
Theoretical Framework). As the codes require objective judgements and many are checked
automatically through spreadsheet formulas and an accessibility tool, the codebook achieves high
reliability without requiring a second coder.
Data Analysis
After the applicability of the codebook was tested through the pilot test and no errors were
identified, the analysis of the sample was conducted. The established codes enable an identification of
common patterns and correlations of (hindered) accessibility of the charts. Statistical analysis was
conducted in SPSS to find correlations or trends in the dataset. For example, to find differences between
the accessibility of various research papers and chart types. Methods of analysis were descriptive
statistics (such as mean, standard deviation, frequency analysis, and co-occurrence of codes), inferential
statistics to identify the relationship between variables.
Results
This section presents the quantitative results and qualitative observations regarding the four
sub-questions introduced in the Theoretical Framework. For this purpose, the effect of the charts on
accessibility and the reader will be examined and any discernible patterns highlighted.
33
Common Mistakes in Visual Accessibility of Graph Design for People with Low Vision
(SQ1)
The frequencies reveal that accessibility rules are very often not followed (see Figure 3). The
findings show a significant lack of usable Alt-Text. In 99.5%, the Alt-Text is identical to the figure
number or title and therefore, does not provide any novel information. In 99% of the cases, the Alt-Text
does not disclose information about the content of the graph it is connected to. This problem becomes
even more noticeable, as the Alt-Text is only provided in 14.4% of the sample, for an accessibility tool
that is considered standard. While this quota is better than none, screen readers are still unable to
describe the image in more than 85% of the cases. One positive example is that only 13.4% of the Alt-
Text contains phrases like “Image Description” or “Alt-Text”, which is repetitive for screen readers,
which automatically announce that they are reading out image descriptions. If this amount can be
transferred to a scenario where ALT-Text is provided 100% of the time, the repetitive phrases would
be present in 1.9% of the graphs, which means this mistake is highlighted by the general absence of
Alt-Text, rather than distinct mistakes in screen reader unfriendly phrases. Favourably, the long
description is present in 90% of the cases and about as often gives insight into the content of the
visualisation. Positively, problems with figure titles only occur in 0.2% of the graphs, indicating that
this category is being satisfied with nearly 100% accuracy. Category 6.3 Alt-Text Length is not
applicable as most Alt-Texts are not present. The average number of characters of Alt-Text, if available,
has an average length of about 9-10 characters, which makes the Alt-Text not very informative.
Surprisingly, it is 89% as likely that the Long Description is provided, even when the Alt-Text
is not, since the Long Description has been provided 90% of the time, and the Figure Title is available
for 99.8% of the graphs. In 20.9%, the Long Description and Figure Title were not unique from each
other and the long description lacked informative content in 14.3% of the graphs, the extents of which
are illustrated in Table 6.
Figure 3
34
Bar chart with percentages of how much accessibility rules were followed, sorted by frequency.
Other prominent issues are design related problems, such as colour contrast in comparison to
the background as well as between graph colours, which are enhanced by a lack of Redundancy Gain.
In 79.9% of the visualisations, the colour contrast between bar chart or line chart lines and the
background was not legible in a regular line size. This is particularly important as visualisations need
to be rich in contrast to be robust when being compressed, which decreases contrast even more, during
publishing or printing. 78.9% of the graphs also did not provide any redundancy gain, potentially
providing more visual input than just colour. This problem is further enhanced, as additionally, 75.6 %
of the graphs did not fulfil the requirement of colour contrast between lines/charts, and thereby did not
allow visually impaired readers to sufficiently visually distinguish between informative elements of the
graph based on colour. In about half the visualisations (57.2%), the colour contrast between bar graphs
or line graph lines and background was clear, as many designs relied on saturated colours, paired with
a white or light blue/grey background. It should be noted that several visualisations nearly reached a
sufficient contrast ratio, and were a few ratio measurements removed (e.g., reaching a ratio of 2.966:1
35
instead of 3:1). Overall, it is crucial to ensure that other visualisations have enough colour contrast
between the graph elements to be able to differentiate them by colour.
Table 6
Examples of gradations on the informativeness of the long descriptions.
Super informative
Relatively
informative
Not informative
Not present
(W. Wang et al., 2020)
ArguLens: Anatomy of
Community Opinions
On Usability Issues
Using Argumentation
Models
ID: 0004-005
(Alaimi et al., 2020)
Pedagogical Agents
for Fostering
Question-Asking
Skills in Children
ID: 0039-006
(Gallo et al., 2020)
RunAhead: Exploring
Head Scanning based
Navigation for
Runners
ID: 0069-005
(Lai et al., 2020)
“Why is ‘Chicago’
deceptive?” Towards
Building Model-
Driven Tutorials for
Humans
ID: 0012-006
“In this figure, we
report the
classification
performance for all the
three tasks addressed
in our study. Task 1
refers to the
classification of
argumentative vs. non-
argumentative quotes.
Task 2 concerns the
classification of
argument components,
i.e. claim, warrant, and
ground. Task 3 refers
to the classification of
support vs. against.
For each task, we
report bar charts to
compare the model
performance for the
classification task in
the six feature settings,
“Number of questions
asked during the pre vs
post-intervention
fluency of question
asking test by
intervention condition.
The bar graph shows
that the fluency of
question asking has
improved post-
intervention,
regardless of
intervention
condition.”
“Bar Graph”
[empty]
36
namely TF-IDF,
LIWC, Politeness,
POS n-gram,
Conversational, and
ALL. Additionally, we
compare the F1 of
Linear SVM and
Naive Bayes
classifiers. The best
configuration for all
three classification
tasks is obtained using
Linear SVM with TF-
IDF features,
achieving an average
F1-score of 77.22%,
61.92%, and 61.42%
for the three tasks
respectively. The TF-
IDF setting always
outperforms the one
including all the
features. The worst
performing setting is
the one relying on
Politeness features
only.”
Moreover, it became apparent that there is a trend to use blue in graphs. Unfortunately, many
backgrounds, if not white, also have a grey or blue colour, which reduces the contrast noticeably (see
Figures 4A, B). Furthermore, any colourful background, even if it is to indicate the range of an amount,
tends to make the contrast for the main data points significantly worse (see Figures 4C, D).
Figure 4A
Adams et al. (2020), ID: 0578-012
Figure 4B
Baughan et al. (2020), ID: 0037-001
37
Figure 4C
Park et al. (2020), ID: 0036-009
Figure 4D
Dehesa et al. (2020), ID: 0685-002
Distribution of Accessibility Issues and Their Correlation with Graph Type (SQ2)
Design accessibility issues seem to be balanced with co-occurrences of graph types. Of the
75.6% colour contrast issues between graph colours (see code 2.1), the majority (40.5%) occurs within
line charts, while column charts make up 31.6%. Colour contrast was inaccessible in 42.8% cases
especially between the background with large areas surrounding the line of the line chart. Here, line
charts contribute 19.7%, while column charts cause 21.1% of the problems. In the 79.9% of colour
contrast difficulties from regular sized graph elements, the line chart causes 39.1% of the graphs’ issues,
while the column chart also contributes 35.8%. Lastly, the Redundancy Gain causes problems within
31.6% of line charts, and up to 41.8% for the column charts. Additionally, it is noteworthy that
accessibility issues of one design category arise more often together, when another design problem has
been identified as well, indicating that design accessibility might often be provided more completely or
not at all.
38
In either type of graph, redundancy gain and Alt-Text were identified rarely (~15%). Yet, the
focus of the accessibility issues seems to differ per type of chart, with text (therefore, screen reader)
problems associated with bar/column charts, while design and contrast difficulties occur with line
graphs more often. Textual accessibility issues seem to arise (1-3 percent points) more often in column
charts over other types of charts. This includes the presence of Alt-Text or long description, as well as
the absence of repetitive phrases for screen readers.
Issues Outside the Annotation Scheme (SQ3)
The annotation scheme coded categories only in terms of presence and absence. Therefore,
issues related to wrong application of elements of the coding scheme fall under the third sub-question.
In cases where Alt-Text was present, it only contained redundant figure numbers, or additional
information like figure title or setting values, but no actual description of graphs.
Furthermore, it was found that while some graphs indicate the usage of redundancy gain in their
legend, the distinction is not really discernible in the visualisation. For example, in Figure 5, even if the
data points per line resemble different symbols, this is only vaguely perceivable when zooming
extensively. This shows that even when following the guidelines and using redundancy, it is crucial to
monitor that the differences can actually be identified, not just when shown in large format, but that the
features remain distinguishable in compressed or small visuals as well.
Figure 5
Difficulties with recognisable redundancy gain in the graph of Liu et al., 2020, ID: 0361-002.
39
Moreover, interpretation could also be hindered by unclear proximity of the columns. For
people with low vision this does not help in seeing which elements are related more strongly to each
other. In Figure 6, the grouping of pairs of columns in the “predict” and “view” categories are not
distinct enough to positively affect accessibility.
Figure 6
Lack of connection between columns of connected variables in Heyer et al.,2020, ID: 0133 – 006
Lastly, even when providing data labels (the percentage or total) for each bar or line in a chart,
trying to keep a good structure (identical alignment) can reduce accessibility. As shown in Figure 7, it
becomes increasingly difficult to determine which column cells belong to the same row, since a lack of
lines, or even proximity, prevents the reader from connecting the visual elements more clearly.
Figure 7
Too much distance between bars and data totals in S. Y. Park et al., 2020, ID: 0043-002
40
Extent of Increased Textual Accessibility Used to Improve Understanding (SQ4)
As virtually no Alt-Texts were useful and figure titles were limited in aiding any in-depth
understanding of the graphs, only Long Descriptions could be assessed in this question. Nearly all
graphs (394; ~90%) in the sample and pre-test were provided with a Long Description. 43% of the
graphs included useful Long Descriptions not only about the content, but also the main events of the
graphs, but often were very technical. The majority of them entailed qualitative descriptions of the
graph's purpose, variables, axes and even main events or graph results. 9% of the graphs included Long
Descriptions which actually guided the reader in what the elements of the graphs were and how they
relate to the purpose of the study, thereby providing significant benefit. Those observations show that
currently, the textual choices, specifically long descriptions, could be successful in making graphs
accessible, if they adhere to the WCAG 2.1 guidelines and are mindfully implemented to support the
understanding of the visually presented research.
However, even a very extensive and complete long description can be very technical, and
therefore, difficult for readers to follow (Table 6 - section “super informative”). Especially only hearing
the information, without having a visual to compare to, can increase the burden on people with low
vision. Overall, textual accessibility occurred only incidentally and sporadic, inconsistent (e.g., not
always with the same author) or with strong variations, so that it was not possible to generate universal
conclusions on the extent with which they fostered understanding.
41
Observations Beyond the Sub-Questions
Some observations did not relate to any of the sub-questions but are still worth presenting. Our
exploratory investigation of a non-representative sample showed a tendency that redundancy gain as
well as long descriptions are more known to people researching in the field. This was determined
because the small subgroup of the dataset, which researches topics related to accessibility, like blindness
and published about them in the same conference, on average scored better on the quality of long
descriptions and provided redundancy gain more often. Surprisingly, no significant difference between
people researching accessibility could be determined with regards to the quality of the Alt-Text. This
indicates that even researchers in the field of accessibility do not apply the standards they conduct
research on. Additionally, different levels of accessibility can be found in various graphs within the
same paper that make use of slightly different colours, thereby causing some graphs to be more
accessible than others (see Figures 8A-C).
Figure 8A
Abdul et al. (2020), ID: 0196-
000
Figure 8B
Abdul et al. (2020), ID: 0196-
001
Figure 8C
Abdul et al. (2020), ID: 0196-
003
Discussion
This thesis investigated the question “To what extent do papers from the Human-Computer
Interaction conference of 2020 follow best practices regarding visual accessibility for low-vision
users in designing comparative quantitative charts (like bar and line charts) in research papers?”
42
The accessibility issues described in the results show the limited extent to which accommodations for
visually impaired people are considered standard nowadays in scientific literature. At most, 15% of the
graphs generally fulfil the accessibility requirements specified in the codebook, which are based on
guidelines from the CHI 2020 conference itself and the World Wide Web Consortium. The main
findings are that Alt-Texts are especially underserved in this conference community, even though their
usage goes beyond accessibility. It was observed that different graph types have distinct weaknesses;
bar/column charts showed more accessibility issues of textual elements, while line graphs had more
design-related problems.
The sub-questions aimed to identify the main mistakes and how issues were distributed.
Regarding common mistakes in visual accessibility of comparative quantitative graph design for people
with low vision (SQ1), a lack of Alt-Texts and absence of redundancy gain occurred in 85% of the
cases. Therefore, only about 15% of the graphs satisfied the requirements in those two areas. Another
significant problem was the overlap between the figure title, Alt-Text or the long description in about
50% of the graphs. This overlap means that in many cases, if any information about the visualisation
was present, it was not presented with much detail and, potentially, was repeated in the wrong places.
The distribution of accessibility issues and their correlation with chart type (SQ2) demonstrate
that more accessibility issues arise in the text elements of bar/column charts, while design issues are
more often found in line graphs. The line graphs were significantly more often than bar graphs
accessible in colour contrast between lines and fore- and backgrounds in big and regular sizes. However,
achieving accessibility in smaller sizes is crucial due to the information density and size of visualisations
in scientific graphs.
Problems with the graphs for low-vision users that are outside the presented guidelines for
accessibility of scientific graphs (SQ3). became apparent as graph size and picture quality were not
suitable to read or zoom. However, scientific literature indicates that users with low vision also require
the ability to adjust the zoom or contrast of the screen to their own needs and have to rely on proper
integration of the visuals into the text flow if they use screen readers (Hattie & Yates, 2013). Elements
43
such as the in-text integration of graphs (the connection of visual and text for the screen reader) were
not within the scope of this analysis, therefore, more platform-specific (like Word, Adobe and other
text processors) accessibility guidelines have not been assessed, and guidelines for such programmes
will require more specifications beyond the information provided in this paper. When focusing solely
on the visualisations, guidelines could be expanded by requesting data tables to support the information
provided in long descriptions, in order to make sense of graphs.
The extent to which improved textual accessibility might be helpful for interpretation of the
graphs (SQ4) could not be determined conclusively. Literature has shown that alternative text is
beneficial when a website is not loading graphics well, as the user can still understand what an image
is about. Also, long descriptions can be practical in explaining the most relevant findings of the research
in a graph by clarifying the relations and how trends have developed, particularly when reading text
from a less familiar discipline (Lundgard & Satyanarayan, 2022; Williams et al., 2022). Therefore, the
well written long descriptions can offer many benefits to non-disabled and disabled users alike. Since
the scientific community frequently generates complex graphs to visualise their research, making it
more standard to explain the relevant points or trends of an extensive graph can significantly increase
the ease when reading and understanding the importance of the findings (Trewin et al., 2009).
Overall, the hypotheses were both formulated with conservative expectations about the
observation of accessibility guidelines. The assumption that “Data in graphs cannot be accessed
completely (100%), because most people opt for summarised Alt-Text and thereby (accidentally) omit
information about the data” (H1) was found to be true. Not only did many Alt-Texts not provide the
full description of what a visual user could see, but many Alt-Texts were not even provided. The
estimation that “About half of the inspected visualisations (50%) will not have redundancy gain
incorporated in the graph design” (H2) was found to be significantly higher than expected. 85% of the
inspected graphs did not incorporate appropriate redundancy gain into the design. Hence, the
expectation that half of the charts provide visual information in more than one way has not been met.
44
The findings of this study are expected to be generalisable with regards to which elements of
accessibility guidelines are easiest or most difficult to incorporate in scientific literature, as it can be
assumed that the required knowledge and effort to incorporate the accessibility guidelines will not differ
substantially between publications. The difficulties established in this research also relate to other
scientific literature, as the graphs discussed here can be found in many publications. Beyond that, it is
unlikely that other graphs would be significantly more visually accessible, especially based on colour
contrast, since similar pre-sets of visualisation programmes that were used for the graphs in this study
would also be applied to other graph types. Yet, it might be possible that especially more complex
graphs could receive more in-depth explanations to guide the reader along, since the authors could
realise a need for explanation for such complex graphs themselves. which might partially benefit people
with low vision.
Nonetheless, awareness about Alt-Text for images is increasing as the practice is becoming
more common, for example, on social media and could translate to the scientific community in the
future, since it is fairly easy, meaning that no specific programmes are required in addition to creating
scientific papers (Baker et al., 2021). Further, providing redundancy gain, data points and colour
contrast for graphs could improve based on pre-sets of visualisation software, once WCAG guidelines
are also applied to them. Therefore, redundancy gain, data points and colour contrast are more likely to
be more common in future publications than shown here. Moreover, the sample researchers of CHI,
who already pay attention to accessibility norms, are likely a positively skewed example on the
incorporation of accessibility features, compared to members of the scientific community that have not
needed to adjust their papers for accessibility. Therefore, it is likely that the general scientific
community publishes less accessible graphs than assessed in this research. This is supported by the
likelihood that when authors of the CHI 2020 conference generally already do not provide the most
standard accessibility features most other articles will currently not have much better accessibility.
The results largely met the expectations, as most scientific graphs assessed in this study did not
meet the established accessibility requirements. Some included accessibility features, but these were
individual elements rather than fully accessible charts in design and text. Most authors did not provide
45
graphs that entirely fulfilled the design or text criteria, let alone both. The findings align with other
scientific papers like Chintalapati et al. (2022), highlighting a lack of sufficient Alt-Text usage. Their
paper showed that outside of a few specific conferences, even with the same publisher, few articles
provide Alt-Text. Even within a more accessibility-focused conference, as is the case in the present
study, not even 15% of the graphs provided alternative text, a bare minimum for visual accessibility.
Our research expands on a lack of Alt-Text but also reports an absence of long descriptions, which
intensifies the accessibility problem.
Furthermore, this research provides insight into difficulties of visual accessibility that are often
overlooked when discussing requirements for screen readers. The difficulties are, namely, visual design
and the need for clarity through colour and contrast, which is very important for many people with
declining vision. While in a different field, Cornish et al. (2015) found that graphic designers on average
considered visual accessibility on 51.5% of their projects. They found that the designers usually did not
have the resources (time or budget) to consider accessibility (testing), unless specifically requested
(Cornish et al., 2015). This experience could be transferred to researchers as well, who are busy with
their tasks and unless required for their submissions, might not (be able to) spend time or effort on
additional accessibility provisions.
Limitations
This research established that there are several visual accessibility issues in the dataset. It was
beyond the scope of the research, and therefore remains unknown, which factors or combinations
thereof cause the visual accessibility issues. Notably, some accessibility guidelines were established
two years before the conference on which this research is based upon. This timeframe could have led
to problems and a lack of experience applying the guidelines by the time the papers were submitted for
the CHI 2020 conference.
46
Recommendations
Practical implications
Practically, this research shows that instructions on accessibility guidelines need to be
communicated more clearly and distinctly for the various elements of research papers, including graphs,
in the future. Additionally, the guidelines are used to help guide the publication process, but
consequences are usually not enforced. This situation shows another problem, namely that it takes time
to make publications accessible, and if guidelines are not enforced, then people might not put in the
effort. Conferences or publishers could include accessibility controls in the “camera ready” stage, when
papers are being prepared anyway to fit their specific format. Moreover, lawmakers, publishers or
conference organisers could increase the requirement to create accessible documents in a multinational
group like the scientific community. Furthermore, this research highlights that if visualisation software
would standardly adjust some of their common features, it could have a big positive impact. A few
suggested changes to standardly correct the colour scheme between colours as well as the background,
to include patterns for lines or bars to make them distinct, and to activate chart elements to include data
labels and tables swiftly comply with WCAG 2.1 standards. A solution would be for accessible features
to require active opting-out could greatly improve accessibility, because people are less likely to put in
effort to remove beneficial features than they are to actively include them (Habib et al., 2020).
Further research
In future research, it would be beneficial to enrich this paper's findings with a qualitative small-
scale user study in which practitioners or visually disabled people provide insights into their actual
needs, especially going beyond the assessed guidelines. Moreover, since even accessibility focused CHI
researchers do not make their work accessible consistently, research is needed to find out what the
barriers are to making scientific work more accessible. Especially with the introduction of updated
WCAG standards, it might be valuable to observe in which manner improvements are adopted by the
scientific community in order to gain insight into the difficulties of the implementation of accessible
methods.
47
Another interesting consideration for future research could be to purposefully, not only explore
on a small-scale what extent researchers publishing about accessibility create more accessible papers
than other researchers (at the same conference). As it is conceivable that people who actively look into
challenges of or solutions for accessibility might be more attuned to accessibility requirements and more
motivated to provide access to their research, it is interesting to see if any effect of their field of research
can be observed. This effect could be increased even more, if the accessibility provisions that are
controlled for (in this case low vision) are about the same type of impairment they research. Due to the
small, non-representative quantity in this dataset, the question could not representatively be answered
in this research. This research focused on academic publications, but it might be interesting to see if
academia is “running behind”, or if other textual documents such as project reports or stakeholder
documents for proposals but also outside of academia are at similar accessibility levels.
Lastly, while this thesis addressed the extent of accessibility issues in scientific papers, the
question of why people do not make things accessible naturally emerges. It is easy to attribute the
problems to unawareness or laziness, but it is also possible that cognitive biases and a lack of
understanding of another person’s lived reality contribute greatly. For example, the knowledge
reporting bias, which means people only describe some parts of their reality and the Curse of Knowledge
(Kennedy, 1993), when people assume that their knowledge is the baseline for others’ knowledge, can
cause a warped perception of which information needs to be included. Especially in regards to
accessibility, this can become a trap if the information is perceived as obvious and unnecessary because
the writer can perceive visual (or other forms of) information that not everyone can use. Unawareness,
paired with a tendency in the scientific community to advise against redundancy and repetition of text
and graphics, can often result in inaccessibility. Further research on how to make accessibility more
appealing and normalised can intersect with psychological biases.
48
Conclusion
The study highlights that visualisations in academic papers of the CHI 2020 conference does
not yet fulfil the international requirements in both textual and design aspects. At most, 15% of the
visualisations were sufficiently accessible in all categories. This shows that even in an educated
environment, efforts to provide accessibility and inclusion still fall short. Based on an exploration of
the standard settings of consumer and scientific graphing software, it becomes clear that, for now,
creating accessibility requires extra effort beyond the topic the article is written about. Additionally, it
is possible that users consider the standard settings as approved and good designs, especially if they
seem visually pleasing to users. This paper contributes to raising awareness, pointing out accessibility
dangers in using default settings and aims to give pointers on which issues have presented as most
crucial and can be meaningfully adjusted by individuals.
A desired impact of this paper is that the results and negative findings about the accessibility
of visualisations resonate with readers, who are willing to go the extra mile and help make their research
more accessible for everyone. The most impactful, least-effort solution determined in this paper is to
provide an alternative text for every visual, providing clear information about what can be seen in the
chart. This description should go beyond saying it is a figure or purely stating it is a chart. A great way
to expand on the chart description is to use the long description option to detail the most crucial elements
of the chart to make the graphs meaningful for people with no or low vision. Aside from the Alt-Text,
researchers are encouraged to make use of redundancy gain by making sure that any element in a graph
can be interpreted in more than one way, e.g. using meaningful positioning, using patterns, or providing
data tables (or a combination of these things). Additional steps are to be mindful of the colour contrast
between the bars or lines between each other as well as the background. These few adjustments can be
included in seconds but might make a world of difference for disabled users.
It is important to remember that it is, unfortunately, way too easy to forget about accessibility
49
difficulties. A major suggestion of this paper to researchers is to put oneself into the audience's shoes
and envision how people with different disabilities could interact with the text. Additionally, many
programmes, such as Word, offer accessibility checks, evaluating whether the text structure is suitable
for screen readers, if tables or figures are well integrated and if images have Alt-Text. Similarly, it could
be useful to install colour contrast checkers as used in this research, to control for visual aspects of
graphs. A quick review of accessibility when preparing for submission can have a significant impact.
Aesthetically pleasing pre-sets of visualisation programmes are not always accessible, and especially
small, compressed graphs are an excellent reminder to provide redundancy gain to ensure information
can be clearly distinguished. As a rule of thumb, it helps to think: If the documents or anything else you
create does not make sense if you restrict your senses in any way, the product might likely not be
accessible to everyone.
50
Reference List
Abdul, A., Von Der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020). COGAM: Measuring and
Moderating Cognitive Load in Machine Learning Model Explanations. Conference on Human
Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376615
Abela, A. V. (2020). Choosing a good chart - The Extreme Presentation(tm) Method.
https://extremepresentation.typepad.com/blog/2006/09/choosing_a_good.html
Ackland, P., Resnikoff, S., & Bourne, R. (2017). World blindness and visual impairment: despite many
successes, the problem is growing. Community Eye Health, 30(100), 71.
/pmc/articles/PMC5820628/
ACM SIGCHI. (2020a). CHI. https://sigchi.org/conferences/conference-history/chi/
ACM SIGCHI. (2020b). CHI 2020 Free Proceedings. https://chi2020.acm.org/chi-2020-free-proceedings/
Acosta-Vargas, P., Antonio Salvador-Ullauri, L., & Lujan-Mora, S. (2019). A Heuristic Method to
Evaluate Web Accessibility for Users with Low Vision. IEEE Access, 7, 125634–125648.
https://doi.org/10.1109/ACCESS.2019.2939068
Adams, A. T., Mandel, I., Shats, A., Robin, A., & Choudhury, T. (2020). PuffPacket: A Platform for
Unobtrusively Tracking the Fine-grained Consumption Patterns of E-cigarette Users. Conference
on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376373
Adobe. (2022). Aliasing & Anti-aliasing. https://helpx.adobe.com/photoshop-elements/key-
concepts/aliasing-anti-aliasing.html
Alaimi, M., Law, E., Pantasdo, K. D., Oudeyer, P. Y., & Sauzeon, H. (2020). Pedagogical Agents for
Fostering Question-Asking Skills in Children. Conference on Human Factors in Computing
Systems - Proceedings. https://doi.org/10.1145/3313831.3376776
Alcaraz Martínez, R., Ribera, M., Roig Marcelino, J., Pascual Almenara, A., & Granollers Saltiveri, T.
(2022). Accessible charts are part of the equation of accessible papers: a heuristic evaluation of the
highest impact LIS journals. Library Hi Tech, 40(3), 594–622. https://doi.org/10.1108/LHT-08-
2020-0188/FULL/XML
51
Allan, J., Kirkpatrick, A., & Henry, S. L. (2016). Accessibility Requirements for People with Low
Vision. W3C. https://www.w3.org/TR/low-vision-needs/
Allen, P. A., Madden, D. J., Groth, K. E., & Crozier, L. C. (1992). Impact of Age, Redundancy, and
Perceptual Noise on Visual Search. Journal of Gerontology, 47(2), P69–P74.
https://doi.org/10.1093/GERONJ/47.2.P69
American Optometric Association. (2020). Low Vision and Vision Rehabilitation.
https://www.aoa.org/healthy-eyes/caring-for-your-eyes/low-vision-and-vision-rehab?sso=y
Anscombe, F. J. (1973). Graphs in statistical analysis. American Statistician, 27(1), 17–21.
https://doi.org/10.1080/00031305.1973.10478966
Australian Government. (2022). Accessibility. https://www.australia.gov.au/accessibility
Baker, M., Nightingale, E. M., & Bills, S. (2021). An Editing Process for Blind or Visually Impaired
Editors. IEEE Transactions on Professional Communication, 64(3), 275–287.
https://doi.org/10.1109/TPC.2021.3090722
Baughan, A., August, T., Yamashita, N., & Reinecke, K. (2020). Keep it Simple: How Visual
Complexity and Preferences Impact Search Efficiency on Websites. Conference on Human Factors
in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376849
Bernhaupt, R., Mueller, F. “Floyd,” Verweij, D., & Andres, J. (2020). CHI ’20: Proceedings of the 2020
CHI Conference on Human Factors in Computing Systems.
https://dl.acm.org/doi/proceedings/10.1145/3313831
Bradford, A. (2012). The Brussels Effect. CLEAN - Soil, Air, Water, 37(1), 15–16.
https://doi.org/10.1002/CLEN.200990001
Brady, E., Zhong, Y., & Bigham, J. P. (2015). Creating Accessible PDFs for Conference Proceedings.
https://doi.org/10.1145/2745555
Brown, N., Thompson, P., & Leigh, J. S. (2018). Making Academia More Accessible. Journal of
Perspectives in Applied Academic Practice, 6(2), 82–90.
https://doi.org/10.14297/JPAAP.V6I2.348
52
Bunbury, S. (2019). Unconscious bias and the medical model: How the social model may hold the key to
transformative thinking about disability discrimination: International Journal of Discrimination and
the Law, 19(1), 26–47. https://doi.org/10.1177/1358229118820742
Campoverde-Molina, M., Luján-Mora, S., & Valverde, L. (2021). Accessibility of university websites
worldwide: a systematic literature review. Universal Access in the Information Society, 22(1),
133–168. https://doi.org/10.1007/S10209-021-00825-Z/TABLES/21
CHI 2020. (2020). Guide to an Accessible Submission - CHI 2020.
https://chi2020.acm.org/authors/papers/guide-to-an-accessible-submission/
Chintalapati, S., Bragg, J., & Wang, L. L. (2022). A Dataset of Alt Texts from HCI Publications:
Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in
Scientific Papers. 11(22). https://doi.org/10.1145/3517428.3544796
Cleveland, W. S. (1984). Graphs in scientific publications. American Statistician, 38(4), 261–269.
https://doi.org/10.1080/00031305.1984.10483223
Cornish, K., Goodman-Deane, J., Ruggeri, K., & John Clarkson, P. (2015). Visual accessibility in
graphic design: A client–designer communication failure. Design Studies, 40, 176–195.
https://doi.org/10.1016/J.DESTUD.2015.07.003
Craven, T. C. (2006). Some features of alt texts associated with images in Web pages. Inf. Res.
Cupples, M. E., Hart, P. M., Johnston, A., & Jackson, A. J. (2012). Improving healthcare access for
people with visual impairment and blindness. In BMJ (Online) (Vol. 344, Issue 7842). BMJ.
https://doi.org/10.1136/bmj.e542
Dawson, J. H. (1992). Avoid Redundancy in Writing. Weed Technology, 6(3), 782–782.
https://doi.org/10.1017/S0890037X0003623X
De Jong, M., & Schellens, P. J. (2000). Toward a document evaluation methodology: What does research
tell us about the validity and reliability of evaluation methods? IEEE Transactions on Professional
Communication, 43(3), 242–260. https://doi.org/10.1109/47.867941
Dehesa, J., Vidler, A., Lutteroth, C., & Padget, J. (2020). Touché: Data-Driven Interactive Sword
Fighting in Virtual Reality. Conference on Human Factors in Computing Systems - Proceedings.
https://doi.org/10.1145/3313831.3376714
53
Diamond. (2020). State of Accessibility Report. https://diamond.la/soar/
Durso, F. T., Pop, V. L., Burnett, J. S., & Stearman, E. J. (2011). Evidence-Based Human Factors
Guidelines for PowerPoint Presentations: Http://Dx.Doi.Org/10.1177/1064804611416583, 19(3),
4–8. https://doi.org/10.1177/1064804611416583
European Commission. (2015). Web Accessibility Directive — Standards and harmonisation. Shaping
Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/policies/web-accessibility-
directive-standards-and-harmonisation
European Commission. (2019). European accessibility act. Employment, Social Affairs & Inclusion.
https://ec.europa.eu/social/main.jsp?catId=1202
European Dyslexia Association. (2020). What is dyslexia. https://eda-info.eu/what-is-dyslexia/
European Telecommunications Standards Institute (ETSI). (2019). Accessibility requirements for ICT
products and services (EN 301 549 - V3.1.1).
Franzblau, L. E., & Chung, K. C. (2012). Graphs, Tables, and Figures in Scientific Publications: The
Good, the Bad, and How Not to Be the Latter. The Journal of Hand Surgery, 37(3), 591–596.
https://doi.org/10.1016/J.JHSA.2011.12.041
Gallo, D., Shreepriya, S., & Willamowski, J. (2020). RunAhead: Exploring Head Scanning based
Navigation for Runners. Conference on Human Factors in Computing Systems - Proceedings.
https://doi.org/10.1145/3313831.3376828
Gerber, E. (2003). The Benefits of and Barriers to Computer use for Individuals who are Visually
Impaired. Https://Doi.Org/10.1177/0145482X0309700905, 97(9), 536–550.
https://doi.org/10.1177/0145482X0309700905
Gleason, C., Pavel, A., McCamey, E., Low, C., Carrington, P., Kitani, K. M., & Bigham, J. P. (2020).
Twitter A11y: A Browser Extension to Make Twitter Images Accessible. Conference on Human
Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376728
Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs.
Habib, H., Pearman, S., Wang, J., Zou, Y., Acquisti, A., Cranor, L. F., Sadeh, N., & Schaub, F. (2020).
“It’s a scavenger hunt”: Usability of Websites’ Opt-Out and Data Deletion Choices. Conference on
Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376511
54
Hattie, J., & Yates, G. (2013). Visible Learning and the Science of How We Learn. https://visible-
learning.org/de/2013/07/visible-learning-and-the-science-of-how-we-learn/
Henrikson, R., Grossman, T., Trowbridge, S., Wigdor, D., & Benko, H. (2020). Head-Coupled Kinematic
Template Matching: A Prediction Model for Ray Pointing in VR. Conference on Human Factors in
Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376489
Henry, S. L., Abou-Zahra, S., & Brewer, J. (2014). The role of accessibility in a universal web. W4A
2014 - 11th Web for All Conference. https://doi.org/10.1145/2596695.2596719
Heyer, J., Raveendranath, N. K., & Reda, K. (2020). Pushing the (Visual) Narrative: The Effects of Prior
Knowledge Elicitation in Provocative Topics. Conference on Human Factors in Computing
Systems - Proceedings. https://doi.org/10.1145/3313831.3376887
Jandrey, A. H., Ruiz, D. D. A., & Silveira, M. S. (2021). Image Descriptions’ Limitations for People with
Visual Impairments: Where Are We and Where Are We Going? ACM International Conference
Proceeding Series. https://doi.org/10.1145/3472301.3484356
Jensen, G., Iwarsson, S., & Ståhl, A. (2002). Theoretical understanding and methodological challenges in
accessibility assessments, focusing the environmental component: An example from travel chains
in urban public bus transport. Disability and Rehabilitation, 24(5), 231–242.
https://doi.org/10.1080/0963828011007022-1
Kennedy, J. (1993). Debiasing Audit Judgment with Accountability: A Framework and Experimental
Results. Journal of Accounting Research, 31(2), 231. https://doi.org/10.2307/2491272
Kletenik, D., & Adler, R. F. (2022). Let’s Play: Increasing Accessibility Awareness and Empathy
Through Games. SIGCSE 2022 - Proceedings of the 53rd ACM Technical Symposium on
Computer Science Education, 1, 182–188. https://doi.org/10.1145/3478431.3499277
Lai, V., Liu, H., & Tan, C. (2020). “Why is ‘Chicago’ deceptive?” Towards Building Model-Driven
Tutorials for Humans. Conference on Human Factors in Computing Systems - Proceedings.
https://doi.org/10.1145/3313831.3376873
Langdon, P. M., Lazar, J., Heylighen, A., & Dong, H. (2014). Inclusive Designing. In Inclusive
Designing. Springer International Publishing. https://doi.org/10.1007/978-3-319-05095-9
55
Lazar, J., Allen, A., Kleinman, J., & Malarkey, C. (2007). What Frustrates Screen Reader Users on the
Web: A Study of 100 Blind Users. Http://Dx.Doi.Org/10.1080/10447310709336964, 22(3), 247–
269. https://doi.org/10.1080/10447310709336964
Liu, W., Gori, J., Rioul, O., Beaudouin-Lafon, M., & Guiard, Y. (2020). How Relevant is Hick’s Law for
HCI? Conference on Human Factors in Computing Systems - Proceedings.
https://doi.org/10.1145/3313831.3376878
Lundgard, A., & Satyanarayan, A. (2022). Accessible Visualization via Natural Language Descriptions:
A Four-Level Model of Semantic Content. IEEE Transactions on Visualization and Computer
Graphics, 1–1. https://doi.org/10.1109/TVCG.2021.3114770
Masum Billah, S., Ashok, V., Porter, D. E., & Ramakrishnan, I. (n.d.). Ubiquitous Accessibility for
People with Visual Impairments: Are We There Yet? Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems. https://doi.org/10.1145/3025453
Mayring, P. (2000). Qualitative Content Analysis. 1(2). http://www.qualitative-
research.net/index.php/fqs/article/view/1089/2386
MD. (2004). File:Anti-aliasing.jpg - Wikimedia Commons.
https://commons.wikimedia.org/wiki/File:Anti-aliasing.jpg
Microsoft Design. (2000). https://www.microsoft.com/design/inclusive/
MKB Toegankelijk. (2018). Hoe bekijken mensen met een beperking je website.
https://www.mkbtoegankelijk.nl/website/hoe-bekijken-mensen-met-een-beperking-je-
website/?utm_medium=email&utm_source=sendinblue&utm_campaign=MKB_3&utm_content=b
utton1
Muth, L. C. (2022). It’s time for a more sophisticated color contrast check for data visualizations.
https://blog.datawrapper.de/color-contrast-check-data-vis-wcag-apca/
National Eye Institute. (2020). Low Vision. https://www.nei.nih.gov/learn-about-eye-health/eye-
conditions-and-diseases/low-vision
Park, K., Kim, D., Heo, S., & Lee, G. (2020). MagTouch: Robust Finger Identification for a Smartwatch
Using a Magnet Ring and a Built-in Magnetometer. Conference on Human Factors in Computing
Systems - Proceedings. https://doi.org/10.1145/3313831.3376234
56
Park, S. Y., Moore, D. J., & Sirkin, D. (2020). What a Driver Wants: User Preferences in Semi-
Autonomous Vehicle Decision-Making. Conference on Human Factors in Computing Systems -
Proceedings. https://doi.org/10.1145/3313831.3376644
Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis.
Journal of Applied Communication Research, 27(3), 258–284.
https://doi.org/10.1080/00909889909365539
Ridpath, C., & Andershonis, J. (2005). Test 3 - Image Alt text is short. HTML Test Suite for WCAG 2.0.
https://www.w3.org/WAI/GL/WCAG20/tests/test3.html
Ronan, R. (2021). A Guide To International Web Accessibility Laws And Policies. Forbes.
https://www.forbes.com/sites/forbesbusinesscouncil/2021/11/19/a-guide-to-international-web-
accessibility-laws-and-policies/?sh=1b1d78936083
Roser, M., & Ortiz-Ospina, E. (2018). Literacy. Our World in Data. https://ourworldindata.org/literacy
Saltes, N. (2020). Disability Barriers in Academia: An Analysis of Disability Accommodation Policies
for Faculty at Canadian Universities. Canadian Journal of Disability Studies, 9(1), 53–90.
https://doi.org/10.15353/CJDS.V9I1.596
Schauerte, B. (2018). Conference Ranks.
http://www.conferenceranks.com/index.html?searchall=Human+Factors+in+Computing+Systems#
data
Schwabish, J., Popkin, S. J., & Feng, A. (2022). Do No Harm Guide: Centering Accessibility in Data
Visualization. https://www.urban.org/research/publication/do-no-harm-guide-centering-
accessibility-data-visualization
Shaheen, N. L., & Lohnes Watulak, S. (2019). Bringing Disability Into the Discussion: Examining
Technology Accessibility as An Equity Concern in the Field of Instructional Technology. Journal
of Research on Technology in Education, 51(2), 187–201.
https://doi.org/10.1080/15391523.2019.1566037
Shinohara, K., Kawas, S., Ko, A. J., & Ladner, R. E. (2018). Who teaches accessibility? A survey of U.S.
computing faculty. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on
Computer Science Education, 2018-January, 197–202. https://doi.org/10.1145/3159450.3159484
57
Statista. (2020). Disabilities among college students U.S. 2019.
https://www.statista.com/statistics/827023/disabilities-among-us-college-students/
TPGi. (2021). Color Contrast Checker. https://www.tpgi.com/color-contrast-checker/
Trewin, S., Laff, M., Hanson, V., & Cavender, A. (2009). Exploring Visual and Motor Accessibility in
Navigating a Virtual World. ACM Transactions on Accessible Computing (TACCESS), 2(2).
https://doi.org/10.1145/1530064.1530069
U.S. Department of Justice Civil Rights Division. (1990). Americans with Disabilities Act of 1990, As
Amended | ADA.gov. https://www.ada.gov/law-and-regs/ada/
U.S. Equal Employment Opportunity Commission. (1973). The Rehabilitation Act of 1973.
https://www.eeoc.gov/statutes/rehabilitation-act-1973
U.S. Equal Employment Opportunity Commission (EEOC). (2008). ADA Amendments Act of 2008. PL
110-325 (S 3406). https://www.eeoc.gov/statutes/ada-amendments-act-2008
United Nations. (2016). Article 24 – Education. Convention on the Rights of Persons with Disabilities
(CRPD). https://www.un.org/development/desa/disabilities/convention-on-the-rights-of-persons-
with-disabilities/article-24-education.html
United Nations Department of Economic and Social Affairs. (2015). Goal 4. Sustainable Development
Goals. https://sdgs.un.org/goals/goal4
University of Washington. (2022). How long can an “alt” attribute be? | DO-IT.
https://www.washington.edu/doit/how-long-can-alt-attribute-be
W3C. (2018). Web Content Accessibility Guidelines (WCAG) 2.1. W3C Recommendation.
https://www.w3.org/TR/WCAG21/
Wang, C., Huang, D. Y., Hsu, S. W., Lin, C. L., Chiu, Y. L., Hou, C. E., & Chen, B. Y. (2020). Gaiters:
Exploring Skin Stretch Feedback on Legs for Enhancing Virtual Reality Experiences. Conference
on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376865
Wang, J., Yang, H., Shao, R., Abdullah, S., & Sundar, S. S. (2020). Alexa as Coach: Leveraging Smart
Speakers to Build Social Agents that Reduce Public Speaking Anxiety. Conference on Human
Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376561
58
Wang, W., Arya, D., Novielli, N., Cheng, J., & Guo, J. L. C. (2020). ArguLens: Anatomy of Community
Opinions on Usability Issues Using Argumentation Models. Conference on Human Factors in
Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376218
Web Accessibility Initiative (WAI). (2018). Web Accessibility Laws & Policies.
https://www.w3.org/WAI/policies/
Web Accessibility Initiative (WAI). (2019). How to Meet WCAG (Quickref Reference).
https://www.w3.org/WAI/WCAG21/quickref/?versions=2.0#text-alternatives
Web Accessibility Initiative (WAI). (2020). Web Content Accessibility Guidelines (WCAG) 2 Level AA
Conformance. https://www.w3.org/WAI/WCAG2AA-Conformance
Web Accessibility Initiative (WAI). (2022). Complex Images. W3C Images Tutorial.
https://www.w3.org/WAI/tutorials/images/complex/#long-descriptions
WebMD. (2022). What Does It Mean to Be Legally Blind? https://www.webmd.com/eye-health/legally-
blind-meaning
WHO. (2011). Blindness and Vision Impairment. https://www.who.int/news-room/fact-
sheets/detail/blindness-and-visual-impairment
Wickens, C. D., Lee, J., & Becker, S. G. (2004). An Introduction to Human Factors Engineering Second
Edition (Pearson ne). Pearson Education Limited.
Williams, C., De Greef, L., Harris, E., Findlater, L., Pavel, A., & Bennett, C. (2022). Toward supporting
quality alt text in computing publications. Proceedings of the 19th International Web for All
Conference, W4A 2022, 12. https://doi.org/10.1145/3493612.3520449
Wu, K., Petersen, E., Ahmad, T., Burlinson, D., Tanis, S., Szafr, D. A., & Szafir, D. A. (2021).
Understanding data accessibility for people with intellectual and developmental disabilities.
Conference on Human Factors in Computing Systems - Proceedings.
https://doi.org/10.1145/3411764.3445743
Xiao, Z., Zhou, M. X., Chen, W., Yang, H., & Chi, C. (2020). If i Hear You Correctly: Building and
Evaluating Interview Chatbots with Active Listening Skills. Conference on Human Factors in
Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376131
59
Appendix - Codebook
Type of Code
Code Group
Specific Code
Description
Example
(1) General
1. Chart Type
1.1 Bar Chart
Vertical
(Column)
The visual displays a
column chart.
1.2 Bar Chart
Horizontal
The visual displays a
bar chart.
1.3 Line Chart
The visual displays a
line chart.
(2) Design
The colour-picker
tools will be
2. Colour
2.1 Non-text
Contrast Pass
The colours of for
example the lines of
the line graph are (or
are not)
distinguishable from
one another.
(According to the
“Non-text contrast for
2.2 Non-text
Contrast Fail
60
focused on the
middle of a line to
avoid fading due
to aliasing.
(MD, 2004)
graphical objects”
(WCAG 2.1 AA
contrast ratio > 3:1))
3. Colour Contrast
3.1.1 Large Text
Pass
The colour contrast
foreground vs
background is in line
with the minimum
accessibility
guidelines.
(WCAG 2.1 1.4.3
Contrast (Minimum)
(AA) Text [...] has a
contrast ratio of at
least 4.5:1 for
"regular" sized text;
at least 3:1 for large
scale text [...].
3.1.2 Large Text
Fail
3.2.1 Regular
Text Pass
3.2.2 Regular
Text Fail
4. Redundancy Gain
4.1 Redundancy
Gain Pass
The information in
the graph can (or
cannot) be
61
4.2 Redundancy
Gain Fail
distinguished through
more than one
method (e.g., patterns
additionally to
colour, etc.).
(3) Text
5. Figure Title
5.1 Figure Title
Present
The figure has a/no
title.
Fig.: A graph showing an
example figure title.
5.2 Figure Title
Missing
[no text]
6. Alt-Text
6.1.1 Alt-Text
Present
The image is
provided with an Alt-
Text of max. 100
characters.
Bar chart comparing
carrots & potatoes as best
vegetables; categories
smell, appearance &
nutrients.
6.1.2 Alt-Text
[no text] or “Figure 1.”
62
Missing
6.2.1 Phrase
Image
Description
Present
When the Alt-Text is
present, it does NOT
contain phrases like
“Image description”,.
Screen readers
announcing that the next
item is an image. If
“Image Description” info
is also added to the Alt-
Text it becomes
redundant.
6.2.2 Phrase
Image
Description
Absent
6.3 Length
The Alt-Text is max.
100 characters long.
A formula in the
spreadsheet indicates the
amount of characters as a
number. 0-100 is fine.
Anything above is not.
6.4 Distinct from
Figure Title
The Alt-Text is
distinct from the
figure title.
If the Alt-Text is only the
same as the figure title it
does not provide
additional information for
visually impaired users
and is redundant.
6.5.1 Content
Provided
The main
finding/result of the
Does the Alt-Text provide
more insight than saying
63
6.5.2 Content
Missing
graphic is (not)
described.
it is a graph? Does it
provide variable names or
a trend of the finding?
7. Long Description
7.1.1 Long
Description
Present
If Alt-Text is longer
than 100 characters, a
long description is
present.
The graph presents a
comparison of carrots and
potatoes to determine the
best vegetable measured
in points from 0-5 in three
categories. With regards
to smell, carrots (4.3) are
preferred over potatoes
(2.4). In appearance
carrots (2.5) underlie the
potatoes (4.4). In the
category nutrients carrots
(3.5) are rated higher than
potatoes (1.8). This means
overall carrots are rated
higher than potatoes.
7.1.2 Long
Description
Missing
[no long description]
7.2.1 Title
Description
The description given
in the long
64
7.2.2 Axes
Description
description matches
what is shown in the
visual and does not
omit relevant
information. (Main
events refer to peaks
or trends of the data.)
The graph presents a
comparison of carrots and
potatoes [Variables] to
determine the best
vegetable [Title]
measured in points from
0-5 in three categories
[Axes].
With regards to smell,
carrots (4.3) are preferred
over potatoes (2.4). In
appearance carrots (2.5)
underlie the potatoes
(4.4). In the category
nutrients carrots (3.5) are
rated higher than potatoes
(1.8). This means overall
carrots are rated higher
than potatoes [Main
Events].
7.2.3 Variable
Description
7.2.4 Main Event
Description
7.3 Long
If the long description is
65
Description vs
Figure Title
the same as the figure title
it does not provide crucial
additional information for
visually impaired users
and is redundant.
(4) Other
8. Other
8.1 Other
Unclear elements to
re-evaluate
individually later on.
[No additional recurring
elements were
discovered.]
