Final evaluation of the accessible content authoring prototypes

Deliverable D2 (Final evaluation of the accessible content authoring prototypes)

Document Technical Details

Document History

Executive Summary

Social networks have permeated every facet of modern society daily life as they allow people to engage with one another. In particular, for people with disabilities, these platforms have, since early on, played an important role in disability advocacy, as they provide a vehicle for meeting new contacts with disabilities, learning about issues and news related to it, and discussing challenges and solutions for improving social media accessibility. Despite the contributions to accessibility, social networks present substantial accessibility barriers for users with disabilities. The complexity of their interfaces compared to many typical websites comes from the fact that they are primarily composed of content created by users themselves. While most major platforms have an interface somewhat accessible, most user-generated content is still inaccessible.

The mechanisms developed in this project support users in creating accessible content, by proposing a new interaction flow for accessible content authoring. SONAAR also retrieves alternative descriptions that had been already provided by other users in order to increase the amount of accessible user-generated content. These descriptions, combined with machine generated ones, are used both to support users in providing better image descriptions, and to deploy accessible content in different websites and applications.

This report describes the activities conducted to validate the new interaction flow for accessible content authoring, and the effectiveness of the documentation for authoring of accessible content, both developed during this project. ​​Preliminary results obtained indicate that SONAAR has the potential to improve social media accessibility. Still, a future evaluation with a larger poll of participants may provide a more robust assessment of SONAAR prototypes. The documentation for authoring accessible content was considered to be complete, clear, and helpful. In addition, participants declared being more motivated to improve the accessibility of their content.

Contents

• Introduction
• Objectives
  • Scope and Goals
  • Problem statement
  • Research questions
• Method
  • Validate the new interaction flow
  • Validate the effectiveness of the documentation
  • Analysis
• Findings
  • Validate the new interaction flow
    • Social media usage
    • Accessible practices
    • Using SONAAR
    • Open answers and interviews
    • Discussion
  • Validate the effectiveness of the documentation
    • Social media usage
    • Accessible practices
    • Documentation for authoring of accessible content
    • Open answers and further comments
    • Discussion
  • Usage log
    • Authoring accessible content with SONAAR
    • Requesting alternatives descriptions for SONAAR
    • Discussion
• Results
  • Research questions
  • Further improvements
• Conclusions
• Annex I: Dissemination of the call for participation
• Annex II: Using SONAAR - Initial survey
• Annex III: Using SONAAR - Final survey
• Annex IV: Getting Started with SONAAR

Introduction

The SONAAR project aims to respond to the accessibility challenge present in current social networks due to the large amount of user-generated content observed in these platforms. That means that, for these platforms to be truly accessible, they have to go beyond ensuring that their interfaces are accessible: they need to ensure that the content their users produce is also accessible. Despite the recent efforts in accessibility features, major social networks still present substantial accessibility barriers for users with disabilities, and little is known about the barriers that mainstream users encounter when creating media content, which is at the root of the issue.

In this context, SONAAR’s overall goal is to prototype a mechanism capable of increasing the amount of accessible user-generated content available on social networks on mobile and desktop platforms. To achieve this goal, the development of our prototypes was guided by the findings obtained in the first user study and reported in previous deliverables (Deliverables D1 and D4) on the barriers and motivations for accessible content authoring. With the information gathered, we established that the new interaction flow supported by SONAAR must guide users into an intuitive and accessible process, followed by support documentation informing users how people with disabilities currently access social media, how they benefit from accessing accessible content, and, finally, how to create accessible media content.

To evaluate if the solution deployed meets our main goal, we conducted two user studies aiming to gather the overall impressions of the SONAAR prototypes and the newly deployed interaction flow for accessible content authoring, and to validate the effectiveness of the documentation provided. Both studies were conducted considering the last version of our prototypes and the documentation provided as reported in previous deliverables (Deliverables D6.2, D7.2, D6.3, and D7.3).

This documentation is structured as follows: Section 2 presents the objectives established such as the research questions addressed in this report. Section 3 outlines the methodology adopted for the user studies conducted. Section 4 details the gathered data and the results obtained. Section 5 analyzes the findings and discusses how they contribute within the context of the project. Section 6 describes our final conclusions and future directions on accessible social media authoring and engagement.

Objectives

Scope and Goals

SONAAR aims to facilitate the user-generation of accessible content on social network services by developing a solution that supports the authoring and consumption of media content on social platforms on both desktop and mobile devices.

This deliverable concerns the first work package (WP1) of the SONAAR project, focused on validating the new interaction flow for accessible content authoring. The activities detailed in this document aim to assess the new interaction flow for accessible content authoring as well as the effectiveness of the documentation for authoring accessible content provided and integrated into our prototypes.

Problem statement

The accessibility of social media is currently addressed by major platforms by embedding an automatically generated description, such as the case in Facebook, or by providing a proper input box for users themselves to provide their own description, as currently employed by Twitter. However, according to our findings reported in previous deliverables, none of these approaches are yet sufficient to meet the desired description quality, in the first case, and quantity, in the second case. According to blind users consuming social images, a better quality of image descriptions is achieved when the description is provided by the content authors themselves. Meanwhile, users sharing images on these platforms are not aware of how and why to provide descriptions for their content. Considering these findings, our prototypes aimed to provide a new interaction flow to better guide them into an accessible content authoring, and support documentation to better inform authors on the practices and benefits of considering accessibility in their social media activities.

Research questions

To achieve the main objectives guiding this user study, we established the following research questions:

• RQ1: Does the perceived quality and quantity of image descriptions increase when using SONAAR suggestions?
• RQ2: Does the quality of image descriptions increase with SONAAR?
• RQ3: Does the content authoring experience of users improve with SONAAR?
• RQ4: Does SONAAR increase the awareness of social media users to the need to create accessible content?

Method

To address the research questions established, two different studies were conducted. The first one aimed to assess the use of SONAAR prototypes and its impacts on the authoring and consumption of accessible content. The second one aimed to assess the effectiveness of the additional documentation provided.

Validate the new interaction flow

A call for participation was disseminated through blind-related communities and through the research team’s social media. The complete list of entities contacted for this study is presented in Annex I. Participants from the first study conducted by the project were also contacted to take part in this new study and, finally, a recruitment message was also integrated into our prototypes to reach possible SONAAR users who learned about our prototypes through other channels.

For this study, participants were asked to respond to an initial survey containing questions about their social media usage and accessible practices. This survey, presented in Annex II, was constructed using Microsoft Forms in Portuguese and English - the two languages currently supported by our prototypes. This survey gathered 9 participants, including 4 sighted users and 5 users self-reporting as having a visual disability. Demographic information about them is presented in Table 1.

Following that, participants were asked to use one or both SONAAR prototypes in their daily routine on social media for two weeks, and, after this period, to complete a final survey on their experience. This survey, presented in Annex III, was also constructed using Microsoft Forms, both in Portuguese and English. From these first nine participants, 3 of them completed the final survey (VIP1, VIP2, VIP5) and two of them, VP1 and VP5, agreed to a more detailed interview.

Validate the effectiveness of the documentation

The second study was designed to have a more robust assessment of the documentation for authoring of accessible content. For that, a second call for participation was disseminated through the research team’s social media. A new survey, presented in Annex IV, was constructed using Microsoft Forms, both in Portuguese and English. Participants were asked to read the documentation provided and respond to specific questions about the usefulness and completeness of the information available. We gathered a total of 24 valid answers and more information about the participants is presented in Table 2.

Analysis

To answer the research questions previously established, the data gathered through the questionnaires and interviews was further explored through a quantitative analysis for the closed questions, and a qualitative analysis for the open questions and interviews. Finally, in order to complement the data gathered through the user studies, we conducted an analysis of the logs registering actions taken by SONAAR users, including ones that have used SONAAR without registering for the study, during the user study period.

Findings

The first study counted a total of 9 participants, with 3 of them concluding the study, i.e. completing our final survey on their experience using SONAAR. The first survey contained questions on their social network usage and accessible practices. We will first present this data, followed by the results obtained from the final survey and the follow-up interviews conducted. Next, we present the data gathered from the second study, focused on assessing the documentation provided. In the sequence, we present the information registered in our database concerning how users are using SONAAR. Finally, a brief discussion is presented on each one of these topics.

Validate the new interaction flow

In this section we present the data obtained from the first study conducted to evaluate the use of SONAAR, considering how it influences users' content authoring experience.

Social media usage

As previously presented in Table 1, most users declared using Instagram, followed by Facebook and Twitter. Two participants also reported using Youtube, and one of them also mentioned using other social networks, such as LinkedIn, Xing, and Slack. Instagram is the most used social network among visually impaired participants, followed by Facebook and Twitter. As for sighted participants, Instagram and Twitter are the most popular social networks, followed by Facebook and Youtube. This information is summarized in Table 3.

Concerning their posting frequency, most visually impaired participants declared posting images several times per month, while sighted participants reported posting images several times per day. This information is summarized in Table 4.

Accessible practices

We also investigated the experience of these participants of authoring accessible content, particularly, of providing alternative descriptions for their image posted on social networks. Among the 9 participants, 2 of them stated not including alternative descriptions in their images posted on social networks. The first one, a sighted participant, for not knowing it was possible, and the second one, a visually impaired user, for now knowing where to write it.

Among those who provide an alternative description for their images, most participants use the text field provided by platforms themselves to write an alternative description, as presented in Table 5.

We also investigated their perceived effort and difficulty in engaging in such activity. First, we asked them how long it takes them to provide an alternative description and whether they considered this value to be very or little time, on a 5-point Likert scale. Most users declared taking very little time, with an average of 1 to 3 minutes as the given value. Two users reported spending some time, one providing 5 minutes as a reference, and the other providing 1 minute as an answer. One participant shared with us having to get help from a sighted family member to make sure of creating a good description, taking her 10 or 15 minutes, which is too much time, in her opinion. This information is summarized in Table 6.

Concerning their perceived difficulty in providing an alternative description, most participants consider this task somewhat easy, as presented in Table 7.

Next, we asked additional questions for visually impaired participants concerning the accessibility of the images present in their social networks streams, collecting 4 answers. The majority stated that almost none of their social network contacts post images containing an alternative description. Following a similar direction, most participants also declared rarely encountering an image containing an alternative description. However, most of them consider descriptions, when present, to be of good quality. The complete information provided by the visually impaired users is available in Table 8.

Finally, participants were also invited to share additional thoughts or comments about the subject that will be analyzed later.

Using SONAAR

To analyze the impact of SONAAR on the accessible authoring and the consumption of images in social networks, participants were asked to use our prototypes for two weeks and complete a final survey afterwards.

Of the initial 9 participants, 3 visually impaired participants completed the study. Among them, two reported using SONAAR for authoring accessible content, even though one of them, only for a test purpose, not publishing any images, as declared in the follow-up interview. Both of them used Facebook, one using the Android application, and the other one using the Chrome extension. We also asked them for how many images they used the descriptions suggested by SONAAR and for how many they modified any of these suggestions. This information is summarized in Table 9. As expected, the first participant stated that he had not posted any images, but that, in his case, he would have modified the descriptions originally provided by SONAAR. The second participant stated having included a description generated by SONAAR for most of the images posted during the study period, and modifying its original content for half of them. The two of them also declared posting about the same amount of images when using SONAAR as they usually do. The level of difficulty of creating descriptions using SONAAR was also a consensus, neither difficult nor easy.

Finally, only one participant declared having encountered any particular difficulties while using SONAAR, concerning the availability of suggestions in Portuguese.

Open answers and interviews

Concerning the answers gathered for the open questions on the initial survey, two sighted participants shared with us some additional comments and thoughts on the subject. The first one mentioned working in a highly known disability-related institution, having extra attention concerning accessibility when posting on professional networks, but forgetting when doing the same for her personal account. The second one shared the challenge of making descriptions engaging, creative, and still conveying their message. As for the final survey, concerning the use of SONAAR, one Portuguese participant declared having difficulties in understanding the suggestions provided in English. Even though SONAAR interfaces support both English and Portuguese, we do not yet have an extensive dataset of descriptions provided by other users. Therefore, the suggestions provided for most images are only the image concepts extracted by Clarifai, available only in English. Another participant also suggested additional support for other browsers, such as Mozilla Firefox and Microsoft Edge - although the latter is currently supported by SONAAR, but the participant was not aware of it.

Two participants agreed to provide more details about their interaction with SONAAR, such as their content authoring experience, thoughts on the current state of social media accessibility and the awareness of accessibility needs, and how SONAAR can be used and improved in this context.

Both interviewees used the Android application, mainly to request descriptions on Facebook. The first participant had no difficulties in using SONAAR, but struggled with the concepts in English, as previously mentioned. The second participant reported not being able to use SONAAR for some time. We identified the issue as being due to a Facebook interface change, one of our current limitations, as mentioned in other deliverables. Concerning the accessibility of our prototypes, both users declared considering it quite accessible, providing further suggestions, such as improving the Android application main screen. Our application runs on the system background, and, after installing it, SONAAR will create a shortcut on the image-sharing system menu and intercept the authoring flow when an image is recognized in it. Participants reported trying to access the application and finding no action available. For that, the SONAAR main screen has to be clearer about its purpose and the way SONAAR is used. This suggestion and other possible improvements will be further explored in the following sections.

Participants also shared their thoughts on image description quality. Both of them agreed on the lack of quality of current descriptions available on Facebook. One participant stated that he is constantly asking his contacts to include a description in their images, being often ignored. According to him, “people just don't get it”. The other participant mentioned that most of the time, he ends up skipping the images he encounters on Facebook, as he already expects a poor description, mostly automatically generated. In his opinion, a good alternative description depends on the context of the image. As an example, a description for an outdoor picture could include what the landscape looks like and how people are dressed, providing more details about the image.

Concerning suggestions for improving SONAAR, one participant highlighted the need of going beyond just embedding an alternative description and providing a brief explanation to authors of how to access these features along with suggested descriptions. Considering that this is currently implemented on SONAAR prototypes, further user trials with sighted people would be useful to assess whether this new interaction flow helps them become aware of accessibility features. Another suggestion provided by the second participant is to expand the current capabilities of SONAAR to support other social networks, such as Instagram.

Finally, we asked participants their thoughts about the documentation for authoring of accessible content provided. The first participant stated that the information is clear and has the potential to help people understand the importance of engaging in accessible practices. The second participant read it after the interview, filling out the corresponding form. His answers will be analyzed together with the other answers in the following sections.

Discussion

In contrast to the first study carried out in the scope of this project, all visually impaired participants declared using Instagram, while Twitter and Facebook were the choices of the majority of sighted participants. However, the same lower rate was obtained when asking about their posting frequency, with most visually impaired declaring posting only some times per month. The most common frequency among sighted participants corresponds to several times per day, in contrast. Concerning engaging in accessible practices, the majority of our participants, including sighted and visually impaired users, stated providing an alternative description for the images they post on social networks, and, most of them, using the proper field provided by platforms. Also, most users consider this task somewhat easy, spending very little time on this. On the other hand, visually impaired users rarely encounter an image containing an alternative description on their social network stream, reinforced by the statement of the majority of them declaring almost none of their contacts adopt accessible practices on their posting and sharing routine. While this information may seem contradictory, it is possible to assume that people willing to take part in a study on accessibility are already aware of the importance and the steps needed to publish accessible content, representing those few who are, indeed, providing the alternative descriptions mentioned by the visually impaired participants. However, it is possible to draw a limit to this engagement, since, of the 9 participants who started the study, only 3 completed it, all visually impaired users. As observed in our first user study and reported in previous deliverables, a great percentage of users are not aware of the possibility to include an alternative description or where to write it. However, those who are aware of it may find this activity too time-consuming, thus reinforcing this limit of time and effort that people are willing to spend on it in general.

This low level of engagement also had an impact on the interaction of the few participants who agreed to share with us their experience using SONAAR. The number of descriptions suggested by SONAAR is directly linked to the number of users creating new descriptions, and, for that, our interviewees, both visually impaired users, most of the time received only suggestions containing the concepts automatically extracted from the image by Clarifai. In general, participants welcomed the whole concept of SONAAR, reporting some minor bugs during the study, and finally provided us with some future improvements, all related to the expansion of SONAAR, both to other platforms and browsers, and also better support for other languages.

Validate the effectiveness of the documentation

In this section, we present the data gathered on the second user study conducted, focused on evaluating the effectiveness of the documentation for authoring accessible content. For this survey, we gathered a total of 24 valid answers, 6 of them from participants self-reporting as having low vision.

Social media usage

As previously presented in Table 2, most visually impaired participants reported using Facebook, followed by Twitter and Instagram. As for sighted participants, Facebook and Instagram were the most used social networks, followed by Twitter. This information is summarized in Table 10.

Concerning their posting frequency, most visually impaired participants declared posting images several times per day or less than once a month, while sighted participants reported posting images several times per month. This information is summarized in Table 11.

Accessible practices

We also investigated the experience of these participants on authoring accessible content, particularly, on providing alternative descriptions for the images posted on social networks. Among the 24 participants, 7 sighted users stated not including alternative descriptions in the images they post on social networks. Three of them reported not knowing it was possible, 2 that they didn’t know how to write a suitable text description, and one stated that it never occurred to her, considering that she doesn’t know anyone, among her contacts, that would need such a description.

Among those who provide an alternative description for their images, most visually impaired users declare writing the description in the text of the post, in a comment or in a reply to the post. Among sighted participants, writing the description in the proper text field provided by the social network itself was the most popular choice. This information is summarized in Table 12.

We also asked participants about their perceived difficulty in providing an alternative description for their images posted on social networks. As presented in Table 13, most participants declared finding this task neither difficult nor easy.

Concerning the importance of providing an alternative description for their images on social networks, most visually impaired users stated they considered it important or slightly important, while most sighted participants considered it very important. This information is summarized in Table 14.

Documentation for authoring of accessible content

The survey dedicated to validating the effectiveness of the documentation contained additional questions about the participants’ opinions on certain aspects of the information available on the SONAAR website. The first question concerned the clearness of the information. For that, most participants considered the information clear or very clear, as presented in Table 15.

When asked if the information provided was complete, most participants stated the information to be complete, as presented in Table 16.

We also asked participants how helpful they consider the information to be. As presented in Table 17, most of them considered the information to be helpful, followed by those who considered it very helpful.

Following, we asked participants if they had learned new information about the accessibility of digital content by reading our documentation, and if so, what did they learn. We gathered a total of 10 valid answers, such as how to use SONAAR or how to describe an image content according to the context to better support visually impaired users. This information will be further analyzed later in this report. Table 18 indicates that half of the sighted users stated learning something from our documentation, as well as the majority of visually impaired users.

Concerning possible engagement and motivation, most participants stated that this documentation is likely to increase users’ motivation to apply accessible practices, as presented in Table 19.

Similar results were obtained concerning their own accessible practices. The majority of sighted participants are likely more intended to be engaged in accessible practices in their daily routine. Among visually impaired participants, most of them reported being neutral or unlikely to be more engaged, as presented in Table 20.

We also asked participants if they had tried the SONAAR prototypes. All 6 visually impaired participants and 4 sighted participants, stated having already used SONAAR, with most of them declaring being likely or extremely likely to continue using it, as presented in Table 21.

For the 10 sighted participants not having used SONAAR, 6 of them declared they will likely start using it, as presented in Table 22.

We also invited participants to share what other information they would have liked to find in our documentation as well as additional thoughts or comments. This information will be further analyzed in the following section.

Open answers and further comments

In addition to the information previously reported, the survey focusing on validating the effectiveness of the documentation also contained open questions concerning new information learned from the documentation, what other information participants would like to see covered, and any other possible comments and suggestions.

Of the 13 participants declaring having learned new information, 10 of them provided more details about it. General information on diversity of capabilities and accessibility of digital content was mentioned by 4 participants. Another topic that emerged was the importance of adopting accessible practices and how to better describe an image for visually impaired people, such as taking the context into account or to avoid starting the description with "image of", for instance. This subject was mentioned by 6 participants. In a broader context, the appropriate use of emoji was also listed as new information by 3 participants. They mentioned learning how emojis are read by assistive technologies and avoiding the overuse of emojis, for instance. Finally, 2 participants mentioned SONAAR, the first one learned how to use it and the second one shared having learned that there is an application to support the authoring of image descriptions to facilitate its access for visually impaired people.

Following this, we asked participants what other information would be also useful to include in our documentation. We gathered a total of 5 answers stating the information available is adequate, one of them mentioned considering the information quite comprehensive. Additionally, 5 answers contained further suggestions. Considering the accessibility context in general, one participant was interested in knowing more also about other types of disabilities. Other participants mentioned that it would be interesting having people with disabilities themselves, through videos or images, describing their personal experiences with image descriptions and the benefits of accessible content on social media. In her opinion, stories have a powerful influence on driving people to spend more time describing an image, as it creates a sense of empathy. One participant was also interested in having more accessibility tips, including do's and don'ts and different examples. Another suggestion was including a more detailed step-by-step process about how to use SONAAR, possibly using images to provide a visual reference. This comment goes along with another participant's answer, who stated not having found a very clear reference on how to use the Android application. On another topic, one participant suggested including some business data, as she uses social media as a business tool, it would be interesting to know about the commercial impact of accessibility. Finally, one participant shared a question about the use of emojis. A possible follow-up to this comment, besides including the requested information, would be the inclusion of a form for people to submit possible questions.

As a follow-up question for the 10 participants stating having used SONAAR, we asked them to describe one or more problems that they may have encountered during this interaction. Two participants declared having issues when using the Android application but no further details on the context of use were provided. Considering both reported this issue on the same day, we considered it to be due to an update on the Facebook interface already identified and integrated on the last version of the Android application.

Finally, participants were invited to share additional thoughts or comments. One participant reinforced a previous comment on the need of providing a more thorough step-by-step on how to use SONAAR, particularly taking into consideration the differences between the many versions of the Android operating system. We also had one comment on how SONAAR may be more useful to commercial social networks accounts than for single users. One participant also stated that he hopes SONAAR can bring convenience to more people.

Discussion

In this study, our participants were mainly Facebook users, followed by Instagram and Twitter users. They also had a lower image posting frequency, with the majority reporting posting images several times per month. Concerning their accessible practices, the majority is also used to provide an alternative description for their images, mainly using the input field provided by platforms. Most of them also perceive this task as neither difficult nor easy, and considered important or very important.

Regarding our main goal, to validate the effectiveness of the documentation, the results obtained indicate that most participants considered it clear, complete, and helpful. They also declared having learned new information concerning how people with disabilities access digital content and how to improve their content to provide better access for all. As a result, most of them declared this documentation is likely to increase their motivation into providing accessible content, as well as for other users.

Participants also shared with us some possible improvements for this documentation, such as including more user stories and examples of accessible practices. While an expansion of this documentation must be carefully thought to not overload users with information, one of the main problems identified in current accessibility documentation, the results obtained in this study demonstrate how people could be more motivated when aware of the possibility of improving the accessibility of their content and the impact it has in other people's lives.

Usage log

To have more information on how people are using SONAAR, as reported in previous deliverables, we logged the following information:

• User identifier: ID number randomly generated attributed to a unique installation of the Android application or the Google Chrome extension. This information is not linked to any further personal information such as name, email, or store account. It is only used to identify actions originating from the same installation.
• Platform: SONAAR Google Chrome extension or SONAAR Android application.
• Social network: Facebook, Twitter, or none - in the case of any website for the Google Chrome extension, or any shareable image for the Android application.
• Request type: Suggestion, request for descriptions in the authoring page on Facebook or Twitter; Authoring, request containing the user authored alternative description; Consumption, request for descriptions outside the authoring context.
• Alternative description contribution: If, in this request, the user had also provided a new alternative description.

During the study period, we gathered a total of 197 requests for an alternative description, including 43 suggestions for content authoring on social media and 154 for the consumption scenario. We had a total of 17 unique IDs, i.e., application or extension users who have submitted at least one request to our backend, with an average of 11 requests per user. In addition, the vast majority of requests were made through the Google Chrome extension, with a total of 174 requests. Finally, Facebook was the social network most used among our participants, with 27 requests against 16 from Twitter.

Authoring accessible content with SONAAR

We had 13 unique users asking for a description for authoring accessible content using our prototypes. We registered a total of 6 Android application users and 7 Google Chrome extension users. The most exploited scenario was the authoring in Twitter using the Google Chrome extension, followed by Twitter users using the Android application. However, from a total of 43 unique requests, the vast majority was made using the extension on Facebook. Also, only two extension users accessed both social networks, Twitter and Facebook. This information is summarized in Table 23.

Requesting alternatives descriptions for SONAAR

Concerning the consumption scenario, we had 8 unique users asking for descriptions on any web page or screen using SONAAR. In this scenario, we had an average of 19 requests per user, the vast majority made through the Google Chrome extension. Even though we had more application users, 93% of the requests were made by extension users, as presented in Table 24. This is not unexpected because of the way the two prototypes operate: the Android service generates one request when the user asks for a description of an image, while the extension generates as many requests as images on the page the user is at when triggering SONAAR.

Discussion

The results obtained from the analysis of the log registering the actions taken by SONAAR users provide us with indications of how SONAAR is being used and possible user preferences. As expected and reported in the deliverables D6.3 and D7.3, our dissemination efforts for the calls for participations also led to new SONAAR users, not directly involved in our previously described user studies. We identified 17 single users on our database, against 9 study participants.

We identified that most people used SONAAR in the consumption scenario, i.e. asked for image descriptions on different websites or applications, including outside the scope of social networks. Taking into account also a higher engagement of users with visual impairments in our studies, this number may indicate that most of these consultations were made by these users. It can thus be an indication that the majority of SONAAR users currently consists of visually impaired users. ​​Another indicator possibly reinforcing the level of reach and engagement of users is reflected in the number of authorships made during this period as no new descriptions were submitted by SONAAR users. It is worth mentioning that one of the study participants reported having posted one image using the suggestion provided by SONAAR, modifying its original content. This information is not logged on our database. Given that the logging feature was deployed to support our user studies, it is possible that the participant had started using SONAAR during this time frame, between the dissemination of the study and the update of our prototypes integrating our logging feature.

Another important information concerns the difference between the number of requests made through the extension and the application, with 88% of the requests originating from the extension. While we had more application users, this difference in requests is expected. As previously mentioned, when the SONAAR extension is activated, the whole page is scanned and for each image available, a new request is sent to the backend. As for the Android application, the user has to share a specific image with SONAAR service.

Results

In this section, we analyze the data collected and previously presented. This analysis will focus on the findings that can help us answer the research questions established for this deliverable and discuss possible improvements to SONAAR prototypes.

Research questions

• RQ1: Does the perceived quality and quantity of image descriptions increase when using SONAAR suggestions?

Visually impaired participants reported rarely encountering images containing descriptions on their social networks. In addition almost none of their contacts are engaging in accessible practices. Considering that only three participants shared their experiences using SONAAR, and especially, as none of them benefited from their network of friends on social networks using SONAAR, it is not possible to further analyze the impact that a massive use of the prototypes has on the accessibility of the content they consume. However, the log analysis conducted indicates a considerable amount of requests on the consumption scenario. For that, we expect that, at least in terms of quantity, SONAAR was able to assist a parcel of users, providing image concepts automatically extracted by Clarifai, even outside of the scope of social networks.

• RQ2: Does the quality of image descriptions increase with SONAAR?

This question was intended to evaluate the quality of image descriptions provided by users with the support of the suggestions provided by SONAAR. For that, we had planned to conduct an expert scoring for the dataset gathered during our study period in order to evaluate if, through time, this quality would increase. However, as previously mentioned, we had a low level of user engagement concerning the inclusion of new descriptions. While it is not possible to obtain a measure as accurate as intended, participants declared having learned how to improve their image descriptions through the documentation provided. In addition, most of them also declared to be more likely engaged in accessible practices. According to our previous findings, reported in D1 and D4, the main reason for users not engaging in accessible practices is the lack of awareness - both of the possibility and of the steps needed to improve the accessibility of their content. Another of our findings is that better quality is achieved when the image description is provided by the authors themselves. For that, we expect that an increased use of SONAAR will also have a considerable impact on the quality of image descriptions provided by its users.

• RQ3: Does the content authoring experience of users improve with SONAAR?

We gathered positive feedback from the participants who used SONAAR during our study period. They reported some minor issues concerning some of the already known SONAAR limitations, such as language support and interface changes. However, they considered our prototypes as being quite accessible, and most of them intended to continue using SONAAR prototypes. Thus, the results obtained indicate that SONAAR has the potential to contribute to improving the user experience in the authoring of accessible content.

• RQ4: Does SONAAR increase the awareness of social media users to the need to create accessible content?

Considering the results obtained on the survey dedicated to validate the effectiveness of the documentation for authoring accessible content integrated in the SONAAR prototypes, participants declared having learned new information on the benefits of adopting accessible practices and how to better describe an image for visually impaired people. Participants were also interested in knowing more about how people with disabilities use the web and practices to improve the accessibility of their content. Although we cannot presume that the interest shown by participants will improve the accessibility of the content they publish, most of them declared being more likely to be engaged in accessible practices after reading the information provided.

Further improvements

The results obtained also allow us to outline some improvements that can be considered in further developments. One of them is a minor change on the Android application home screen, providing a clearer message about its purpose. As the SONAAR application runs on the system background, this screen has no further utility. For that, it can be useful to provide further instructions on how to use SONAAR, and to display some tips and guidance on how to improve the accessibility of digital content. Participants also suggested expanding current SONAAR capabilities to support accessible content authoring on Instagram. Considering the large numbers of participants reporting using this social network, integrating it into our prototypes could represent a greater reach for SONAAR prototypes and accessibility awareness. Supporting other browsers, such as Mozilla Firefox, was also mentioned and it also should be considered for future works.

Furthermore, we reinforce the benefits of some of the suggested workflows described in previous deliverables. One of the main issues identified on the current version of SONAAR backend could be addressed by the workflows integrating a natural language processing service to translate descriptions to the user’s language. While the SONAAR description database depends on a massive use, particularly involving authoring activities, providing translated descriptions or concepts, even if with a lower quality, could be useful to users speaking other languages than English. Support for multiple languages and possible strategies to increase the number of descriptions was further explored in D3.

Another possible improvement concerns better informing users of the quality of each description. At the moment, SONAAR presents an ordered list, according to the results provided by our quality algorithm. However, a list containing two low quality descriptions is presented in the same way that a list presenting two high quality descriptions. Informing users about the quality of each description can encourage them to modify and improve low quality descriptions. In order to facilitate this communication, SONAAR could use a well known rating system, such as a star scale for each description provided.

Finally, a recurrent topic concerns the interface changes, especially in Facebook. Some participants mentioned not being able to use our Android application. After a further analysis we traced the issue to an interface update on Facebook. The suggested workflows to report a problem, including the identification of required interface elements could be useful to cope with new interface updates. Other approaches on reacting to interface changes were also further explored in D3.

Conclusions

In this report we present the study conducted to validate the new interaction flow for accessible content authoring and the effectiveness of the documentation integrated in our prototypes. Our results showed that participants welcomed the whole concept of SONAAR and reported no further accessibility issues using our prototypes, providing suggestions on how to improve our services. The documentation provided was also positively received by participants. Most of them stated it is clear, complete, and helpful. In addition, they also declared it has the potential to motivate users into accessible practices as it provides more information on how people with disabilities use the web, and the impact that improving the accessibility of their content has in others peoples' lives.

However, it is important to highlight the low level of user engagement in this study, especially among sighted users. As stated in previous deliverables, our call for participation for the first study conducted in this project and the current study had around the same order of reach. Even considering the difference in the commitment required, the conversion rate is very low. This reinforces some of our previous findings. Besides not being aware of the possibility to provide alternative descriptions, and the steps required to do it, people consider accessibility practices to be costly in terms of time and effort. While it was possible to develop a technologically feasible solution, a native and better support by major platforms could more effectively reduce the perceived effort for these users and increase their awareness.

Annex I: Dissemination of the call for participation

Annex II: Using SONAAR - Initial survey

1. Name
2. Email
3. Gender
   1. Female
   2. Male
   3. Prefer not to answer
   4. Other
4. Which operating system(s) do you use?
   1. Windows
   2. MacOS
   3. Linux
   4. iOS
   5. Android
   6. Other
5. Do you have some type of visual impairment?
   1. Yes
   2. No
   3. Prefer not to answer
6. Which one(s)?
   1. Blindness
   2. Low vision
   3. Prefer not to answer
   4. Other
7. Do you use any assistive technology?
   1. Yes
   2. No
   3. Prefer not to answer
8. Which one(s)?
   1. Screen reader
   2. Braille display
   3. Screen magnifier
   4. Prefer not to answer
   5. Other
9. Which social network(s) do you use?
   1. Twitter
   2. Facebook
   3. Instagram
   4. Other
10. How often do you post images on social networks?
    1. Several times per day
    2. Once a day
    3. Several times per month
    4. Less than once a month
    5. Never
11. Did you ever provide an alternative description for any image you post on your social networks?
    1. Yes
    2. No
12. Why not?
    1. I didn’t knew it was possible
    2. I don't know how to write a suitable alternative textual description
    3. I don't know where to write an alternative textual description
    4. It's too time consuming
    5. Other
13. How did you provide alternative descriptions for those images?
    1. Writing the description in the text of the post
    2. Writing the description in a comment or a reply to your post
    3. Writing the description in the proper text field provided by the social network
    4. Using an automatic service for alternative text generation
    5. Other
14. Considering the last 10 images you posted on your social networks, approximately for how many did you provide an alternative text description?
    1. All of them
    2. Most of them
    3. Half of them
    4. Some of them
    5. None
15. Approximately, how long does it take you to provide an alternative description for your images?
16. Do you consider the time spent on this task as being…
    1. Too much time
    2. A lot of time
    3. Some time
    4. Very little time
17. How easy or difficult is it to provide an alternative description for images?
    1. Very difficult
    2. Somewhat difficult
    3. Neither difficult nor easy
    4. Somewhat easy
    5. Very easy
18. How often do you encounter an image containing an alternative description on your social network stream?
    1. Always
    2. Very often
    3. Sometimes
    4. Rarely
    5. Never
    6. I don’t know
19. What do you think about the descriptions provided for the images you encounter on your social network stream?
    1. Very good
    2. Good
    3. Acceptable
    4. Poor
    5. Very poor
    6. I don’t know
20. How many of your social network contacts post images containing an alternative description?
    1. Everyone
    2. Most people
    3. Some people
    4. Almost no one
    5. No one
    6. I don’t know
21. Do you have some additional thoughts or comments about the subject that you would like to share?

Annex III: Using SONAAR - Final survey

1. Name
2. Email
3. How many images did you post using SONAAR?
   1. None
   2. 1 to 5
   3. 5 to 10
   4. 10 to 15
   5. More than 15
4. In which social network?
   1. Twitter
   2. Facebook
   3. Both
5. In which device
   1. Mobile device using the app
   2. Desktop or laptop using the Google Chrome extension
   3. Both
6. Considering the images you posted using SONAAR, approximately, for how many did you use the descriptions suggested by SONAAR?
   1. All of them
   2. Most of them
   3. Half of them
   4. Some of them
   5. None
7. Considering the images you posted using SONAAR, approximately, for how many did you modify the descriptions suggested by SONAAR?
   1. All of them
   2. Most of them
   3. Half of them
   4. Some of them
   5. None
8. Did you post more or fewer images than usual while using SONAAR?
   1. More
   2. About the same
   3. Fewer
   4. I don’t know
9. Did you post more or fewer images with alternative descriptions while using SONAAR?
   1. More
   2. About the same
   3. Fewer
   4. I don’t know
10. How easy or difficult was it to create descriptions for your images using SONAAR?
    1. Very difficult
    2. Somewhat difficult
    3. Neither difficult nor easy
    4. Somewhat easy
    5. Very easy
    6. Easy
11. Did SONAAR provide satisfactory descriptions for your images?
    1. Never
    2. Rarely
    3. Sometimes
    4. Very often
    5. Always
12. Did you have any particular difficulties while using SONAAR? 11. Yes 12. No
13. Which ones?
14. Do you have any suggestions for further improving SONAAR?
15. Do you intend to continue using SONAAR?
    1. Yes
    2. No
    3. Maybe
16. Why not?
17. Do you have some additional thoughts or comments about the subject that you would like to share?
18. Are you available for being contacted to provide more details about your experience?
    1. Yes
    2. No

Annex IV: Getting Started with SONAAR

Demographic questions

1. Age

2. Do you have any visual impairment?

   1. Yes
   2. No
   3. Prefer not to answer

3. Which one?

   1. Blindness
   2. Low vision
   3. Prefer not to say
   4. Other

4. Do you use any assistive technology?

   1. Yes
   2. No
   3. Prefer not to answer

5. Which one(s)?

   1. Screen reader
   2. Braille display
   3. Screen magnifier
   4. Prefer not to answer
   5. Other

   Social network usage

6. Which social networks do you use?

   1. Twitter
   2. Facebook
   3. Instagram
   4. Other

7. How often do you post images on social networks?

   1. Several times per day
   2. Once a day
   3. Several times per week
   4. Several times per month
   5. Less than once a month
   6. Never

   Engagement in accessible practices

8. Have you ever provided a text description for an image you posted on your social networks?

   1. Yes
   2. No

9. Why not?

   1. I didn’t knew it was possible
   2. I don’t know how to write a suitable text description
   3. I don’t know where to write a text description
   4. It’s too time consuming
   5. Other

10. How did you provide text descriptions for those images?

    1. Writing the description in the text of the post
    2. Writing the description in a comment or a reply to your post
    3. Writing the description in the proper field provided by the social network
    4. Using an automatic service for alternative text generation

11. How easy or difficult is it for you to provide a text description for images?

    1. Very difficult
    2. Somewhat difficult
    3. Neither difficult nor easy
    4. Somewhat easy
    5. Very easy

12. How important do you consider adding a text description for your images on social networks?

    1. Very important
    2. Important
    3. Moderately important
    4. Slightly important
    5. Not important

    Getting Started with SONAAR

    Concerning the Getting Started page provided at http://www.di.fc.ul.pt/~cad/SONAAR/documentation/doc.html:

13. How clear do you consider the information provided to be?

    1. Not clear at all
    2. Barely clear
    3. Neither clear not unclear
    4. Clear
    5. Very clear

14. How complete do you consider the information provided to be?

    1. Not complete at all
    2. Barely complete
    3. Neither complete nor incomplete
    4. Complete
    5. Very complete

15. How helpful do you consider the information provided to be?

    1. Not helpful at all
    2. Barely helpful
    3. Neither helpful nor unhelpful
    4. Helpful
    5. Very helpful

16. Have you learned new information about the accessibility of digital content by reading the Getting Started page?

    1. Yes
    2. No

17. What did you learn from that?

18. Do you think the information provided increases users’ motivations to apply accessible practices?

    1. Extremely unlike
    2. Unlikely
    3. Neutral
    4. Likely
    5. Extremely likely

19. Do you intend to be more engaged in accessible practices in your daily routine?

    1. Extremely unlikely
    2. Unlikely
    3. Neutral
    4. Likely
    5. Extremely likely

20. What information would you have liked to find on the Getting Started with SONAAR page?

21. Did you try the SONAAR prototypes?

    1. Yes
    2. No

22. Can you describe one or more problems that you may have encountered?

23. Do you intend to continue using SONAAR in your daily routine?

    1. Extremely unlike
    2. Unlikely
    3. Neutral
    4. Likely
    5. Extremely likely

24. Do you intend to start to use SONAAR in your daily routine?

    1. Extremely unlike
    2. Unlikely
    3. Neutral
    4. Likely
    5. Extremely likely

25. Do you have some additional thoughts or comments about the subject that you would like to share?