Backend service

Deliverable 5 (Backend service)

Document Technical Details

Document History

Contents

• Introduction
• Functionalities description
• Setup instructions

Introduction

SONAAR aims to facilitate the user-generation of accessible content on social network services by developing a solution that supports the authoring and consumption of media content on social platforms in both desktop and mobile devices. In addition to improving the accessibility of this content, the proposed solution has also the potential to raise awareness to the importance of authoring accessible content by engaging users in accessible authoring practices. This deliverable concerns the second work package of the SONAAR project, documenting the efforts to deploy a backend infrastructure to support user-generated accessible media content. SONAAR provides a mechanism to distribute an image’s alternative text to all its users. First, when an alternative text is provided, SONAAR stores the pair including the image and the user-generated alternative text in a remote repository. Then, when an image (without alternative text) is displayed to a user of a social network platform, SONAAR searchers this repository and displays the existing alternative text if the image is found. This report details the technical aspects of the backend service, as specified in Task 2.1, developed to support the operation of the two prototypes proposed in the SONAAR project – Google Chrome extension (D6.1) and Android mobile service (D7.1). Even though this deliverable is the final version of the backend service, and it deploys the support required for the core features of the SONAAR browser extension and mobile service, we anticipate that there will be updates to the backend service so that additional features of the browser extension and mobile service can be adequately supported. These updates will be detailed in deliverables D6.2, D7.2, D6.3, D7.3, D8 or D9. This document is structured as follows: Section 2 describes the functionalities deployed in this prototype. Section 3 details the instructions to be followed in order to setup the SONAAR backend service.

Functionalities description

The SONAAR backend service is responsible for supporting the operation of both the SONAAR browser extension and the SONAAR mobile service. The support is made available through a REST architecture that offers the following endpoints:

• /clarifai/search/

This endpoint is used to fetch an alternative text for an image. The image must be provided as part of the POST request as a byte array for the SONAAR browser extension, or as a base64 string for the SONAAR mobile service. The backend service uses an internal process to search for the image in the current image library. If if finds the image, it then searches for known alternative texts for that image in the alternative texts library. If it finds alternative texts, these are returned as a response to the request in JSON format.

• /clarifai/insertBuffer and /clarifai/insertBase64

These endpoints are used to store an alternative text of an image. The image and the text must be provided as part of the POST request. The image must be sent as a byte array from the SONAAR browser extension, or as a base64 string from the SONAAR mobile service. In both instances, the text must be sent as an encoded string. The backend service uses an internal process to search for the image in the current image library. If it finds the image, the known alternative texts for that image are updated with the addition of the new text in the alternative texts library. If it does not find the image, then the image is added to the current image library and a new entry for that image is created in the alternative texts library and initialized with the new text.

The backend service is responsible to search for “equivalent” images. Given that one image can be found online with small changes (e.g. reduced size, cropped, with a signature added, etc.), it is unrealistic to expect that an image can be detected deterministically. Therefore, the backend uses an external service that given two images returns a confidence level that they are the “same” image. To optimize the process, the image library is stored in this external service, and the service provides an ordered list of the degree of confidence that the image matches any of the images in the library. The external service that is being used in this project is provided by Clarifai. Currently, the free version of the service is being used, supporting an image library up to 1000 images and a total of 1000 requests per month. For the current development stage of the browser extension and mobile services this service is enough. In the remaining stages of development, this will, probably, be upgraded to one paid plan, supporting a larger number of images and requests per month.

Setup instructions

The SONAAR backend service was developed in Node.js. Any operating system that can run Node can support this service.

1. Install Node.js 14 if it is not already installed (follow instructions on https:// github.com/nvm-sh/nvm);
2. Get a Clarifai API key from https://www.clarifai.com/ (account required);
3. Install MySQL Database Server and create a new user (MariaDB also works);
4. Download the backend code from https://github.com/SONAARProject/backend;
   1. Define the folder in the server to hold the backend installation;
   2. Inside the backend folder install the dependencies by runnning npm install;
   3. Inside the backend folder build the code by running npm run build;
5. Create the database schema using the database.sql inside the backend folder;
6. Create the clarifai-api.key file outside the backend folder and write your own Clarifai API key in this file;
7. Create the claraifai-db.json file outside the backend folder and write your own database user credentials in this file (check figure 3.1);
8. Inside the backend folder, start the server by running npm run start. The service will be available on port 3001. It is necessary to configure a proxy to redirect the requests and responses from the server trough HTTPS (an SSL certificate is required for this).