A new user-friendly app developed by the CSIR will help learners struggling with early literacy difficulties, dyslexia, attention deficit hyperactivity disorder (ADHD) and/or some form of visual impairment.

The app, named iSinkwe, which is isiZulu for a bushbaby, does this by turning human-narrated or computer-generated audio into text.

Using South Africa’s 11 official languages, the app allows users to create more accessible textbooks, class notes and other documents, opening up an interactive world of reading and learning for them.

“Conventional reading is currently accessible only to the traditionally literate person,” said CSIR voice computing research group leader Karen Calteaux.

“This app targets both the day-to-day content of educators and publishers’ textbooks. It adds value to the publishing industry by breaking down these barriers to turn non-readers into readers.

“iSinkwe adds audio, which was either recorded manually by a voice artist or generated automatically by a computer, to a standard EPUB document.”

Tapp automates the synchronisation of the audio to the text of the document at word, sentence and paragraph levels.

This allows users to switch between these levels when they read or navigate through the document using audio and highlighting.

iSinkwe does not translate the content from one language to another, but simply adds audio to documents that are already written in the language of the user’s choice.

“If users only have a Microsoft DOCX or PDF document, they can convert those documents automatically to the standard EPUB3 format using iSinkwe Convert. They simply upload the document to the iSinkwe website, and it will provide them with an EPUB3 version that they can download,” said product leader Georg Schlünz.

“iSinkwe Synchronise allows users to upload a standard EPUB3 document or the pre-existing human-narrated audio files to the iSinkwe website. When no human-narrated audio is available, computer-generated audio is added instead.”

It also allows for a combination of human-narrated and computer-generated audio. Users can then download the EPUB3 document with the synchronised text and audio included.

To read any of the EPUB3 documents with synchronised text and audio, users need to install the iSinkwe reading app, which is currently available for Android, with iOS and Windows versions coming soon.

Once the EPUB3 document is added to the library and opened, users can listen to and navigate through the audio on word, sentence and paragraph levels.

Text highlighting on these levels follows the audio automatically as it is played. Users can customise the reading experience further by changing the font type, font size and background colour, as well as the playback speed of the audio and associated text scrolling behaviour.

The CSIR piloted iSinkwe to teachers and learners at Jan Kriel, Langerug and Tafelberg schools.

“The schools have already requested additional features. The reading assistance provided by the app must further convert text, math and other visuals in images to audio. The app must also give writing assistance in the form of automated dictation.

“We are excited to include these features in the product roadmap of the next three years,” said Schlünz.