Talk to Me: Media Overlays

As you might suspect at this point, the reading system can’t synchronize or play content back any way but what has been defined; as a reader you cannot, for example, dynamically change from word-to-word to paragraph-by-paragraph read-back as you desire. The magic is only as magical as you make it, at least at this time.

With only a single level of playback granularity available, the decision on how fine a playback experience to provide has typically been influenced by the disability you’re targeting, going back to the origins of the functionality in talking books. Books for blind and low-vision readers are often only synchronized to the heading level, for example, and omit the text entirely. Readers with dyslexia or cognitive issues, however, may benefit more from word-level synchronization using full-text full-audio playback.

Coarser synchronization—for example at the phrase or paragraph level—can be useful in cases where the defining characteristics of a particular human narration (flow, intonation, emphasis) add an extra dimension to the prose, such as with spoken poetry or religious verses. The production costs associated with synchronizing human-narrated ebooks to the word level, however, has typically meant that only short-prose works (such as children’s books) get this treatment.

Let’s turn to the practical construction of an overlay to discover why the complexity increases by level, though. Understanding the issues will give better insight into which model you ultimately decide to use.

Building an Overlay

Every overlay document begins with root smil element and a body, as exemplified in the following markup:

<smil
    xmlns="http://www.w3.org/ns/SMIL"
    xmlns:epub="http://www.idpf.org/2007/ops"
    version="3.0">
    <body>

    </body>
</smil>

There’s nothing exciting going on here but a couple of namespace declarations and a version attribute on the root. These are static in EPUB 3, so of little interest beyond their existence. There is no required metadata in the overlays themselves, which is why we don’t need to add a head element.

Of course, in order to now illustrate how to build up this shell and include it in an EPUB, we’re going to need some content. I’m going to use the Moby Dick ebook that Dave Cramer, a member of the EPUB working group, built as a proof of concept of the specification for the rest of this section. This book is available from the EPUB 3 Sample Content project page.

If we look at the content file for chapter one, we can see that the HTML markup has been structured to showcase different levels of text/audio synchronization. After the chapter heading, for example, the first paragraph has been broken down to achieve fine synchronization granularity (word and sentence level), whereas the following paragraph hasn’t been divided into smaller parts.

Compressing the markup in the file to just what we’ll be looking at, we have:

<section id="c01">
    <h1 id="c01h01">Chapter 1. Loomings.</h1>

    <p><span id="c01w00001">Call</span>
        <span id="c01w00002">me</span>
        <span id="c01w00003">Ishmael.</span>
        <span id="c01s0002">Some years ago…</span> …</p>

    <p id="c01p0002">There now is your insular city of the Manhattoes…</p>
    …
</section>

You’ll notice that each element containing text content has an id attribute, as that’s what we’ll be referencing when we get to synchronizing with the audio track.

The markup additionally includes span tags to differentiate words and sentences in the first p tag. The second paragraph only has an id attribute on it, however, as we’re going to omit synchronization on the individual text components it contains to show paragraph-level synchronization.

We can now take this information to start building the body of our overlay. Switching back to our empty overlay document, the first element we’re going to include in the body is a seq:

<body>
    <seq
        id="id1"
        epub:textref="chapter_001.xhtml#c01"
        epub:type="bodymatter chapter">

    </seq>
</body>

This element serves the same grouping function the corresponding section element does in the markup, and you’ll notice the textref attribute references the section’s id. The logical grouping of content inside the seq element likewise enables escaping and skipping of structures during playback, as we’ll return to when we look at some structural considerations later.