Internationalization

[Implementers’ doc] [Authors’ doc] [WIP]

The aim of this document is to provide implementers with a baseline for internationalization.

The current situation

Internationalization is an ongoing process, with browsers offering subpar interoperability (typography, writing modes, etc.), Operating Systems sometimes lacking fonts for some languages, and documentation providing little information on topics of interest to Reading Systems (a11y, settings, etc.).

This could well explain why the most popular Reading Apps tend to implement the least common denominator for all languages, especially when it comes to user settings, and to not support more complex languages implementers either know they can’t support well or are used on smaller markets the app/service is not targeting.

Occasionally, when an app does support a language, it can put some constraints on authors, as there is no other way to make sure the publication will be handled well otherwise e.g. specifying the Hans or Hant script for Chinese (zh).

We can’t overemphasize the importance of the internationalization process though, as the 24 languages we added extend support to 3,049,150,507 speakers, from the 1,150 people speaking Western Canadian Inuktitut, to the 1,200,000,000 speaking Chinese. Implementing right to left scripts will extend support to 411,000,000 native speakers, while vertical writing to at least 130,200,000 – excluding Chinese and Korean.

In total, we can cover the needs of 5,262,900,507 speakers. Credit where credit is due, this wouldn’t have been possible if Operating Systems and browsers didn’t tackle this process upstream, added fonts for those languages and improved support in rendering engines.

Resources

What implementers need

Supporting the maximum amount of languages and scripts is a complex process.

As a consequence, work for internationalization should be tackled early, as the changes and adaptations needed will have a significant impact on an existing implementation. It indeed impacts the entire implementation, and not only CSS.

Retrieving the significant information in the OPF

Implementers will need a way to retrieve page-progression-direction and the primary language (<dc:language>) of the publication.

Page-progression-direction

This attribute is set on the <spine> item, and the value rtl should be considered important information for the whole process.

This value signals the publication is either an RTL script, or is using the vertical-rl writing mode, which is the reason why we must find the primary language of the publication next.

The value is important to store, as it will be the one used for the dir attribute to append if it is missing in a document.

Primary Language

It is very important to note the primary language must be checked in all cases, and not only when the page-progression-direction is set or has an rtl value.

Indeed, this piece will be even more critical in the following steps, as it will trigger the list of fonts to load for the publication, the user settings to provide, and the xml:lang attribute to append if it is missing in a document.

Possible issues

The OPF file should not be considered a single source of truth for the publication, since issues may arise relatively quickly. We can’t call the process “heuristics” per se, it’s more of a chain of educated guesses.

Multiple language items

There exists an increasing corpus of EPUB files with multiple <dc:language> items. Some authoring tools, for instance, list all languages a publication contains.

In this case, page-progression-direction should serve as a hint, if present. For instance:

  1. the first <dc:language> item is English;
  2. the second <dc:language> item is Japanese;
  3. the page-progression-direction is rtl;
  4. the primary language is Japanese.

Obviously, this can quickly become an issue if both languages share the same page-progression-direction…

  1. the first <dc:language> item is English;
  2. the second <dc:language> item is Japanese;
  3. the page-progression-direction is missing;
  4. we can’t guess the primary language from the OPF.

In such an edge case, to achieve the best interoperability possible, the first <dc:language> element must be considered the primary language, unless you can pre-process all documents in a publication to determine it beforehand.

Missing page-progression-direction

For some reason the page-progression-direction may be missing in the OPF, which can be true if the publication is EPUB2 for instance – which supports the direction CSS property and, in theory, could support RTL scripts.

The following guidance is informal:

  1. if the page-progression-direction is missing;
  2. if there is only one <dc:language> item which clearly signals the page-progression-direction:
    • ar, fa, and he;
    • zh-Hant, or zh-TW.
  3. then you can assume the page-progression-direction is rtl.

The decision to handle this edge case is up to each implementer, especially as it can be considered a patch of an authoring failure.

Triggering the correct page-progression-direction

Once the page-progression-direction is defined as rtl, it must be reversed in the app:

  • the previous resource (document) is on the right;
  • the next resource (document) is on the left.

Navigating the publication should follow this pattern.

Appending information into documents

Missing attributes in each document is far from an uncommon or edge case.

Since the page-progression-direction or <dc:language> are already set in the OPF, some authors might think they will automatically apply to all the resources in the EPUB file, and explicitly set it only when it differs from those global values. More importantly, some Reading Apps are automatically managing this, and should authors only check their files in those apps, it could lead them to believe it just works.

Language

The language is important as it will enable hyphenation and use the proper rules specific to each language if a dictionnary is available, change the default typeface for some languages, and even apply language-specific styles for layout (e.g. pagination, defaults for unstyled publications, etc.).

The following process must be implemented:

  1. if xml:lang can’t be found on html;
  2. check if xml:lang can be found on body, copy and set it to html, and stop there if it is;
  3. if it can’t be found on body, use the primary language retrieved from the OPF file and set it to html and body.

Direction

The dir attribute is critical too, as it will reverse the column direction for RTL scripts.

The following must only apply if the primary language is ar, fa, and he. It MUST NOT apply to CJK.

The following process must be implemented:

  1. if the dir attribute can’t be found on html;
  2. check if dir can be found on body:
    1. if it is the same value as the one retrieved from the OPF file, copy it;
    2. if it differs from the one retrieved from the OPF file, change the value;
  3. set the dir attribute with the correct value on html.

Loading the correct styles

Readium CSS provides implementers with specific stylesheets for RTL scripts and CJK (including the Mongolian script), those styles must be loaded accordingly. Otherwise, styles for LTR scripts in the horizontal writing mode will apply.

Guidance can be found in the Injection and Pagination document.

Modifying the UI of the app

Ideally, several parts of the app should be customizable depending on the publication. Another option is implementing the least common denominator for all languages.

All Languages

The list of fonts the app offers to users should be specific to the primary language of the publication, and writing-mode if it applies – Japanese currently.

This means fonts for Latin language can’t be reused for Indic, RTL scripts, CJK, etc.

RTL

Several parts of the UI must follow the direction (rtl) of the primary language:

  • the running header (title of the publication or chapter);
  • the toc and its entries;
  • user settings e.g. text align;
  • implementers might want to localize the interface based on the language set at the system level, or at least fall back to English.

Moreover, some user settings should be removed if used (letter- and word-spacing) and another one added (arabic ligatures in ar and fa).

CJK

Although the UI can keep an ltr direction with a horizontal-tb writing mode, some extra attention should be paid:

  • make sure the “UI font” can display the characters needed in those languages;
  • implementers might want to allow authors to set a vertical-* writing mode for the navigation document (nav.xhtml);
  • implementers might want to localize the interface based on the language set at the system level, or at least fall back to English.

Moreover, some user settings should be removed if used (text-align, hyphens, letter- and word-spacing, number of columns in vertical writing, etc.). See User Prefs for further details.

Internationalize features

Implementers should make sure features like search, highlighting, etc. can work well with bidirectional text and unicode – CJK, especially as some characters change from horizontal to vertical writing modes.

Another issue to take into account is that input methods might not allow users to use some features easily, in which case extended research should be made to check realistic options.

Fonts

Implementers should make sure they offer at least two options: the publisher’s font and the default. If they want to extend the list of fonts they provide for some languages, they should follow guidance in the User Preferences doc.

How authors can help

Authors can help in different ways, from authoring files to giving valuable information we can use to improve the entire Readium CSS project.

Authoring

In some cases, EPUB files themselves will help us improve support tremendously.

The metadata in the OPF file will help us trigger the expected behavior. For instance:

  • declaring only one <dc:language> will help us determine the primary language of the publication;
  • if this is not possible, add the primary language as the first <dc:language> item in the OPF;
  • declaring the script e.g. zh-Hant or mn-Mong will help us trigger the correct writing-mode;
  • using the EBPAJ guide meta will help us improve support across platforms.

Setting attributes in each document of the publication will also help us make educated choices and could eventually allow us to handle some edge cases like mixed directions or mixed writing modes. Our aim is obviously not to put the burden on authors there, but collaboration is the only solution to get around those edge cases the EPUB specification doesn’t discuss.

What you can do:

  • embed fonts when necessary, so that users have at least two options to choose from in settings and/or proper support of the language;
  • set the xml:lang attribute on html;
  • set the dir attribute on html;
  • avoid using the direction CSS property, as it is now recommended.

Reporting issues

First and foremost, please be assured that we’re as annoyed as you are if there is an issue. It doesn’t mean we don’t care, far from it… it just means we lacked information or misunderstood some requirements when designing Readium CSS. Our goal is to achieve the best support possible, and we know we can’t fulfill it without your expertise.

If you encounter any issue specific to a language or script, do not hesitate to open a new issue in the github repository. A reduced sample will definitely help. Check our guidelines for further details.

The most information we have, the better the fix will be. Please feel free to go into extensive details and provide links, documentation, examples, etc. we can check to get a solid grasp of the issue we have to fix.

Sending feedback

We’re not saying you should implement a support of script yourself if we don’t support it at all or very well, but it is noteworthy that we can’t necessarily get all the details right, especially when we don’t know how to speak and write the language you wish were supported or improved.

If our font-stacks are not good enough, a solider one will help. If we don’t have any font-stack for a language, a basic one will do. Additionally, you can point us to libre/open-source fonts which you think should be recommended to implementers.

If we’ve got something wrong in terms of typography, an example – picture, codepen demo, etc. – will help us see how we got there. Please bear in mind we might be limited by what web browsers can do though, and some issues could be consequently deemed “out of scope.” We will at least do our best to report the issue to whom it may concern whenever possible.

Check our contribution guidelines for further details.

Overarching issues

Implementers should be aware there are overarching issues for which we haven’t reached consensus, or couldn’t discuss yet.

The most important issue, by very far, is that checking the writing-mode at runtime can blow performance in extreme ways. It can indeed take 15 seconds to render some complex XHTML files in vertical-*. Needless to say, this would obviously be worse in terms of UX. And this is the reason why we try to guess the writing-mode from the OPF file.

Longer terms issues include:

Out of scope

There are some typography and layout issues which are not the responsibility of apps’ implementers but rendering engines’. Those issues include:

If those issues arise, please report them to whom it may concern (e.g. Chromium, Firefox, Microsoft, Webkit, etc.). The entire web platform will indeed benefit. You can additionally report the issue to us so that we can document it for other implementers.

Glossary

Check the i18n Glossary doc for a glossary of terms you may encounter during bug reports, issues, or feedbacks.