Extract UI and DataViews translations to a separate package

[jsnajdr]

Implements @aduth's idea from #80714 (comment): extract the wp-ui and wp-dataviews translations to a separate non-bundled packages wp-ui-i18n and wp-dataviews-i18n.

When a plugin bundles one of these libraries, the __() calls are no longer present in the bundled asset. They are in an external package that is provided by WordPress itself. Or someone else (?)

Instead of:

import { __ } from '@wordpress/i18n';
__( 'Clear' );

we now write:

import i18n from '@wordpress/ui-i18n';
i18n.CLEAR();

The ui and dataviews i18n catalogs thus become part of Core translations.

Open questions:

1. If the translations are now part of Core, then a plugin that bundles a new version of a library, and runs on an old version, can miss some new translations. They are not part of the old language pack. Are we OK with that?
2. Missing translations are currently going to crash. If there is no FOO message, calling i18n.FOO() will fail. This is solvable by exporting a proxy object that implements a fallback getter.
3. Could we possibly ship the translations also with the plugin? Like, registering multiple versions of wp-ui-i18n, and the best match wins at runtime?

[github-actions]

Warning: Type of PR label mismatch

To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.

• Required label: Any label starting with [Type].
• Labels found: .

Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: jsnajdr <jsnajdr@git.wordpress.org>
Co-authored-by: tyxla <tyxla@git.wordpress.org>
Co-authored-by: youknowriad <youknowriad@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[tyxla]

To be honest, having to create extra packages for just the translations feels like too much. I'm worried by the precedent this is establishing - each bundled package needing its own corresponding i18n package - it looks like a major architectural flaw. And as you described, it comes with its own set of additional tradeoffs, some of which have serious downsides:

• Potential crashes - that's the one that worries me most. Especially considering that this is technically introducing a new set of public APIs. Missing translations or missing keys must never crash the packages IMO.
• Potentially missing translations - this is what we're trying to avoid, no?
• Creating new problems - like concurrency when multiple versions of the i18n packages are loaded

I wonder if we could try a "simpler" alternative. For example, have we considered keeping ordinary gettext calls beside their translator comments? During the package build, extract them, generate the Core catalog, and rewrite bundled output to a safe lookup containing the English fallback, which will help preserves locality and avoids the shallow interface introduced for DataViews.

Alternatively, as a short-term solution, a way is to rewrite into a "consumer" textdomain, similar to how some plugins like Jetpack implement it (see https://github.com/Automattic/jetpack/tree/trunk/projects/js-packages/babel-plugin-replace-textdomain). It works on old WordPress and requires no Core changes, but a downside is that it duplicates translations and permits inconsistent wording between plugins.

It also makes me think if it's a good time to explore a better long-term solution for those i18n problems. Especially considering that all plugins that use those bundled packages are also currently hitting those problems (for example, WooCommerce is one where we've already seen reports for). One way could be to extend the generated asset metadata with bundled package identity, version/hash etc. WordPress.org would use it to avoid attributing package strings to plugins; Core would use it to load multiple shared catalogs without treating them as hard JavaScript dependencies. Of course, this would require coordinated changes across Gutenberg tooling, WordPress.org, language-pack generation, and Core.

I wonder what other alternatives we've considered.

[youknowriad]

Did we consider a build tool solution for this instead? Like having a the wp-build, scripts tooling add the right plugin domain when bundling these packages?

[jsnajdr]

For example, have we considered keeping ordinary gettext calls beside their translator comments?

Not sure what you mean by this, but it's possible to keep the actual string as the key in the message catalog:

import i18n from '@wordpress/ui-i18n';
i18n.__( 'Clear' );

and the actually extracted translation is also in @wordpress/ui-i18n:

catalog[ 'Clear' ] = () => __( 'Clear' );

The main change is that we're replacing the symbolic CLEAR constant with the actual string. That way, no information is really lost, it's still at both places and is potentially extractable.

rewrite into a "consumer" textdomain, similar to how some plugins like Jetpack implement it

Yes, this is an alternative solution that is already used in practice. Rewrite the textdomain and duplicate the string. Translation tools typically have a "translation memory" feature that lets you quickly match and reuse already translated strings.

One way could be to extend the generated asset metadata with bundled package identity, version/hash etc. WordPress.org would use it to avoid attributing package strings to plugins; Core would use it to load multiple shared catalogs without treating them as hard JavaScript dependencies.

This could be another viable solution. The bundled packages would use a custom textdomain for their translations:

__( 'Clear', 'wordpress-ui' );
__( 'Optional', 'wordpress-dataviews' );

These textdomains end up in the bundled JS asset, and are extracted by GlotPress/make-pot. The tool drops the textdomain info, but the extraction is performed.

When generating the language pack for the plugin and the JS asset, GlotPress could include the wordpress-ui translations there, in the JSON chunk. The JSON format has support for multiple domains in one file.

Then we need to patch the Core function that prints the translations (print_translations in class-wp-scripts.php) to print translations not only for the default plugin domain, but also for additional domains in the same chunk.

[tyxla]

One way could be to extend the generated asset metadata with bundled package identity, version/hash etc. WordPress.org would use it to avoid attributing package strings to plugins; Core would use it to load multiple shared catalogs without treating them as hard JavaScript dependencies.

This could be another viable solution. The bundled packages would use a custom textdomain for their translations:

Maybe something long those lines is the build tool solution @youknowriad has in mind?

[youknowriad]

The problem with the "core" domain or the custom ones per package is the divergence between the strings available in Glotpress and the code, there's no easy solutions here. (you could have strings on glotpress not available in the version used by the plugin or vice versa).

[jsnajdr]

Maybe something long those lines is the build tool solution @youknowriad has in mind?

That sounds more like the Jetpack solution with babel-plugin-replace-textdomain. That leads to each plugin domain having translation strings that the plugin author never added, and that need to be translated again over and over. Unless copying the shared translations is somehow automated.

[youknowriad]

That sounds more like the Jetpack solution with babel-plugin-replace-textdomain. That leads to each plugin domain having translation strings that the plugin author never added, and that need to be translated again over and over.

Sure, yes, there's duplication but I still think it's the best tradeoff personally.

[jsnajdr]

The problem with the "core" domain or the custom ones per package is the divergence between the strings available in Glotpress and the code, there's no easy solutions here.

Is this really different from any other situation where there are new Glotpress translations, and not everything is yet 100% translated? When releasing a new version of a bundled library, make sure that the Glotpress project is updated accordingly. Not really different from releasing a new version of a plugin.

[youknowriad]

not everything is yet 100% translated?

It's not just: not everything is 100% translated, it's also, some strings have been removed I think and will never be translated (using an old version with an old string)