Repository navigation
Support for batched translation #85
Description
A limitation we observe with the translate API is that it processes one input at a time.
Our investigation shows that batched inference significantly improves performance on CPU, we see a 2‑fold performance gain with a batch size greater than 32, which we believe is due to lower kernel dispatch, scheduling overhead, better tiling, and improved hardware utilization.
From a developer perspective, use cases are translating content across multiple unrelated elements or several individual user-generated comments on a page require passing a collection of strings (with independent context) and translating them together with the source sentence identity preserved so that they can be replaced back in the DOM.
Concretely, the proposal is to support two overloads on the translate, translateStreaming API.
Promise<sequence<DOMString>> translate(
sequence<DOMString> inputs,
optional TranslatorTranslateOptions options = {}
);
ReadableStream translateStreaming(
sequence<DOMString> input,
optional TranslatorTranslateOptions options = {}
);
The readable stream at each step will return a Sequence of DOMString.
Activity
toreini commentedon Aug 25, 2026
Hi @sushraja-msft , Thanks!
I think this point can be related to #60 . It would be great if you could have a look and if relevant, join the discussions there, especially the performance edge is an interesting discussion with the folks. If yes, then I will close this issue.
Cheers,
Ehsan
sushraja-msft commentedon Aug 31, 2026
Although there is overlap in developer use case of translating web pages, I do think batching is a separate topic than supporting HTML inputs (for example the batching API can support mime type text/html as input as well once the discussions in #60 resolve).
Batching, this is purely a way to tune performance as the developer wants for his application.
anssiko commentedon Sep 3, 2026