What Is Lemmatization, and Why It Makes Translations Better

Lemmatization is the step that turns "went" into "go" before anything else happens to it. Search engines, spell checkers and dictionaries rely on it. For a language learner it matters for a plain reason: dictionaries list base forms, and the words you meet in a film almost never are. This post explains what is lemmatization, how it differs from stemming, and how InterSub uses it in its dictionary.

Lemmatization meaning, in plain words

A lemma is the form of a word you would look up in a dictionary. To lemmatize a word is to find that form:

• running → run
• went → go
• mice → mouse
• better (adjective) → good
• German ging → gehen, Spanish fuimos → ir

The last two show why it isn't trivial. "Ging" shares no ending with "gehen", and Spanish "fuimos" can come from "ir" (to go) or "ser" (to be). A program that does this, a lemmatizer, needs a vocabulary of forms and some grammar, and sometimes the sentence around the word.

Stemming vs lemmatization

Both reduce words to a common base, and they are often confused.

Stemming chops endings off by rules. It is fast and needs no dictionary, but the result is not always a word: "studies" becomes "studi", "running" becomes "run" but "ran" stays "ran". Search engines used it for years because a rough match was good enough.

Lemmatization returns a real dictionary word. It knows irregular forms ("ran" → "run"), and with part-of-speech information it can tell "better" the adjective (→ good) from "better" the verb, as in "to better yourself" (→ better).

In short, the stemming vs lemmatization choice is speed versus correctness. For translation, correctness wins: a stem like "studi" has no entry in any dictionary.

Why lemmatization makes translations better

When you hover over a word in a subtitle, the word is in whatever form the sentence needs: past tense, plural, a case ending. Look that form up as is and one of three things happens: the dictionary finds nothing, finds a narrower entry, or finds a different word that happens to look the same.

Lemmatization fixes the first two. The dictionary gets "gehen", not "ging", and returns all its meanings instead of none.

The third problem needs one more step: knowing which part of speech the word is in this sentence. Compare:

• She is running a business. (verb: to manage)
• He bought running shoes. (used as an adjective: for running)

Without context, a dictionary shows the most common meaning first, which can be the wrong one here. With part-of-speech information from the sentence, the right meaning goes to the top.

How InterSub uses it

InterSub is a pop up dictionary for subtitles and web pages. When you hover over a word:

1. The word is lemmatized, so the dictionary receives its base form.
2. Its part of speech is estimated from the sentence.
3. Dictionary results are ranked by that, and the most likely meaning is shown first.
4. A translated slice of the sentence around the word appears under the entry, so you can check the meaning in context.

The base form is shown as a separate label in the pop-up. That helps you learn too: you see "ging" in the film and "gehen" in the dictionary, and the two get linked in memory.

Advanced lemmatization works for English, Spanish, German, French, Russian, Italian, Polish and Dutch. InterSub also detects phrasal verbs, idioms and compound nouns, so "give up" is translated as one unit and not as "give" plus "up".

Words you look up are saved to your Wordbook with the sentence they came from. The dictionary extension works the same way on ordinary web pages, not only in video subtitles.

Add InterSub to Chrome

Lemmatization outside language learning

The same step sits inside a lot of text processing: search, spell checkers, text classification. Developers usually take a ready lemmatizer from an NLP library such as spaCy, NLTK or Stanza rather than write one. For a learner, none of that is visible; what you notice is that the translator extension found the right word.