During an error analysis of a cosine-similarity search that uses TF-IDF vectors, you discover that the tokens "go", "going", "goes", and "went" are still treated as separate features after Porter stemming. This inflates the dimensionality and causes semantically similar reviews to appear dissimilar. Which single preprocessing adjustment would most directly reduce this sparsity without introducing out-of-vocabulary strings?
Replace the Porter stemmer with a lemmatizer that uses part-of-speech tags and a morphological lexicon to return dictionary lemmas.
Expand the tokenizer to generate bi-grams and tri-grams instead of only unigrams.
Swap the TF-IDF representation for Word2Vec embeddings trained on the same corpus.
Introduce character-level byte-pair encoding (BPE) tokenization before computing TF-IDF vectors.
Porter and other rule-based stemmers trim affixes heuristically; they therefore leave irregular forms such as "went" untouched and sometimes emit non-words (e.g., "univers"). A lemmatizer, on the other hand, consults part-of-speech tags and a morphological dictionary, mapping all inflections-including irregular past tenses and comparative adjectives-to their canonical dictionary lemma ("went" → "go", "better" → "good", "children" → "child"). Replacing the stemmer with a POS-aware lemmatizer therefore collapses these variants into a single, valid token and directly lowers vector sparsity. The other options change representation (Word2Vec), token segmentation (BPE), or n-gram span but do not guarantee that irregular forms converge to one dictionary word, so they would not address the specific sparsity caused by morphological variation.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is the difference between stemming and lemmatization?
Open an interactive chat with Bash
How does a lemmatizer handle irregular words like 'went' or 'children'?
Open an interactive chat with Bash
Why does replacing TF-IDF with Word2Vec embeddings not solve the sparsity issue here?