An analyst must feed a 40-page report into a large language model that accepts at most 8,000 tokens per prompt. Which preprocessing step is commonly applied to split the document into smaller, overlapping sections so the model can process and recall the full context?
Vector quantization
Chunking
Tokenization
Embedding
Chunking divides long text into manageable, usually overlapping segments. This lets downstream steps-such as creating embeddings or sending prompts-fit within the model's token limit while still maintaining enough contextual overlap for accurate responses. Tokenization breaks text into individual tokens but does not manage length limits; embeddings convert text to vectors after any needed splitting; vector quantization compresses numerical vectors rather than dividing documents.
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
All IT & Cybersecurity Package plans include the following perks and exams .
Our pricing is simple. Full access to all certifications and exams in each package, for one price.
As many practice tests for as many topics as you want.
Use study mode non-stop, no limits.
Access to our AI assistant, Bash, trained to help you pass your exam.
Track your scores over time in study mode and report cards.
See how you improve over time, and where you need to focus.
Access our store with even bigger discounts than before.
Unlimited access to all performance questions and be prepared for the real thing.
All IT & Cybersecurity Package plans include unlimited access to the following study materials.
Create an account or sign in to access our study materials.