While preparing text for a foundation model, an engineer converts the sentence "AWS accelerates innovation" into a list of numeric IDs that represent subword tokens. Which generative AI concept is the engineer applying?
Chunking
Vector search
Tokenization
Generating embeddings
The process of converting raw text into discrete units (tokens) that a model can process is called tokenization. Tokenization breaks sentences into words or subword pieces and maps each token to an integer ID. Chunking refers to splitting longer content into manageable sections, embeddings map tokens to high-dimensional vectors, and vector search uses those vectors to find semantically similar items-none of which describe the initial text-to-ID conversion.
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
All IT & Cybersecurity Package plans include the following perks and exams .
Our pricing is simple. Full access to all certifications and exams in each package, for one price.
As many practice tests for as many topics as you want.
Use study mode non-stop, no limits.
Access to our AI assistant, Bash, trained to help you pass your exam.
Track your scores over time in study mode and report cards.
See how you improve over time, and where you need to focus.
Access our store with even bigger discounts than before.
Unlimited access to all performance questions and be prepared for the real thing.
All IT & Cybersecurity Package plans include unlimited access to the following study materials.
Create an account or sign in to access our study materials.