AWS Certified AI Practitioner AIF-C01 Practice Question

To know if the foundation model is helping achieve the stated business goal-longer user sessions-you must measure the KPI itself. An A/B test that compares average session duration for users who receive the new AI-generated summaries versus a control group directly ties model performance to the engagement metric the business cares about. ROUGE scores and BLEU scores evaluate linguistic quality against references but may not correlate with user engagement. Inference latency measures system performance, while parameter count says nothing about the business impact. Neither of those metrics reveals whether the summaries keep users in the app longer.