AIF-C01 Domain 3 - Applications of Foundation Models
Domain 3 is 28% of the scored AWS Certified AI Practitioner (AIF-C01) exam. It is the largest domain and covers model/application design, RAG, prompt engineering, customization, agents, and evaluation.
Model and application design
Choose a model using representative task tests plus cost, modality, language, latency, complexity, context window, output length, customization, safety, licensing, and regional availability. A larger model is not automatically the best model.
Inference parameters
- Temperature: rescales token probabilities. Lower values are usually more focused/repeatable; higher values usually increase variation.
- Top-p (nucleus sampling): samples from the smallest candidate set whose cumulative probability reaches
p. Lower values narrow the set; higher values generally permit more diversity.p = 1does not mean least random. - Top-k: limits candidates to the
kmost probable tokens; smallerknarrows choice. - Maximum output tokens: caps response length, cost, and worst-case latency.
- Stop sequence: ends generation when a specified sequence appears.
Provider implementations differ, and temperature/top-p interact. Change one control at a time and evaluate rather than relying on absolute rules.
Retrieval-Augmented Generation (RAG)
RAG grounds generation in external information without changing model weights:
- ingest and clean source documents;
- split them into chunks;
- create embeddings and store them in a vector index;
- embed the user’s query and retrieve similar chunks;
- optionally filter or rerank results;
- augment the prompt with retrieved evidence;
- generate a response and return source citations where supported.
RAG is strong for private, current, or frequently changing knowledge and supports traceable citations. It can reduce hallucination, but poor source data, chunking, retrieval, or prompting can still produce wrong answers.
The current exam examples name Amazon OpenSearch Service, Amazon Aurora, Amazon Neptune, and PostgreSQL on Amazon RDS as vector-capable stores. Amazon DocumentDB also supports vector search, and newer AWS capabilities such as S3 Vectors may appear in architectures even when not named in the objective examples.
Amazon Bedrock Knowledge Bases manage much of the ingestion, embedding, retrieval, and generation workflow.
Customization tradeoffs
| Approach | Changes weights? | Best for | Relative effort/cost |
|---|---|---|---|
| Prompt/in-context learning | No | instructions, examples, rapid iteration | Low |
| RAG | No | private/current factual knowledge and citations | Low-medium |
| Fine-tuning/instruction tuning | Yes | stable behavior, style, format, task specialization | Medium-high |
| Continued pretraining | Yes | adding broad domain language/knowledge from unlabeled corpora | High |
| Distillation | Trains a smaller student | lower latency/cost while retaining teacher behavior | Medium-high |
| Pretraining from scratch | Yes, all weights | unique base-model requirements at very large scale | Very high |
RAG is usually the first choice for changing facts; fine-tuning is usually better for repeatable behavior. They can be combined.
Prompt engineering
A useful prompt separates:
- role/system behavior;
- explicit task instruction;
- relevant context and delimiters;
- input data;
- constraints and negative instructions;
- examples;
- required output format and quality criteria.
Techniques
- Zero-shot: instruction with no example.
- Single-shot/one-shot: one example.
- Few-shot: several representative examples.
- Chain-of-thought prompting: asks for intermediate reasoning; useful conceptually, but production systems can request concise rationale or structured verification rather than exposing hidden reasoning.
- Prompt template: reusable parameterized structure.
- Negative prompt: specifies what to avoid, common in image generation and output constraints.
Best practices are specificity, concision, clear delimiters, representative examples, explicit output schemas, systematic tests, guardrails, and version control. Amazon Bedrock Prompt Management can save, version, test, and reuse prompts.
Prompt risks
- Prompt injection/hijacking: untrusted input tries to override instructions.
- Jailbreaking: attempts to bypass safeguards.
- Prompt poisoning: malicious content enters retrieved data, examples, or other context.
- Prompt exposure: system instructions or confidential context leak in output.
Treat retrieved text and tool output as untrusted data; use least privilege, input separation, allow-listed tools, output validation, monitoring, and human approval for high-impact actions.
Fine-tuning data and methods
- Curate relevant, representative, legally usable, high-quality data.
- Remove sensitive data and duplicates; document lineage and licenses.
- Use consistent labels/instructions and split evaluation data to avoid leakage.
- Balance demographic and task coverage.
- Instruction tuning improves compliance with task directions.
- Domain adaptation specializes vocabulary/behavior for a field.
- Transfer learning adapts learned representations to a downstream task.
- RLHF uses human preference feedback to train a reward signal and align behavior.
AI agents
Agents break a goal into steps, choose/call tools, observe results, use memory, and continue. Use them for multi-step tasks that require external systems or actions. Prefer deterministic workflows for fixed, predictable sequences.
As of July 30, 2026, Bedrock Agents is named Bedrock Agents Classic and is closed to new customers. Use Amazon Bedrock AgentCore and an agent framework such as Strands Agents for new production agent workloads.
Evaluation
Approaches
- held-out representative datasets and regression suites;
- human-in-the-loop review and preference comparisons;
- standard benchmark datasets;
- Amazon Bedrock Model Evaluation;
- LLM-as-a-judge, with calibration and human checks for judge bias/error;
- end-to-end application evaluation of retrieval, agents, tools, and workflows.
Metrics
- ROUGE: reference/generated n-gram or sequence overlap; commonly used for summarization recall.
- BLEU: precision-oriented n-gram overlap with brevity penalty; commonly used for translation.
- BERTScore: semantic similarity using contextual embeddings.
- Perplexity: next-token predictive fit; lower is better for a given comparable setup, but it does not directly measure usefulness or truth.
- Business/application metrics: task-completion rate, grounded-answer/citation quality, user satisfaction, productivity, engagement, latency, cost per interaction, and escalation/error rate.
No single metric captures factuality, safety, usefulness, and business value. Evaluate the full application, not only the base model.