AI-300 Operationalizing Machine Learning and Generative AI Solutions (beta) Questions and Answers

The Evaluation experience in Microsoft Foundry ' s prompt flow editor shows aggregate quality scores across prompt variants — which variant produces more coherent answers, which scores higher on groundedness. However, it does not expose per-run raw telemetry: individual token counts per call, per-request latency in milliseconds, or the exact input-output pairs for each execution. The Evaluation experience is designed for comparative quality scoring, not for detailed operational telemetry. To capture inputs, outputs, token usage, and latencies at the granular run level, Tracing must be enabled in Microsoft Foundry. Tracing records each LLM call as a structured span with timing, token consumption, and the complete input-output payload — a fundamentally different view than evaluation scores that directly satisfies all four capture requirements.

Microsoft Learn Reference Topic: Trace and debug prompt flows in Microsoft Foundry – Tracing vs. Evaluation

All three requirements point to Git as the underlying mechanism, but each uses a different Git capability. For reviewing changes before production: a branch-based workflow where prompt changes are made on feature branches and merged to the main branch only after pull request approval enforces the review gate. For restoring previous versions: Git ' s commit history and tag system provide a precise, immutable record of every prompt state, and a git revert or checkout to a specific commit SHA restores any prior version instantly. For governance parity with application code: by storing prompts in the same Git repository as application code, all the same CI/CD, branch protection, code review, and audit trail policies apply automatically. The alternatives such as Blob Storage or embedded configuration files lack native review workflows, branch protection, and full audit history.

Microsoft Learn Reference Topic: Prompt management and versioning with Git integration in Microsoft Foundry

The two requirements target fundamentally different stages of the AI application lifecycle. Continuous runtime monitoring addresses production behavior — an application that intermittently times out needs real-time visibility into latency, error rates, and request volumes for every call in production. Tracing backed by Application Insights or OpenTelemetry exporters is correct here because it captures granular, per-request telemetry continuously without requiring manual triggering — it is always-on by design. Release-time validation addresses pre-release quality gates — before new prompts or model updates go live, you need to verify they have not introduced regressions. An evaluation pipeline in Microsoft Foundry ' s prompt flow is correct here: it runs the updated application against a predefined test dataset, scores outputs on quality and safety metrics, and passes or fails the release based on thresholds. Evaluation pipelines run on demand triggered by CI/CD, not continuously.

Microsoft Learn Reference Topic: Microsoft Foundry observability – Tracing for runtime monitoring vs. Evaluation pipelines for release gates

Microsoft ' s Azure AI Foundry documentation defines built-in roles scoped to the resource hierarchy. The Azure AI Administrator role grants permissions to manage the Azure AI Hub resource itself — including network settings, shared connections, quota, and governance configuration — which is appropriate for platform engineers who configure resource-level settings. The Azure AI Developer role grants permissions to create, update, and manage project-level assets such as deployments, prompt flows, fine-tuning jobs, and evaluations, without access to resource-level configuration. This precisely matches the developer requirement and enforces least privilege. Disabling Entra ID authentication (option B) violates security policy and removes the identity-based access control that makes RBAC possible. Sharing a single API key (option D) violates the least-privilege principle — all users would have identical, undifferentiated access with no audit trail.

Microsoft Learn Reference Topic: Manage access to Azure AI Foundry – Built-in roles: Azure AI Administrator and Azure AI Developer

For tracking changes across contributors, Git integration is the answer: by connecting the Microsoft Foundry project to a Git repository, every prompt file change is tracked as a commit with author attribution, timestamp, and diff view, and pull requests enforce review before changes reach production. The Git history provides the complete audit trail and rollback capability needed for traceability. For allowing applications to consume updated prompts without requiring redeployment, Microsoft Foundry ' s prompt management feature allows prompts to be stored and versioned as named artifacts in the project. Applications reference prompts by name and load the latest approved version at inference time, rather than having prompt text hard-coded in the application deployment artifact. This decoupling means updating a prompt is a content operation — not a code deployment — so applications automatically pick up the new prompt without any redeployment.

Microsoft Learn Reference Topic: Prompt management in Microsoft Azure AI Foundry – Git integration and dynamic prompt versioning

The key concept here is low-priority (spot) VMs, which are available at a significant discount because Azure can reclaim them at any time. Azure Machine Learning compute clusters are the only target in the list that directly supports low-priority VM nodes as a cost-saving configuration. You set the minimum and maximum node counts and specify that new nodes should be provisioned as low-priority. Azure Container Instances (ACI) does not support low-priority pricing. Local deployment runs on the developer ' s machine with no Azure billing model. Azure Kubernetes Service (AKS) does support spot node pools but requires significantly more infrastructure management and is not the primary mechanism for low-priority compute in Azure Machine Learning. The exam tests whether you know that AML compute clusters are the managed way to leverage low-priority discounts inside Azure Machine Learning.

Microsoft Learn Reference Topic: Create and manage Azure Machine Learning compute clusters – Low-priority VMs

In Azure Machine Learning, a workspace is the top-level resource that acts as a centralized hub for every ML activity. It stores experiments, pipelines, datasets, models, environments, and compute targets all in one place. Before you can register assets or deploy endpoints, the workspace itself must exist and be shared across the team. Fabrikam ' s core challenge is inconsistent experiment tracking, ad-hoc versioning, and no standardized deployment path — all symptoms of the absence of a single, governed workspace. Option A (registering assets) is a subsequent action only possible after the workspace exists. Option C (online endpoint) is a deployment concern, not an asset-management foundation. Option D (Foundry project) is appropriate for generative AI workloads, not for the traditional ML pipeline standardization described here. The workspace is the prerequisite that makes every other governance action possible.

Microsoft Learn Reference Topic: Azure Machine Learning workspaces – Microsoft Learn: Manage Azure Machine Learning workspaces

Imagine a hospital analytics firm with data scientists who kick off dozens of training jobs throughout the week. During peak hours, five jobs compete for the same GPU cluster and fail or queue for hours. On quiet nights, that cluster sits completely idle, burning money. Managed compute targets with autoscaling solve both problems: the cluster scales out automatically when multiple jobs arrive simultaneously and scales back to zero when idle. Option A (single shared cluster) is exactly the resource-contention problem Fabrikam already has. Option B (fixed-size cluster) wastes money during off-peak hours. Option C (dedicated per-experiment clusters) eliminates contention but is prohibitively expensive for a cost-conscious healthcare firm. Autoscaling managed compute is the cloud-native answer to variable workload demand.

Microsoft Learn Reference Topic: Azure Machine Learning compute targets – Configure autoscale for compute clusters

MLflow is the industry-standard open-source platform for experiment tracking, and Azure Machine Learning has first-class native integration with it. When you use MLflow within an Azure ML job, parameters, metrics, and artifacts are automatically logged to the run history of the AML workspace, making every run reproducible and comparable. Option A (AML job output logs) only captures console output and lacks structured parameter and metric logging. Option C (Application Insights logs) is designed for application-level telemetry, not ML experiment metadata. Option D (Azure Monitor alerts) is a reactive notification tool, not a tracking system. MLflow ' s autologging capability means that for common frameworks such as scikit-learn, XGBoost, and PyTorch, parameters and metrics are captured without a single line of custom code, directly answering Fabrikam ' s requirement for consistent experiment tracking.

Microsoft Learn Reference Topic: Track ML experiments with MLflow – Azure Machine Learning MLflow integration