Conditional Access Essentials: From Report-Only to Enforced Mode

Introduction

Welcome to the final part of my Conditional Access Essentials series! By now, you have your policies defined – great - but how do you safely move from report-only mode to full enforcement? And how do you monitor their impact along the way?

Conditional Access is powerful, but it’s also risky. Misconfigurations can lock users out of critical apps, and even put privileged accounts at risk of losing access to your tenant entirely. That’s why testing, monitoring, and staged rollouts are essential before hitting “enforce.”

In this post, I’ll cover:

• how to safely transition from report-only to enforced mode

• tools to monitor policy impact, including the “What If” tool, Policy Impact feature, and Log Analytics integration

• using KQL queries to dig deeper into Conditional Access logs

Let’s dive in and make sure your Conditional Access policies protect your organisation without breaking productivity.

Table of contents

Conditional access essentials: how to safely transition policies from report-only to enforced mode

Why conditional access report-only mode matters?

Ring-based rollouts for conditional access policies

Monitoring conditional access policies

Using the ‘What If’ tool

Using per-policy ‘Policy Impact’ feature

Integrating with Log Analytics in Azure Monitor

Reporting, monitoring and KQL Queries

Workbooks and Conditional Access Gap Analyzer

Advanced monitoring with KQL queries

Conclusion

Conditional access essentials: how to safely transition policies from report-only to enforced mode

Why conditional access report-only mode matters?

Conditional access essentials start with understanding the risks of enforcing policies too quickly. Conditional Access is a powerful security tool, but it can unintentionally block users - especially when combined with Intune compliance policies that restrict access for non-compliant devices. Without adequate testing, enforcing a policy could prevent users from signing in, causing operational disruption and urgent remediation.

Privileged accounts require even more caution. A misconfigured policy could lock you out of your tenant, leaving you dependent on Microsoft Support to regain access. Thankfully, report-only mode in Entra ID allows organisations to evaluate conditional access policies safely, identifying potential issues before full enforcement.

Starting with report-only mode lets you monitor sign-ins, spot policy conflicts, and make informed adjustments. While some scenarios, like enforcing MFA for all users and apps, might require immediate enforcement, most organisations benefit from a staged, ring-based rollout to reduce risk.

Ring-based rollouts for conditional access policies

Personally, a ring-based rollout is my recommended strategy in conditional access. By gradually enforcing policies across user groups, you can minimise disruption and ensure operational continuity. Start with a small proof-of-concept group, such as IT staff, then expand incrementally while continuing to monitor report-only policies for all users.

Example: Implementing a policy to block legacy authentication.

• Step 1: Create a report-only policy scoped to all apps and all users, you can even include “Report Mode” in the name to distinguish it clearly.

• Step 2: Deploy an enforced policy targeting only IT team members, who are least likely to rely on legacy authentication.

• Step 3: Gradually expand the enforced policy to additional groups, using insights from the report-only policy to identify any exceptions or operational needs.

This approach lets you determine whether legacy authentication is genuinely required for certain apps. For instance, a legacy financial application or an older medical device may still need it. If modern authentication is supported, encourage users to migrate. If not, evaluate the business risk and consider temporary exceptions or an application upgrade.

Monitoring conditional access policies

Understanding sign-in logs is critical for any conditional access strategy. Report-only mode logs show what would have happened if policies were enforced, highlighting potential failures, unexpected blocks, or required exceptions. After deploying report-only policies, organisations should use the following tools to analyse the impact before moving to enforced mode:

• What If tool - simulate the effect of a policy on specific users or groups.

• Per-policy ‘Policy Impact’ feature - see which users and applications would be affected if a policy was enforced.

• Conditional access integration with log analytics in Azure Monitor - collect, query, and analyse logs for deeper insights.

• Kusto Query Language (KQL) (in general)- advanced queries allow you to investigate sign-ins, policy failures, and blocked access events

Using these tools provides visibility into policy impact, helping you refine conditional access settings and minimise operational disruption.

Using the ‘What If’ tool

What it is

The ‘What If’ tool is a safe way to test your Conditional Access policies. It predicts the outcome of a sign-in without enforcing the policy. Think of it as a crystal ball for conditional access.

How it works

The tool simulates a user login. It applies your existing policies to that simulated login. You can see if the login would be allowed or blocked. You can test for a single user or group. You don’t touch real accounts. No risk. No downtime.

Step-by-step guide

1. Go to Entra ID Portal: https://entra.microsoft.com

2. Navigate to Conditional Access > Policies > What If
   

   

3. Select a user or group to simulate

4. Choose applications the user is trying to access

5. Set conditions if needed, like location, device compliance, or client app
   

   

6. Click What If
   

   

7. Review results: you’ll see which policies would apply, if access would be granted or denied, and which controls (like MFA) would trigger. In additional, you can also review policies that will not apply too.

The ‘What If’ tool is a lifesaver. It doesn’t just help you tweak Conditional Access policies before they go live - it can stop you from accidentally locking someone out when policies are already enforced. Think of it as your preemptive “don’t break stuff” button.

Real-life scenario

You’re onboarding a vendor or contractor. They could be a guest or full member account. Before giving them access, you simulate their login with the ‘What If’ tool.

·         You know they’ll log in from a Windows device

·         You know their IP address or country

Run the simulation. The tool will tell you if a policy would block them. Maybe you forgot to exclude guest accounts from MFA. Maybe a location-based restriction trips them up. Fix it before they ring IT in a panic.

My favourite use case

I like to call it the “What makes me look good in front of my boss” tool.

Picture this: the CEO tells you their daughter or son is coming in for work experience. You’re responsible for setting up their account. You don’t want them emailing on day one, complaining they can’t access anything. Fire up the ‘What If’ tool, simulate their login, and make sure everything works. Hand over the credentials. You look like a hero.

Unless, of course, you have a crush on the CEO’s daughter/son and don’t mind looking incompetent in front of your boss for a chance of more interactions… whatever you do, don’t say I didn’t warn you! (Definitely not speaking from experience! :D )

Using per-policy ‘Policy Impact’ feature

What it is

The per-policy ‘Policy Impact’ feature shows the effect of an individual policy on your users. It’s different from report-only mode because it focuses on one policy at a time.

How it works

It analyses sign-in logs to calculate:

• Success: users who meet policy requirements

• Failure: users who would be blocked

• Not applied: users the policy does not affect

• User action required: users who would need to perform actions, like MFA

This feature doesn’t just help you fine-tune policies before enforcement. It can prevent accidental lockouts, even when policies are already live.

Example: blocking legacy authentication

Suppose you want to block legacy authentication across your tenant. You create a policy to enforce modern authentication, but you’re not sure which apps or users would be affected.

By using the per-policy ‘Policy Impact’ feature, you can see:

• Which users would be blocked from signing in using legacy auth.

• Which applications would fail authentication for certain users.

• Any exceptions that may need to be applied for critical business apps that don’t yet support modern auth.

For instance, a legacy financial application or an old X-ray machine in a medical environment may still rely on legacy authentication. Without checking the policy impact first, enforcing the block could disrupt essential services.

As we haven't set up any Log Analytics workspaces, let us manually check an impact of a particular policy.

Step-by-Step Guide

1. Go to Entra ID Portal

2. Navigate to Conditional Access > Policies

3. Select the policy you want to analyse

4. Click View policy impact
   

   

5. Review the summary dashboard

   • Success, Failure, Not applied, User action required
     

     

6. Drill down on specific users to see detailed results. Simply click anywhere on the graph and scroll down to see a list of items, or use the filters above the graph. Example: Apply a specific filter:

   

   Hover over specific sign-in activity over time and click on a specific date

   

   Then scroll down to see all events matching your filter where you have the ability to go directly to a specific sign-in log
   

   

7. Adjust the policy if needed before moving from report-only to enforced mode

Use this feature regularly. It gives visibility into who would be affected. It prevents surprises.

Integrating with Log Analytics in Azure Monitor

If you want to get serious about Conditional Access, you need visibility. That’s where integrating Entra ID with Log Analytics in Azure Monitor comes in. A Log Analytics workspace is essentially a central store for all your logs - Azure, non-Azure, apps, you name it. This is where we can monitor conditional access in real time, analyse trends, and spot issues before they become a nightmare.

Step 1: Create a Log Analytics workspace

Before we can stream logs, we need a workspace. You’ll need:

• An Azure account with an active subscription.

• Microsoft.OperationalInsights/workspaces/write permissions on the Resource Group (Log Analytics Contributor role works fine).

Here’s how:

1. Go to the Azure portal and search for Log Analytics Workspace.
   

   

2. Click Create.
   

   

3. Select your subscription.

4. Use an existing Resource Group or create a new one.

5. Give the workspace instance a unique name (e.g., EntraWorkspace).

6. Pick a region.

7. Click Review + Create, then Create.
   

   

Default pricing is Pay-as-you-go, so don’t panic - you won’t get charged until you start sending logs.

Step 2: Tagging and budget alerts

Tags help with billing visibility, e.g., Production or Testing. Optional, but useful.

Before connecting Entra ID, I strongly recommend setting up a budget alert. Log ingestion can get pricey if you’re not careful.

Create an action group for alerts:

1. Search for Monitoring section within your workspace in the portal.

2. Go to Alerts > Create + > Action Group.
   

   

3. Pick the same Resource Group as your workspace, give it a name, display name, and region.

4. Set up an email notification (or Logic App if you’re fancy).
   

   

5. Skip Actions and Tags unless you want extra automation.

6. Review and create.

Pro tip: If you hit an error, don’t panic. Often it’s a missing namespace. Go to Resource Providers, search, and Register it. Then try again.

Set the budget alert:

1. Go to Resource Groups > select your group > scroll to Budgets > Add budget.

2. Set a low amount to avoid surprises.

3. Attach the Action Group for notifications.

4. Hit Create.

Step 3: Connect Entra ID to Log Analytics

Once the workspace is ready, head to the Entra ID portal: entra.microsoft.com

1. Go to Entra > Monitoring & Health > Diagnostic Settings.

2. Click Add diagnostic setting.
   

   

3. Give it a permanent name (you can’t change this later).

4. Select the log categories you want to stream. For Conditional Access, Sign in Logs are a must. Audit Logs are optional.
   

   

5. Select your workspace and subscription.

6. Hit Save.

In case of any errors stating that "The subscription is not registered to use microsoft.insights" or similar, simply navigate to your subscriptions, select the one associated with your workspace, select resource providers, search for the specific provider you need to register, eg. microsoft.insights and select Register (or Re-register if it's been done before).

Logs may take 15–30 minutes to start appearing. Don’t panic if they don’t show up immediately.

Step 4: Why this matters

Streaming Entra ID logs to Log Analytics allows you to:

• Compare sign-ins with Microsoft Defender for Cloud.

• Troubleshoot app sign-in performance via Azure Application Insights.

• Analyse risky users and threat detections.

• Identify applications still using legacy authentication

• Evaluate and tune your Conditional Access policies.

Note: Integrating logs also automatically enables the Microsoft Entra data connector in Microsoft Sentinel.

Reporting, monitoring and KQL queries

Once your conditional access policies are in place and you've integrated your logs with a log analytics workspace you have additional options to see reports and some infographics representing policy impact in a bit more user friendly and more advanced way. Head over to Entra > Conditional Access > Insights and Reporting.

Here, the Impact Summary provides a quick breakdown of how your policies would affect sign-ins.

The interactive tiles show:

• Total - e.g. users in the last 7 days.

• Success - e.g. users granted access with required controls satisfied.

• Failure - e.g. users denied access because policy conditions weren’t met.

• Not Applied - e.g. sign-ins bypassing the policy because conditions weren’t met.

You can also filter the logs and see insights for any specific conditional access policy.

For example, if a login didn’t require MFA, but the policy would have forced it, the report-only logs let you see this without locking anyone out. This is crucial. You can’t just blanket enforce something for everyone without considering context.

Workbooks and Conditional Access Gap Analyzer

You can dig deeper using Workbooks in the Entra ID portal: Entra > Monitoring and Health > Workbooks. Here, the Conditional Access Gap Analyzer shows:

• Legacy Authentication

• Unprotected Applications

• Compromised User Sign-ins

• Unprotected Locations

• Unprotected Named Locations
  

  

This is especially useful if you’re phasing out legacy authentication or ensuring high-risk sign-ins are properly protected. Mistakes can happen, and as policies get more complex, Workbooks help you catch gaps before they bite you.

Advanced monitoring with KQL queries

Kusto Query Language (KQL) is a read-only query language used in Azure services (like Log Analytics, Azure Monitor, Sentinel, Application Insights) to search, filter, and analyse large amounts of log and telemetry data quickly.

It works a bit like SQL but is optimised for fast querying of time-based event data, such as sign-in logs, security events, performance metrics, etc.

With KQL you can:

• Filter and search logs

• Sort and group results

• Create custom reports and dashboards

• Detect trends or anomalies

• Investigate security events and Conditional Access impact

In short, KQL is the language you use in Azure to turn raw log data into useful insights.

If you want to play around with KQL queries whether you’ve never touched them or you use them every week, I recommend reading “The Definitive Guide to KQL” by Mark Morowczynski, Rod Trent and Matthew Zorich, which you can buy here. Additionally, you can always visit the GitHub page for all Kusto queries used in the book GitHub - KQLMSPress/definitive-guide-kql: Sample queries and data as part of the Microsoft Press book, The Definitive Guide to KQL.

Once your logs are streaming to Log Analytics, you can run KQL queries to investigate specific scenarios.

Examples of some useful KQL queries that are useful when investigating sign-in logs and conditional access policies, can be seen below: (credit to the book’s authors and Microsoft employees including Laura Hutchcroft, Krishna Venkit, Cosmin Guliman, Corissa Koopmans).

Failed sign-ins in the last 24 hours:

Sign-ins where Conditional Access succeeded:

Sign-ins where Conditional Access failed:

Using the project operator

First Conditional Access event per app in last 14 days:

Conditional Access failures for apps (pie chart):

Trend over time (30 days):

Failed Conditional Access events using external data (CSV example):

Advanced failure events with mv-apply:

Conclusion

By now, you should feel confident moving from report-only to enforced mode. Start slow, monitor often, and use the tools Entra ID provides: impact summary, workbooks, log analytics, and KQL queries. They give you the visibility to make informed decisions without accidentally locking out your users.

Remember: conditional access is powerful but unforgiving. Treat it like a sharp knife: use it carefully, keep your privileged accounts safe, and always double-check your exclusions. Once properly monitored and enforced, your policies will protect your tenant without crippling your users.