An endpoint detection and response rollout is planned like a software deployment and run like a change management programme. The agent installs in minutes. The schedule is made of the decisions that come after it, about how aggressively the agent acts, on which machines, and in what order. The rollout is finished when the organisation is willing to let the tool block on a production server, and the weeks in the plan are the weeks it takes to get there.
The product here is CrowdStrike Falcon, deployed across an enterprise Windows estate. The sequence holds for any endpoint agent that blocks as well as detects.
In my experience leading teams that deliver managed detection and response, the rollouts that slip are the ones with no agreed order of work. Agents are installed early because installation shows progress against the plan, policy follows later, and the estate spends weeks in a state nobody can describe.
Four phases, each gated by the last
Plan, configure, deploy, protect. Rollouts go wrong where those four are compressed or reordered. The common version is agents pushed out first, for the progress it shows on a status report, with policy configuration behind them, which leaves hundreds of sensors running in a state nobody has defined.
Each phase boundary is a gate. The question at the gate is whether the next phase could be left to run unattended overnight, which is a harder test than whether the tasks are ticked off.
Plan: everything that happens before installation
Vendor onboarding, deployment guides and platform training are worth the calendar time they take, and the time to use them is before the first incident. Learning the console during an incident costs money, and has cost people their jobs.
The decision that shapes this phase is who owns the console day to day. That person holds policy decisions, exclusions and alert triage. Naming someone who already has a full workload produces a rollout that completes on schedule while its policy decisions queue behind that person's existing job.
Four roles carry the work. The console owner is one; the others are a systems administrator who builds and pushes the deployment package through existing tooling, a security team that consumes the alerts and owns the response, and a sponsor who can authorise a change window and absorb the escalations when something breaks.
Estate discovery sets the date
A plan assumes the estate is known. In most organisations that picture is a best effort, and the gaps surface during deployment. Discovery is the least interesting work in the project, which is why it gets skipped, and five categories account for most of what it finds.
- Hosts running an operating system the agent does not support. They do not disappear because they are inconvenient to the schedule.
- Endpoint protection that is already installed. Two agents competing for the same hooks (the low-level points in the operating system where security products attach themselves) is an outage risk. Removing the incumbent, or running both where the organisation has decided it wants both, belongs in the rollout plan rather than in the first week of deployment.
- Hosts nobody owns. Every estate has them, and they are found during deployment rather than during planning.
- Hosts that cannot take change easily (in practice, the systems where an hour offline is measured in revenue).
- Hosts that are rarely online. Laptops that travel, virtual machines powered off for weeks and equipment with long maintenance windows take the agent and every later policy change late; any completion figure is a moving one.
Configure: policies before packages
Prevention policies and sensor update policies are built before any agent reaches production, and tested in a lab where one exists. An agent with no policy attached collects nothing and blocks nothing, so hosts installed ahead of their policy have to be found and corrected afterwards.
A sensor update policy governs which version of the sensor a group of hosts runs. CrowdStrike's own account of the July 2024 incident describes the control; customers select which parts of the fleet install the latest sensor release, one version older, or two versions older. That control exists whether or not anyone sets it, and the rollout plan is where the decision belongs.
Policies attach to groups of hosts, which makes grouping part of configuration. CrowdStrike's documentation for protecting volume shadow copies (the point-in-time copies Windows keeps so files and systems can be restored) recommends audit mode with a test group, allowlisting trusted software before protection is turned on. That guidance assumes the test group already exists. Building groups while the population is small costs less than retrofitting them across a deployed estate.
Settings also depend on each other. Protection against suspicious processes deleting those shadow copies requires the corresponding audit setting to be on first. A policy assembled at speed can show that protection as enabled while the audit setting it depends on is off.
Deploy: non-production first
Deployment starts with test workstations and test servers. In the plans I have worked on, the test window is the longest single line item and the first thing compressed when the date moves, and that compression is where the risk enters the project.
An install check confirms the agent is present, and that is all it confirms. The window exists to watch the sensor while the business runs its own applications on its own builds (a finance team closing a period, a backup job touching shadow copies). Detections raised by legitimate work are what the exclusion list is built from. The ones nobody watches for in this window surface on a production host instead.
Protect: enforcement rises in stages
Enforcement rises in steps, and the method comes from the vendor rather than from local caution. CrowdStrike's policy documentation recommends testing individual settings against critical applications before full deployment, and, for shadow copy protection, audit mode against a test group before protection goes on.
Each environment completes its own ramp before the next one starts (non-production, then production workstations, then production servers). Servers come last because the cost of a false positive rises with the host. On a laptop it is a support ticket. On a domain controller it is an outage.
The update that no version policy stopped
On July 19, 2024, at 04:09 UTC, CrowdStrike released a content configuration update for the Windows sensor that crashed Windows systems. The hosts in scope were Windows machines running sensor version 7.11 and above that were online during the 78 minutes before it was withdrawn; Mac and Linux hosts were unaffected. Microsoft estimated 8.5 million Windows devices affected (under one percent of all Windows machines).
Sensor update policies govern sensor versions, and this was not a sensor version. Rapid Response Content is configuration data delivered from the cloud, and it is not code or a kernel driver. Every online host on a supported sensor took it, including the hosts their owners had deliberately held two versions back.
CrowdStrike's root cause analysis describes a template that defined 21 input fields while the code supplying it provided 20, a mismatch that passed several layers of validation and shipped successfully more than once because earlier versions of the file wildcarded the field that was never supplied. CrowdStrike has since committed to granular control over when and where Rapid Response Content updates are deployed.
Microsoft addressed its response to the whole industry, asking for safe deployment and disaster recovery practices and, from security vendors, minimal kernel-mode components so that update handling sits in user mode, where a failure can be recovered. For a rollout plan the consequence is narrower. Update control has two parts. One is the sensor version policy; the other is the content channel that policy does not reach, where the available controls are whatever the vendor has built since.
What the plan does not tell you
A deployment plan is a task list. The things that decide whether the rollout lands rarely appear as rows on it.
- Exclusions take longer than the plan allows for. Line-of-business software behaves in ways security tooling finds suspicious, and each exclusion is a permanent hole with a reason attached to it. That reason is what an auditor asks for two years later.
- The incident response runbook changes on the day prevention goes on, with new alert sources and containment options that did not exist before. Rewriting it during the rollout costs less than rewriting it during the first real detection.
- Whoever receives the three in the morning alert needs the authority to act on it before prevention is enabled anywhere. Without that authority the estate produces alerts nobody can answer.
- Every step of the ramp needs a way back that has been tested. A group that can be moved down a level in minutes is a group the team is willing to move up.
What I would tell the sponsor
A sponsor eventually asks why an install measured in minutes turns into a schedule measured in weeks. My answer is that the schedule measures confidence, and confidence is bought in the test window and the ramp. Rollouts that keep both end with a security team that no longer asks permission before the tool blocks something on a production server.