An automated test should never change your compliance status.

That sounds like a strange thing for a platform company to say out loud, especially in the same quarter we shipped continuous monitoring. We’re live with 15 integrations, 155 evidence collectors, and 386 automated tests on a 12-hour cadence — pulling from AWS, Azure, GCP, Microsoft 365, Google Workspace, Okta, Intune, Jamf, Kandji, ConnectWise, CrowdStrike, Qualys, Tenable, and Enclave. Then we deliberately capped what all of that machinery is permitted to do.

It cannot mark a control as met.

Here’s the reasoning, because the reasoning is the actual product decision.

Three layers that shouldn’t collapse

There are three distinct things happening when a platform connects to your environment, and most tools blur them together.

Evidence is a raw snapshot. Your S3 bucket policy at 06:00. Your Entra ID conditional access rules. A list of endpoints with disk encryption on or off. Evidence makes no judgment. It’s a fact about a system at a moment in time.

Tests are pass/fail observations against that evidence. Is MFA enforced for privileged accounts? Is logging enabled on the storage account? A test looks at the evidence and returns a verdict about that specific, narrow question.

Control assessment is a human decision about whether your organization satisfies a requirement.

The first two are mechanical. The third is not, and pretending otherwise is where compliance programs go wrong.

In our platform, tests are informational. They surface, they rank, they tell you what changed. They never move a control’s assessment status. A human does that, with the test results in front of them.

Why the gap matters

A passing cloud test tells you that the thing it looked at is configured correctly. It tells you nothing whatsoever about the things it didn’t look at.

Consider what an API doesn’t reach. The on-premises Active Directory forest that was never connected. Physical security at the colo. The penetration test that’s been deferred two quarters running. Vendor contracts with no security exhibit. The offboarding process that lives in one person’s head. The incident response plan that exists as a PDF nobody has exercised.

Our cloud and identity connectors auto-evidence 57% of our assessment library — measured against 289 questions, cloud and identity providers only, as of July 2026. Endpoint, EDR, and vulnerability management add coverage on top of that. We haven’t recomputed the all-in figure, so I’m not going to quote one.

But hold that 57% for a second, because the other 43% is the whole argument.

If a platform lets automated tests set control status, then the moment your connectors go green, your program reports as healthy. Everything in that remaining 43% — every process, every contract, every untested plan — is silently scored as satisfied by a system that never looked at it. Not because anyone lied. Because the architecture made a claim it had no standing to make.

I’ve sat in enough audits to know exactly how that plays out. The platform said green. The auditor asked one question about something outside the connector’s field of view. And now the CISO is explaining to a room why their GRC tool reported a posture it couldn’t actually see.

That’s not an automation failure. That’s a design failure, and it’s avoidable.

The vCISO version of this problem

If you’re a service provider running compliance across dozens of client environments, this compounds fast.

At one client, an over-confident dashboard is a nuisance you’ll catch. At sixty clients, it’s your operating model. You are relying on the platform’s summary because you cannot personally inspect sixty environments every week. If the summary is structurally optimistic — if green means “the connected 57% is fine” but renders as “this client is compliant” — you will find out at the assessment, or worse, at the incident.

The inverse design is better for the business, not just for accuracy. When tests stay informational, the platform’s job becomes surfacing what changed and ranking it by impact, and your team’s job becomes judgment. That’s the split that scales. Automate the 57% so your consultants spend their billable hours on the 43% that requires a human — the tabletop exercises, the vendor reviews, the board conversation.

That 43% is also, not coincidentally, the work clients remember at renewal. Nobody renews a retainer because their evidence collector ran on schedule.

What to ask a vendor

If you’re evaluating any platform with continuous monitoring, three questions separate the serious ones:

Can an automated test change a control’s assessment status? If yes, ask them to walk you through what happens when a control depends on something the connector cannot reach. Listen for whether they’ve thought about it or whether it’s an edge case to them.

What is the collection cadence, and what happens when evidence goes stale? Collection on a schedule is the baseline. Whether the system does anything when evidence ages out is the differentiator. Most let it expire silently.

How is the connection itself secured? Read-only access — enforced or promised? Where do credentials sit at rest, and when are they decrypted? For AWS, is it role assumption with an external ID, or a stored long-lived key? Is there a fixed egress IP you can allowlist, and how much notice before it changes?

For the record: ours is read-only on every integration, envelope-encrypted credentials decrypted in memory only at sync time, AWS via IAM role assumption with an external ID and no stored key, fixed egress IP with a 30-day change notice.

The point

Automation should widen what you can see. It should not quietly narrow what you’re accountable for.

Continuous monitoring is genuinely the right investment — it collapses the assessment work that shouldn’t require a human and it catches drift you’d otherwise find a year late. But the value is in what it shows you, not in what it decides for you.

If your compliance platform moves a control to “met” because an API call came back clean, you don’t have assurance. You have a dashboard.moves a control to “met” because an API call came back clean, you don’t have assurance. You have a dashboard.