AI Workflow Reliability Review
What can your AI workflow get wrong while still looking successful?
A plausible answer can hide a missing source, a failed handoff, or an action the evidence never justified. For a founder-led MSP or boutique IT consultancy, that can mean repeated troubleshooting, misrouted work, or advice your team cannot substantiate.
Counterproof investigates one AI-assisted workflow so you can decide whether to expand its use, constrain it, repair it, re-test it, or stop it.
One workflow. 10 business days. $4,800 fixed fee.
When a review is useful
You have a working pilot or an established workflow, and a concrete decision to make. Perhaps outputs vary without an obvious cause, operators rely on undocumented workarounds, or a change to the model, prompt, retrieval source, or integration has made earlier checks inadequate.
Typical workflows include support triage, incident summaries, document extraction, runbook assistance, retrieval-assisted answers, and routing between tools.
What Counterproof investigates
We trace the actual path from input to consequential output or handoff: prompts, models, retrieved material, tools, permissions, and the people reviewing or acting on the result.
Together, we agree which claims and failure hypotheses matter. We then exercise the workflow under recorded conditions, try to reproduce consequential failures, and examine whether existing controls expose or contain them.
Each conclusion is labeled as observed, inferred, client-reported, or untested. Contradictions, failed interventions, and missing evidence stay in the record. If a suspected failure cannot be reproduced, we report that bounded result.
The outcome is a claim-to-evidence record: what the reviewed workflow supports, what the evidence contradicts, and what remains unknown.
What you receive
- Reliability Evidence Brief: the review boundary, tested claims, material observations, limitations, and a concise decision note.
- As-built workflow and control map: the reviewed path, human decisions, fallbacks, permissions, logging, and points where evidence is lost.
- Case-level test ledger: expected and observed behavior, recorded run conditions, and supporting artifacts.
- Prioritized findings: evidence, confidence, plausible operational consequences, and the smallest justified correction, containment control, or next test.
- Known/unknown ledger and re-test plan: unresolved questions and checks to use after changes.
- 75-minute working readout: discussion with your workflow owner and relevant stakeholders.
The brief is portable; case and finding ledgers are editable.
A defined engagement
The standard review covers one named workflow version, one primary use case and user group, and one test environment or supervised execution path. It includes the current configuration and, when directly comparable, one alternative.
Testing usually covers 20–40 agreed case definitions, with selected repeat runs where variability matters. Case complexity, access, and third-party costs determine the final mix and execution count. We agree the boundary and evaluation criteria in writing before testing. Untested areas remain explicitly untested.
This is a bounded investigation, not certification or a guarantee of future reliability. It does not establish population-wide performance. Legal or compliance assessments, penetration testing, production deployment, implementation, model training, and ongoing monitoring are outside the standard review.
What we need from you
A workflow owner, a decision owner, and at least one regular operator; a 60-minute walkthrough; current configuration artifacts; de-identified examples and available logs; and a safe way to exercise the workflow. You also designate someone who can resolve questions about expected behavior.
If you lack a usable test set, we can build a small exploratory set from your examples and agreed failure hypotheses. These probes are labeled separately and do not establish general performance.
Remove or replace sensitive data unless separate safeguards are agreed. Production credentials are not required by default.
Timing and fee
$4,800 fixed fee. Fifty percent reserves the engagement; fifty percent is due on delivery.
Delivery is scheduled for 10 business days once kickoff is complete and agreed inputs and access are available. The fee includes bounded testing, the complete review package, a clarification checkpoint, the readout, and one consolidated factual-correction round returned within three business days.
The schedule pauses if required access, inputs, or client decisions are unavailable. Material workflow changes may require revised scope, timing, or fees. Additional workflows, implementation, and later re-testing are quoted separately. Client-specific vendor, model, API, travel, or environment costs are extra and require written approval.
Bring one workflow and one decision
Email hello@counterproofconsulting.com with a short description of what enters the workflow, what comes out, what concerns you, and the decision you need to make.
Include any decision deadline and the kinds of de-identified examples or test access available. Keep sensitive records and credentials out of the initial enquiry.