Penguin Health Cuts Audit False Positives 85% with Agentic AI on Amazon Bedrock
Swarm and BetterBrain rebuilt Penguin Health’s clinical documentation audit as an agentic workflow on Amazon Bedrock with Claude Sonnet 4.5, cutting false positives from 20% to 3% while keeping PHI in its own AWS account.
Fewer false positives, from 20% to 3% on the benchmark set
Of engineering time per new audit rule, down from ~2
Provider organizations the rules engine is built to serve
BetterBrain brings together expertise from top institutions and innovative companies, combining deep academic AI knowledge with practical industry experience.
About the client
Penguin Health is a U.S. behavioral health technology startup. Its software audits clinical progress notes against provider QA standards and Medicaid documentation and billing requirements.
The challenge
The audit pipeline already ran on AWS, but its reasoning layer relied on a separate hand-written prompt for every audit rule, sent to an external LLM. Each new rule took about 2 engineering hours. On the customer’s benchmark test set, the system wrongly flagged compliant notes 20% of the time, which created unnecessary reviewer work and eroded trust in the results.
The reasoning was contextual rather than keyword-based. The system had to tell an in-person contact from an incidental phone number, telehealth from payer-defined face-to-face care, and medication management from psychotherapy.
Penguin Health planned to grow from early pilots to more than 100 provider organizations, each with its own terminology, billing rules and QA policies, so engineering effort would have grown with every provider it onboarded. The external LLM also put a third-party vendor in the path of protected health information for a HIPAA-regulated workload.
The solution
Swarm engaged BetterBrain as the specialist Hive team to redesign the audit product as a generalized, agentic workflow in Penguin Health’s HIPAA-compliant AWS account in us-west-2. Swarm and BetterBrain benchmarked candidate foundation models on the customer’s own adjudicated cases. They selected Anthropic Claude Sonnet 4.5 on Amazon Bedrock, which outperformed GPT-OSS on nuanced clinical and billing distinctions and is the only model in production.
Provider documents arrive through an SFTP ingestion service. Amazon Textract extracts their text, and Amazon S3 stores both the source documents and the extracted text. A generalized rules engine on AWS Lambda uses custom orchestration rather than an agent framework. It reads each rule from Amazon DynamoDB, and it retrieves external regulatory, billing or QA reference material through Amazon Kendra only when a rule needs it. It then calls Claude Sonnet 4.5 in two steps: first to extract the facts the rule requires, then to check the rule against those facts and the full note. Each evaluation returns a structured PASS, FAIL or NEEDS_REVIEW verdict with evidence-based reasoning, which is stored in DynamoDB. Uncertain cases go to a human reviewer instead of being forced into a decision.
Rules are now stored as data. A customer writes a rule in plain English, an asynchronous Lambda workflow uses Bedrock to generate the fields to extract, and a human approves them before the rule goes live. Reviewer corrections are linked to the run, document and rule. Bedrock turns them into short clarifications that are stored with the rule and reused on later runs, so the system improves without fine-tuning or retraining.
The external LLM was removed from every production path, and PHI stays in the customer’s account and out of logs. Swarm worked through scoped, session-based IAM access with isolated development resources. Infrastructure and permission changes are submitted as AWS CDK pull requests that the customer reviews.
Impact
The solution passed customer acceptance and has run in production since the final milestone on March 11, 2026. On the customer’s adjudicated benchmark test set, the false-positive rate fell from 20% to 3%, an 85% reduction that beat the 5% target.
Engineering effort for each new routine audit rule fell from about 2 hours to zero, because Penguin Health now creates rules itself. Differences between providers and payers are handled as configuration instead of new prompts or code.
Raising the Amazon Bedrock quota from 10 to 10,000 requests per minute allowed one focused evaluation per rule, instead of bundling several rules into one prompt, which had reduced accuracy. The workload processes about 100 rule evaluations a day on serverless and managed AWS services, with no self-managed servers or model hosting. Removing the external LLM took a third-party vendor out of the PHI path.
