Blog

Originally published: 13 Nov, 2026 Last updated: 13 Nov, 2026 Testing an AI agent means checking the actions it takes, not only the answers it gives. An agent picks a tool, passes arguments, chains steps together and changes real records. So the questions are which tool it called, with what, and what happened when a step failed halfway through. This guide is about testing AI agents as products. It is not about using AI agents to run tests, which is a separate topic we cover elsewhere. The worked example throughout is a customer support agent that creates tickets, updates customer records and calls internal APIs. What is AI agent testing? AI agent testing verifies the actions an agent takes inside your systems. It covers tool selection, the arguments passed, and whether multi-step tasks complete in the right order. It also checks that the agent stays inside the permissions of the user it acts for. The difference from testing a chatbot is consequence. A chatbot that answer...

Originally published: 5 Nov, 2026 Last updated: 6 Nov, 2026 An AI feature needs its own release gate because the usual one cannot see most of its failure modes. A wrong answer returns HTTP 200, a leaked system prompt renders as normal text, and an invented fact passes every assertion you wrote. This checklist covers 29 checks across 5 layers, which issues block a release, and what to re-run when a model or prompt changes. It is written for CTOs, product managers, and QA leads deciding whether a feature is ready to ship. QA Madness provides independent software testing services for SaaS companies and AI product teams across the UK, Europe, and North America. Why does an AI feature need its own release gate? Because a passing test suite tells you the feature runs, not that it is correct. Traditional gates check status codes, schemas and user flows, and an LLM feature can satisfy all 3 while telling a customer something false. The second reason is that the artifact you ship...

Originally published: Oct 30, 2026 Last updated: Nov 2, 2026 A RAG application retrieves documents before it answers, so it has 2 ways to fail that a plain chatbot does not. It can retrieve the wrong source, or it can ignore the right one. According to Stanford HAI's 2026 AI Index Report, hallucination rates across 26 evaluated models range from 22% to 94%, and retrieval alone does not close that gap. This guide gives you a reusable test case table, a failure mode reference and a finding format, drawn from how we run LLM testing engagements. It is written for QA engineers and product managers shipping a retrieval-based assistant. What this guide covers. Test design for the retrieval pipeline: retriever accuracy, context assembly, grounding, citations, stale sources and multi-turn conversations. It does not cover general AI application testing, model selection or how to build a RAG pipeline. The worked example throughout is a SaaS help-centre assistant answering product questio...

Last updated: 16 October 2026 LLM evaluation frameworks turn to a subjective question — is this answer any good? — into a repeatable score you can run in CI. DeepEval, RAGAS and Promptfoo all do this with built-in metrics plus a model acting as judge, and all 3 are open source. None can tell you whether an answer is safe to ship in your domain, which is where AI product testing, a human reviewer, and how testing AI applications differs from traditional QA come in. What does an LLM evaluation framework actually do? An LLM evaluation framework runs your application against a set of test inputs and scores each output on defined metrics. It fails the run when a score drops below a threshold. It replaces manual spot-checking with something a pipeline can gate on. LLM evaluation is the practice of measuring output quality against defined criteria rather than eyeballing samples. The service version of that work is LLM evaluation services. 3 mechanisms do the scoring. Statis...

Last updated: 9 October 2026 This guide explains how to measure and reduce test automation debt, using suite health metrics with stated formulas, a decision table for fixing, quarantining, refactoring or deleting a test, and a practical approach to flaky tests in CI/CD. It is written for automation QA engineers, QA leads, engineering managers and CTOs whose suite has grown large enough that people have started ignoring it. Figures come from Google's published flakiness research, CircleCI platform telemetry and the World Quality Report 2025-26. A test suite does not fail all at once. It degrades. Runs get slower, a few tests start failing at random, somebody adds a retry, and within 2 quarters the team has learned to glance at a red build and merge anyway. At that point the suite still costs money to run and maintain, and it no longer stops anything from shipping. The pipeline data says this is getting worse. CircleCI's 2026 State of Software Delivery, based on 28,738,317 workf...

Last updated: 8 October 2026 This guide compares 10 companies that build and run automated test suites for other teams, and shows which one fits which kind of engineering organisation. Each supplier is assessed on core frameworks, CI/CD integration, who owns the test code, how maintenance is priced, and performance testing capability. The guide includes a comparison scorecard, a framework ownership checklist, and a map from automation maturity to the right engagement shape. It is written for CTOs, VPs of Engineering, QA leads, DevOps engineers and automation managers choosing an external automation partner. Most automation engagements fail in year 2, not year 1. The supplier builds the suite and it runs. Then the framework turns out to be proprietary, the maintenance sits outside the original quote, and nobody on your side can run the tests without them. You bought test coverage. You acquired a dependency. The research says this is where the market sits. The World Quality Repo...

Last updated: 21 October 2026 This guide compares 10 companies that sell manual software testing, and shows which one fits which kind of team. It assesses each on exploratory testing depth, test documentation, web and mobile coverage, and how quickly a team scales, with a comparison scorecard, a breakdown of when manual QA beats automation, and the evidence to request before you sign. Companies were assessed on published capability and review platform data checked in September and October 2026. It is written for product managers, QA leads, CTOs, engineering managers and startup founders choosing a manual QA partner. Manual testing is the part of QA that automation keeps failing to absorb. A script checks what you told it to check. A tester notices that the confirmation email arrives in the wrong language, that the error message blames the user for a server fault, and that the flow works but feels broken. The 2026 research points the same way. The World Quality Report 2025-26 s...

Last updated: 2 October 2026 This guide explains how to build a performance testing programme for a SaaS product, from the first baseline to a pipeline that blocks a bad release automatically. It covers the 4 test types a SaaS team needs, which checks belong at each delivery stage, how to turn a performance budget into a CI/CD gate, and how to choose between k6 and JMeter. It is written for QA leads, DevOps engineers, engineering managers and CTOs who have run occasional load tests and now want a repeatable programme. Examples and thresholds come from official Grafana k6, Apache JMeter and Google documentation. Most teams do not start performance testing. They start load tests. Someone runs JMeter the week before a big launch, the numbers look alarming, everyone works late, and the script is never run again. That is a fire drill, and it tells you almost nothing about whether the next release is safe. A programme is different because it runs on a schedule, gates release automat...

Last updated: 29 September 2026 This guide compares ten QA companies that test gaming apps, and shows which one fits which kind of studio. It covers platform coverage, live operations support, localisation testing and console certification, with a comparison scorecard, a breakdown by game type, and the questions to ask a vendor before you sign. Companies were assessed on published capability, evidenced game work and platform reach, using only claims each vendor makes on its own site or on a review platform. It is written for game producers, studio founders, CTOs, QA leads and LiveOps managers choosing an outsourced QA partner in 2026. Picking a QA partner for a game is not the same job as picking one for an app. A game has to hold a frame rate on a five-year-old Android handset, survive a live event that doubles concurrent players overnight, and pass a platform holder's certification checklist before anyone can download it. Most general software testing firms are not set up for a...