Credence on Guy Freeman

The Ablation That Beat the Agent

Mon, 03 Aug 2026 00:00:00 +0000

In April I wrote, on this site, that a paper was being prepared for arXiv: Credence: A Bayesian Decision-Theoretic Framework for LLM Agent Tool Selection. It was going to argue that choosing which tool to query is a decision problem under uncertainty, and that solving it as one beats solving it as a language-generation problem. I had the number to prove it: the Bayesian agent scored +129.5 against the best LLM variant’s +10.8, a factor of twelve, while being less accurate on the questions it attempted — 62.6% against 76.4%.

Ninety-Six Percent Cheaper and Slightly Better

Thu, 30 Apr 2026 00:00:00 +0000

The production question about LLM agents, once you have gotten past whether they work at all, is how much they cost. A customer-service agent that answers well but costs eight cents per turn is not a customer-service agent; it is a charity. The conventional answer is to pick a cheaper model and hope it is good enough. The less conventional answer is to treat model selection as a decision problem.

Sixty-Two Percent Correct and Winning by a Hundred and Twenty Points

Thu, 30 Apr 2026 00:00:00 +0000

The standard way to evaluate a question-answering system is to measure how often it gets the right answer. This seems reasonable. It is, in practice, a trap.

I ran an experiment to demonstrate why. A Bayesian decision-theoretic agent — built on the Credence DSL, using Beta-Bernoulli reliability tracking and value-of-information calculations — competed against several LLM agents on a 50-question benchmark. All had access to the same four tools. All faced the same questions. All were scored on the same objective.

The Agent That Invents Its Own Rules

Tue, 28 Apr 2026 00:00:00 +0000

The previous post in this series described what I called Tier 1 of the Credence architecture: a DSL for Bayesian decision agents with three types, four axioms, and a constitution forbidding everything else. That post ended with a program the user had to write by hand — a short S-expression encoding a hypothesis about what the environment was like and how to act in it.

Hand-written programs have a well-known limitation: they are only as good as whoever wrote them.

The Prompting Gradient

Thu, 30 Apr 2026 00:00:00 +0000

The accuracy paradox post reported the headline: a Bayesian agent scoring +129.5 against an LLM agent’s +10.8, despite lower accuracy. This post is about the LLM side of that experiment — what was tried, what helped, and where the ceiling is.

The Three Variants

Three LLM agents were tested on the same 50-question benchmark. They differed only in prompting:

LLM Bare. The model receives a description of the four available tools, the scoring system (+10 correct, -5 wrong, 0 abstain, minus tool costs), and the current question with its four candidate answers. No guidance on how to decide. No reasoning format imposed. The model chooses a tool, receives a response, and decides what to do next.

Agentic AI Is Neither Intelligent Nor an Agent

Mon, 23 Feb 2026 00:00:00 +0000

I’ve spent the last few months building agents that maintain actual beliefs and update them from evidence — first a Bayesian learner that teaches itself which foods are safe, then an evolutionary system that discovers its own cognitive architecture. The experience has given me a certain clarity about the industry’s use of the word “agent,” in much the same way that learning to cook gives you clarity about airline food.

What would it take for an AI system to genuinely deserve the word?

How Decision Theory Cuts Your AI Agent's API Bill in Half

Mon, 23 Feb 2026 00:00:00 +0000

The companion essay argued that LLM-based “agents” don’t earn the title. No beliefs, no uncertainty quantification, no principled mechanism for deciding whether a tool query justifies its cost. This post supplies the technical scaffolding for that claim — the mathematics and code behind Credence, the benchmark I built to test it. Think of it as the receipts.

For the philosophical argument, see Agentic AI Is Neither Intelligent Nor an Agent.

The Problem: Every Query Has a Price

Hand a standard LangChain ReAct agent a question and four tools, and it will query most of them most of the time. It possesses no apparatus for reasoning about whether the next query repays its cost. The prompt says “be helpful”; the agent takes helpfulness to mean exhaustiveness.