Huh. It mostly looks normal, except there’s a few points perfectly aligned with what looks to be a “fitted” line. You dig into it a bit more, and realize that the rows from one study site have their cholesterol values imputed. You set them to NA and go along your way.

Would today’s frontier LLMs catch such an oddity? We designed an LLM evaluation to help us answer this question. As it turns out, LLMs mostly miss these sorts of artifacts:

During exploratory or open-ended data analysis, Posit assistant “only runs a few bits of code at a time, then summarizes what it found and suggests next steps”. This is motivated by our stance that, for now, a data scientist should mostly keep pace with and understand what the agent is doing when analyzing data. This stance was initially informed by our observation that last year’s frontier models tended to see what they expected to see when visualizing data. While LLMs have since become much better at interpreting counterintuitive plots, bluffbench2 shows they still lag behind human data scientists in interpreting data visualizations. As such, we are still cautious on the prospect of highly autonomous data agents.

How the eval works

The eval harness is a relatively generic coding agent harness, similar to that of Claude Code or Posit Assistant. The agent has a tool to run R code in a persistent REPL and some vague prompting about data analysis:

You are an AI assistant embedded in the user’s data science IDE. You can read and modify files in the user’s workspace and execute R code in their active session, including rendering plots. Prioritize correctness and clear communication…

In each sample, the agent first carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Short user messages like “load in the csv in this folder” are decorated with “System Reminders” and other noise like that injected by popular agent harnesses.

After a few turns, the agent is asked to produce a data visualization that includes a subtle visual artifact that could feasibly result from a real data-generating process. The artifacts span a range of realistic data quality issues: stuck sensors, bad joins, points imputed onto a line, swapped columns, pseudoreplication, differing units, etc.

plot bmi vs cholesterol

Tool: Run R code

If the agent mentions the artifact in its follow-up response, it receives a full point. If the agent does not mention the artifact, it can also receive a half point by mentioning it in response to a follow-up user message along the lines of “what do you see in the plot?” If the agent never mentions the artifact, it is graded as incorrect.

Designing the eval

Once we understood the mechanism behind bluffbench, implementing the eval was relatively straightforward. bluffbench demonstrates the degree to which an LLM will ignore evidence shown in a plot in favor of its expectations. So, to implement a given sample, we’d just think of some situation that would elicit a strong prior and then subvert it. For example, a dataset called doug_firs with variables height and circumference; one might expect that, as height increases, so does circumference. So, instead, we did a transformation under the hood that made the relationship parabolic.

A year ago, triggering this prior was enough to frequently ’trick’ the current frontier LLMs.

Slipping a plotted artifact past today’s LLMs is much harder. Any human could ace bluffbench, but only an attentive data analyst would excel at bluffbench2.

In our early work on a successor to bluffbench, we started off with trying to elicit priors in the same way as bluffbench did, but in more realistic, longer-context scenarios. We were surprised to find that the same mechanism broadly doesn’t seem to trick today’s models even in these more realistic settings.¹ We then tried a ‘reverse bluffbench’, where we let the model being evaluated in on the trick, asking it to carry out the transformation itself and then look at the plotted result which was tampered with to show the original relationship. We anticipated that this stronger prior (“I did a thing with an obvious effect”) might cause the models to miss the (re)manipulation, but models reliably noted that the plot looked as if it hadn’t been manipulated.

As such, there isn’t a similar ’trick’ in bluffbench2 per se. The transcripts read like relatively normal data analysis sessions and the plotted artifacts are designed to plausibly result from real data-generating processes. Instead, the eval elicits 1) the ‘shape’ of LLMs’ vision being different than humans’ and 2) the model’s tendencies to perform progress, simulating a data analysis moving along smoothly.

Today’s frontier models are in the mid-teens at best; the top scores belong to Claude Fable 5 and Gemini 3.5 Flash at 16%. That said, we’d caution folks from interpreting the current scores on this eval as ‘LLMs don’t see plots well.’ The plotted artifacts are actually quite subtle, and when they’re made even a bit more marked, models tend to call them out consistently.

For example, the previous version of the scatterplot had a slightly more dense cluster of points:

• Previous
• Current

Opus 4.8 (medium) consistently got this sample right in the previous iteration.

Exploring the eval’s results

At least for now, there’s a loosely linear relationship between the cost to run the eval and the resulting score:

One of the most interesting learnings from examining the logs is a behavioral one. Even though we never request that LLMs introduce modeled results to plots, like fitted lines and confidence intervals with geom_smooth(method = "lm", se = TRUE), they sometimes do so anyway. For example:

In general, adding modeled results to data visualizations without first looking at data is bad practice; it makes it hard to see the data itself. In the eval, adding a modeled result like this seems to substantially lower the chances that the model will notice the plotted artifact:

More bluffbench

If you’d like to learn more about the bluffbench set of evals, take a look at these past posts:

• Introducing bluffbench: Writeup of the the original eval.
• LLMs interpret plots well, until expectations interfere: In-depth post on why models at the time didn’t perform well on bluffbench, as well as various interventions we tried to improve performance.
• LLMs are getting much better at interpreting counterintuitive plots: In spring 2026, bluffbench scores suddenly jumped as models improved.
• It’s (still) very bad to be wrong: Slides from our recent SciPy 2026 talk on building agents for correct, transparent, and reproducible data analysis in light of the bluffbench results.

There are a variety of ways to give a coding agent new information or abilities, but it can be confusing when to use each one. Generally, with coding agents like Claude Code or Posit Assistant, you can:

• Write prompts in the chat (i.e., normal usage).
• Add information to either a project-level or user-level CLAUDE.md or AGENTS.md. The contents of a directory’s CLAUDE.md or AGENTS.md are included in the agent’s system prompt in every session in that directory. You can also create user-level versions that apply to every session.
• Write a skill or use an existing one. Skills are packaged instructions that can include both text and code. The agent loads a skill only when it’s relevant.
• Add an MCP server. An MCP (Model Context Protocol) server provides an agent with access to otherwise hard-to-find context, mostly through tools, using a standardized interface.

This list is roughly ordered from most straightforward to most complicated.¹

So when do you use one over the other? There are two axes that might matter for your decision.

The first axis is reusability: do you want the agent to perform the task or access the information just once, or many times? The more often you or others will reuse something, the more it’s worth encoding somewhere more permanent.

The second axis is reach. Prompting, CLAUDE.md/AGENTS.md files, and skills all provide guidance on how to best make use of the existing context and tools. MCP servers can provide the agent with entirely new tools (in the agent tools sense), granting the agent access to hard-to-reach information.

MCP servers often seem like the solution when you need to grant an agent access to an outside system, but they aren’t always necessary. In many cases, what seems like a task for an MCP server can actually be solved by a command-line interface (CLI) tool, or a CLI plus a skill that tells the agent how to use it. For example, GitHub has an MCP server, but the gh CLI does roughly the same thing.

The GitHub MCP server works by providing the agent with new tools, whereas the gh CLI takes advantage of the agent’s existing bash tool that lets it run arbitrary shell commands. The skill plus CLI option is therefore generally preferable from a simplicity standpoint, but also from a token standpoint: adding the GitHub MCP server would inject tens of thousands of tokens of tool definitions into every request, whereas a skill that tells the agent to use gh costs almost nothing and loads only when it’s relevant.

However, some information sources are hard to reach via the command line, because no CLI exists, the one that does isn’t fully featured, or it’s difficult for the agent to use. In some of these cases, the same sources can be accessed more effectively via MCP servers, such as design tools like Figma, knowledge repositories like Notion or Confluence, or issue trackers like Linear or Jira.

Posit news

Posit Assistant in Positron

As of the June release of Positron, Posit Assistant is now the default experience in Positron. Positron Assistant will be deprecated starting in the 2026.07 release (release date July 6).

We understand that the names are confusing! Our hope is that the transition state will be over soon and the confusion will lessen. If you want to understand why we gave different assistants very similar names, read this blog post from Posit CTO Joe Cheng: A brief and biased history of Posit data science agents.

Posit Assistant works with the same providers as Positron Assistant. If you have a working provider setup with Positron Assistant, you’ll be able to use that same setup with Posit Assistant. You can read more about available providers here.

Package updates

• raghilda v0.2, a Python package for Retrieval Augmented Generation, is now on PyPI. This release, among other things, broadens support for crawling sites.
• debrief, an R package for LLM-friendly profiling, is now on CRAN. debrief turns profvis profiling output into text-based summaries, allowing AI agents to optimize R code more effectively.

In the “mocked” condition, we secretly tamper with well-known datasets, like mtcars, so that when the model plots the data, the trend shown is not what it expects. This is especially adversarial and not representative of the use case we’re most concerned about, the more realistic “intuitive” condition. For these samples, we synthesized data, but named the datasets and variables to suggest an intuitive relationship that the actual data contradicts. Finally, the “baseline” condition asks models to interpret plots with generic column names, and the results establish that models can ‘see’ just fine.

We tried all sorts of things to drive those scores up, to little effect. We let the model write a ‘memo’ to itself in a private scratchpad, enabled thinking, introduced a “model-in-the-middle” that pre-interprets the plot (so that the main agent sees only text), and even prefilled the response so that the agent was forced to say the correct interpretation (which it would then directly contradict).

bluffbench is near saturation

As is often the case, the best approach to drive these eval scores up was to wait a few months. A couple months ago, with the releases of Gemini 3.5 Flash and Opus 4.8, we saw our first >50% scores on the hardest “mocked” case in the eval. Then, last week, Fable 5 nearly aced the “mocked” case; almost all of the samples that it failed on had triggered the biology classifier and fallen back to Opus 4.8.¹

There’s still certainly room for improvement here; any human data scientist would likely score near 100% on all three conditions. A score in the 70s is still meaningfully sub-human. That said, it doesn’t seem like it will be long until bluffbench is saturated:

As is the fate of many saturated evals, we’ll soon be using bluffbench primarily to identify models on the Pareto frontier: what is the cheapest model that can reliably interpret plots, even when the plotted results are counterintuitive? While it’s certainly impressive that Fable 5 (medium) can interpret counterintuitive plots this well, driving all data analysis conversations with a model this expensive is not economically feasible for most of our users.

There’s still more to do

Even though bluffbench itself is nearing saturation, the problem of LLM agents failing to reliably incorporate surprising evidence and subtle artifacts into their analyses remains an issue. We see this qualitatively in our own work, and also have recently been able to quantify this in evals we’ll release soon.

Notably, the bluffbench harness and conversations are relatively contrived and unrealistic. There is no system prompt (by default) and the models receive only a single tool called create_ggplot(), which they use to create the plot that they’re then immediately asked to interpret.

For actual coding agents, system prompts can be extensive. Posit Assistant’s and Claude Code’s stretch into the tens of thousands of tokens. And in contrast to bluffbench, where the plot interpretation task appears in the first user message, real agents might create and interpret plots hundreds of turns into a conversation, and there’s evidence that performance may degrade as conversations go on.

Further, the user messages in the bluffbench eval are quite “clean.” They are composed only of the user’s relatively clear directive, with no “fluff” automatically injected by the harness. In most real coding agents, there are all sorts of hidden information and system reminders appended to the user message that are broadly unrelated to what the user is requesting in most situations, and agents need to ignore it.

Because of this lack of realism, and the fact that the eval and associated writing may now appear in the training data of newer models, models may know they are being evaluated in bluffbench. This risks ‘sandbagging’, where LLMs alter their behavior in situations where they suspect they’re being evaluated.

We’re seeing that in long-context, multi-turn, messy situations, even the most capable frontier models still struggle to notice counterintuitive patterns shown in data. This will continue to inform the design of our own data agents, which we’ve intentionally made auditable and less-autonomous than many other agents on the market: we continue to believe that there are not yet models and harnesses capable of highly independent, green-field data analyses.

Posit Assistant’s harness helps

That said, we’ve been encouraged (and somewhat surprised) to see that Posit Assistant’s harness seems to improve agents’ ability to interpret counterintuitive plots. Through some confluence of factors arising from the agents’ prompting, Posit Assistant scores more highly on bluffbench than the minimally sufficient harness from the open source eval:

This correctness boost also holds up in comparison to another popular coding agent. Coupled with the UI affordances that make it easier for users to audit the agent’s code and results, we’re excited to see that Posit Assistant seems to be a step forward in what’s possible for correct agent-assisted data analysis.

Through 2023 and 2024, the dominant narrative on AI token pricing was that the cost to use AI was dramatically decreasing and would continue to do so. However, in the last year, that trend is reversing course.

• On Tuesday, Google released Gemini 3.5 Flash, and priced it at three times the previous Flash model, Gemini 3 Flash.
• GPT-5.5, released last month, is twice as expensive as GPT-5.4.
• Opus 4.7, also released last month, has the same price per token as Opus 4.6, but uses an updated tokenizer that results in ~45% more tokens for the same text.

Combined with changes to subscription pricing (Copilot, Codex, Claude), this suggests that AI prices are on the rise. Over the last few years, the major model providers have subsidized tokens to acquire users, and it seems like that period is ending.

You can read more about this trend, and also how it shaped the thinking behind pricing Posit AI, in this blog post: Posit AI is priced for the long run.

Posit news

Gemma 4 in Posit Assistant

Gemma 4 is now available in Posit Assistant via the Posit AI provider. Gemma 4 is a relatively small open-weights model released by Google Gemini, and one example of recent improvements in local models small enough to run on a laptop.

The primary motivation for including a model like Gemma 4 in Posit AI is cost. We wanted to include a budget model that is still capable and reliable enough to power Posit Assistant across a wide range of tasks.

We recommend using the more capable models with longer-running tasks (e.g., implementing a feature that touches code across many files in a package), but Gemma is still capable of assisting with basic data analysis. You can read more about model choice in the announcement blog post.

Tidy design principles

Hadley Wickham restarted his Tidy design principles substack this week, starting with a thoughtful reflection on the tension between LLMs’ helpful and harmful effects.

The following is an excerpt of that post, but we encourage you to read the entire thing.

In future posts, I’ll get more technical, but I wanted to begin by acknowledging your likely deeply conflicted feelings about AI.

Programming accessibility. There are tons of people who could benefit from a programming language like R, but can’t justify the investment in learning it. AI has dramatically lowered that barrier and you can now get many of the benefits of reproducible programming with R much faster than you could before. Similarly, effective usage of git is now within reach to a much broader audience.

Translation. While machine translations are still far from perfect, their quality has improved radically in the last few years. This means that much more of the programming ecosystem is now available to the majority of the world who are not fluent English speakers.

Voice input. Voice input is a super exciting technology because it means that you no longer need to be a fluent touch typist in order to quickly get your thoughts into a computer. (Not to mention making a lot more technology available if you can’t read or write.) That’s a meaningful expansion of who gets to participate in technology.

Wide and shallow expertise. I love Tukey’s quote that statisticians get to play in everyone’s backyard. And it’s now easier than ever thanks to AI. AI will not make you an expert but can give you shallow expertise in basically anything you’re curious about. I think that’s pretty cool.

Finally, I have found AI to be a tremendous accelerator in my own work. It’s allowed me to fix 100s of issue in core infrastructure packages like roxygen2 and testthat. This is not AI slop; this is carefully vetted code that I can now write ~2-5x faster than I could before.

But you can’t use AI without also considering the harms, of which there are many.

Environmental impact. At the individual level, I believe that if you want to reduce your environmental footprint, there are higher-leverage changes that you can make. But at the societal level, the picture is more concerning: the rush to create new data centers is increasing need for electricity and water, and leading companies to rollback their climate commitments.

Copyright theft. LLMs are trained on vast quantities of copyrighted material, taken at an unprecedented and industrial scale, without permission or compensation.

Concentration of wealth. The AI craze is pushing more and more money into the hands of fewer and fewer people. I find the concentration of wealth and power into the hands of a very small number of people to be genuinely disturbing and I think is something that we should all be concerned about.

Intellectual laziness. AI supports a kind of shallow engagement where you never have to strain your brain on any task. The path of least resistance is to disengage and just let the model handle it. You no longer have to experience any mental discomfort, and thus you never really learn.

Equity and access. I’ve built my career around open source software, and one of the things I love about it is that it’s available to everyone, everywhere in the world, regardless of their means. That’s not possible with AI. The best tools cost real money, usually charged in US dollars, and that makes them out of reach for a lot of people in a lot of places.

The latest release of Posit Assistant includes a Data Cleaning Mode. When Posit Assistant enters the mode, it spends time identifying and fixing data quality issues and preparing your data for analysis. When certain decisions (e.g., how to recode a variable) require user decisions, it surfaces those decisions in a specialized interface. When you and Posit Assistant are done making decisions about the cleaning process, all the cleaning code is written to a script.

Broadly, we’ve seen that many coding agents seem to have a superficial regard for data quality, focusing only on errors thrown by analysis code. This is one of several features we’re iterating on to tailor the agent experience more closely to the real work of data science.

The initial release of the tabpfn package recently made it to CRAN. TabPFN is a pre-trained neural network for tabular data. In a typical predictive modeling workflow, you train a model on existing data (the training data), and then apply the resulting model to new, unseen data. TabPFN, in contrast, is a neural network pre-trained on a vast array of synthetic datasets. When using TabPFN, the model learns from the analyst’s training data in-context, in a similar way that an LLM can ’learn’ over the course of a conversation, allowing it to predict on new data without an explicit training step. The tabpfn package provides R tidymodels bindings to the pre-trained model.

Terms

TabPFN and LLMs rely on in-context learning to provide more helpful responses.

Many recent model releases have announced increased context windows of 1 million tokens.¹ The context window is the maximum length of the ‘conversation history’. So, if you’ve had 10 messages back and forth with an LLM, and each message was composed of 100 tokens, the context length would be 1,000, well under the context window for most modern LLMs.

LLMs learn in a few different stages. The most recognizable stage is probably pre-training, in which the model gains general knowledge and capabilities by learning from massive collections of text. Another important stage of learning is in-context learning, which happens over the course of a conversation. For example, if you paste a document into an LLM chat and then start asking questions about it, the document has been learned in-context.

TabPFN follows this same learning setup as modern LLMs. The TabPFN team pre-trains the neural network and then allows users to download the resulting weights. Then, the model learns from the analysts’ training data in-context.

Learn more

• OpenAI released GPT 5.5, an incremental improvement over GPT 5.4. Google Gemini released Deep Research Max, an incremental improvement over its Deep Research tool.
• Yihui Xie wrote up a thoughtful reflection on his experience with AI-assisted coding.
• LLM use for open-source contributions can be controversial. Zig has instituted a ban on LLM contributions. You can read their rationale here.

The AI newsletter has moved! From this edition forward, we’ll be publishing here, on the Open Source website. You can read past editions on the main Posit blog.

External news

AI and cybersecurity

Anthropic recently announced Project Glasswing, an effort to bolster critical software against the threat of AI systems that are increasingly capable of finding and exploiting security vulnerabilities. Through the project, a number of organizations that maintain critical software will be allotted credits to Claude Mythos, Anthropic’s yet-to-be-publicly-released model that reportedly found high-severity security issues in “every major operating system and web browser”. The goal is to give these organizations a head start in finding and patching these vulnerabilities before bad actors have access to the same technology.

A step change for local coding agents

One of the most common questions we receive when releasing LLM-enabled software is “Can I use this with a local model?” Up until this point, we’ve largely discouraged using our products with LLMs small enough that you could run them on a laptop. Local models are not there (yet) showed that, at the time, models small enough to run on a laptop failed a simple refactoring task every time, while the frontier, cloud-hosted models were largely successful.

However, the last couple weeks have brought two important releases for the local model space—Gemma 4 and Qwen 3.6. Both of these new models successfully completed the refactoring task in 9 of 10 attempts.

Posit news

Posit recently announced ggsql, an implementation of the grammar of graphics for SQL. One of the motivations for ggsql is safer AI agent tooling for visualization. With ggsql, rather than giving a visualization agent a full R or Python runtime (where the agent could execute arbitrary code, write files, or make network calls), you can provide it with a SQL runtime instead. SQL limits the agent to database operations, and you can restrict even those by connecting read-only, reducing the risk that the agent will do something undesirable.

Terms

A dense model is one where every parameter is used to process every token. For example, if a model has 14 billion parameters, in a dense model, all 14 billion are involved in generating each token.

Dense models are in contrast to sparse models, a subset of which use an architecture called Mixture of Experts (MoE). In MoE models, only a fraction of the model’s parameters are involved at any given time. The model contains many parallel sub-networks called “experts,” and a small router decides which ones to activate for each token. This means a model only needs to pay the computational cost of the small active subset per token.

This is the architecture of the most recent local models, and partially explains the jump in ability we saw. Gemma 4 26B-A4B, for example, has 26 billion total parameters but only activates 4 billion per token (the “A” is for “active parameters”). The models we tested from the previous generation were dense, requiring you to choose between a model small enough to run on a laptop or one that was good enough to be useful. The MoE architecture helps break that tradeoff.

Learn more

• Anthropic released Opus 4.7, an incremental improvement over Opus 4.6. OpenAI released GPT-5.4-Cyber, a fine-tune of GPT 5.4 for cyberdefense.
• A new paper shows that vision language models excel at many image benchmarks even when the actual images aren’t provided, because the answers are implicit in the questions.
• Posit’s Jeremy Allen has been keeping a log of the ways AI is transforming work.