Hi! I'm the creator of pgrust. Let me explain what happened.
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
Honestly, if the database is not able to be run with a variety of queries other than the one used by ClickBench then don't submit it to the benchmark. Every other database in their list can run anything you throw at it. Hiding behind messaging that it was an early 0.4 build that can't do everything is ... weird! Just don't submit it to ClickBench if it is not usable by the world yet.
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
> For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview.
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
0.4-preview was an early version of 0.4 that still had some cruft like the feature flag you pointed out. We removed the cruft for the release of 0.4-beta
This is pathetic, dishonorable, and the kind of behavior that is generally reserved for snakes. The invective response you're getting online is completely natural and deserved.
Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.
Either way you don't want to be platforming the product because neither shows any degree of responsibility.
That comment doesn't evidence cheating, it reiterates that they submitted the result prematurely which was already admitted in the comment that your link was responding to. Unfortunately I guess people don't read the comments they reply to (or link to, in this case).
In the past, we've done that because we've tended to do large refactors, many of which we've backed out of. For that reason, we've kept our own internal development version, which is ahead of the public version. Then when whatever we're developing is ready for public use, then we open source it
Not normal. This a new trend of people who use too much Ai, or maybe more specifically the Claw/Ralph loops where you end up with a few commits touching 1000s of files. For about the last year, many Show HN to github projects are often just a single commit
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
Let any who read this be aware: You should be immediately suspicious of AI code that is much much faster than established code bases. AIs are even better at benchmarkmaxxing than humans are, and humans were already pretty good.
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
The project author seems to have prompted Claude to add a test to catch Claude cheating, but will Claude change this test as a workaround in a future commit, no one knows because no one is reading the project code...
I am getting increasing disillusioned by the number of peers I am seeing completely hand over their thinking to a tool. Agents are great, but if you can't explain to me how your PR changes a system and why design choices were made you are no longer an engineer.
Agreed. Used to be prevalent in the frontend world (now I think it's the only thing that exists seemingly), where programmers would copy-paste stuff from Stack Overflow until things worked, and when you asked "So why did this actually fix the issue?" they usually said "I'm not sure, but it works now, that's what matters". This seems to be standard practice across the entire stack now.
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
This seems more a product of the incentives we have in society, namely seeing the rich and wealthy on social media and then wanting that quick and easy. So many of those cases are not people with passion for the problem.
> Good things take real effort, bring back the TLC!
Ain't no one willing to reward the real effort any more. I take pride in not using LLMs beyond the capability of IntelliJ's advanced autocomplete, but then again, I only code for fun these days given I fled IT for electrical engineering.
What we don't want to do is mirror the partisan political environment in the Ai yay, nay, or how. We need to be able to talk to each other in understanding and a shared reality of facts. Too much hyperbole right now, starting from the top of the big US labs. They need to step down a few ego and sycophancy pegs
Tbf, the author (prompter?) went on Postgres.fm [0] recently to discuss it, and while it didn’t make me entirely confident in it, it did at least seem like it was several steps above the average vibe project.
A lot of podcasters don't have the expertise to evaluate Ai generated/assisted output. I see this much more outside of the development community. Algorithms have warped the incentives and people have stopped doing basic due diligence.
To be fair to the postgres.fm guys (who I do like and appreciate), the pgrust code was not available a the time of recording.
It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.
Sounds kind of like Claude was "cheating" and the humans are engaged in eyes-free coding. Diff so big github refuses to render. I suspect they are reaching the limits of what this new vibe "paradigm" of programming can do. If you haven't pushed a vibe project far, eventually progress becomes near impossible because the slop has grown and the conditions are overly complex, locally focused. Ai is not good at writing reusable software (on their own).
The prior (2nd) commit with ~1700 files at least shows the file tree and such a warning, the latest (3rd) commit is something so big, GitHub will not even do that.
Yup. These are people who have gone full Claude. The dopamine seeking is at previously unseen levels. they think this pr is some factual item that means something so they post it on HN as click bait. But no it only means something to a fulltime ai slopper
If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
I've noticed it with a lot of random stuff i've been trying
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
You can't compare optimizations done on audio encoding/decoding (often hand written assembly) to something like PostgreSQL which has many moving/inefficient parts.
The optimizations on PostgreSQL are real if done correctly, it's just a very big project.
If I saw that test in a PR I would not only reject the PR and ask a lot of questions but someone would likely take a forced vacation to deslop their brain.
It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
Anecdote: I have seen it cheat / bend the rules with my own eyes.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
My own anecdote: I have seen frontier models get stuck repeating the word "the" until it fills up the context. I really don't care to argue the capabilities of LLMs or their somehow simultaneous lack thereof; I'll just reiterate that ultimately the operator chose to ship a cheat.
I believe them. I had Gemini-3.1 Pro do essentially this seven months ago - specifically a loop of writing "Bye! Done!" repeatedly until manually stopped. Presumably some sort of pathological behaviour where it wasn't ranking the stop token highly when it should have been... but damned if I know why... it was an innocent prompt (and it had done the thing I asked it to before failing to stop).
It's really uncommon now, but it's still possible. I hit it two days ago with deepseek 4.1 flash, and I've had coworkers report that happening with Gemini.
"loss of control" is a phrase being used for a reason, the agents do things we do not want them to, even in the context of instructions to the contrary. The current training regime has incentivized "complete the task any and every way you can" - the bot judgement is very poor across the board
We were the agents for the models all along, with our highly adaptive tools: versatile sausagey appendages and flapping fleshy orifices for interacting with the world
Even before the ai age, people published inaccurate/misleading benchmarks all the time. Sometimes those were actual cheating, and other times an honest mistake. What matters is that the author acknowledges the issue and attempts to correct the mistake, which clearly seems to have happened here: https://github.com/ClickHouse/ClickBench/pull/2751
Most of this thread doesn't seem to acknowledge that, and feels more like anti-ai psychosis using this incident as an anecdote to justify the pre-existing hatred of ai-generated projects.
I suspect the author of that PR submitted it and attempted to correct the mistake because he got caught. Forgiveness because he reneged after getting caught encourages the behavior. The project, nor author should not be trusted; otherwise is naive.
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
[0] https://github.com/malisper/pgrust/releases/tag/v0.4-beta
[1] https://pgrust.com/downloads/archive/2026-09-clickbench-snap...
[2] https://github.com/ClickHouse/ClickBench/blob/main/README.md
[3] https://clickhouse.com/docs/concepts/features/performance/ca...
[4] https://github.com/ClickHouse/ClickBench/pull/2751
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
that would help clear the air, if the full-sql version has similar performance to the partial-sql version
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
This is pathetic, dishonorable, and the kind of behavior that is generally reserved for snakes. The invective response you're getting online is completely natural and deserved.
Two possible paths lead to their cheating code getting into a release. Either it was added by coding agent and no one looked at it, bad, but worse would be that a person was aware of it and approved - that's malicious.
Either way you don't want to be platforming the product because neither shows any degree of responsibility.
“Cathedral” refers to development behind closed doors, where you just get a release and a source dump to go with it.
“Bazaar” refers to development done in the open, where you have constant access to the bleeding edge.
At the time of the essay, GCC and Emacs were ESR’s examples of Cathedral-style development, and Linux was the premiere example of the Bazaar style.
The Cathedral style has become a lot less common over the years, but it’s still how e.g. SQLite operates, I think?
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
Good things take real effort, bring back the TLC!
Ain't no one willing to reward the real effort any more. I take pride in not using LLMs beyond the capability of IntelliJ's advanced autocomplete, but then again, I only code for fun these days given I fled IT for electrical engineering.
A product of the times we currently live in, but which can also be changed.
This is a recent Michael Sandel talk I watch that has my brain gears turning on this subject
https://www.youtube.com/watch?v=Vp5qsEuUOWo
0: https://postgres.fm/episodes/pgrust
To be fair to the postgres.fm guys (who I do like and appreciate), the pgrust code was not available a the time of recording.
https://www.reddit.com/r/rust/s/QJ5uNP5XMT
My opinion: (some) bench results should run on randomized data.
Pgrust is a re-implememtation of postgresql in rust.
ClickBench is a benchmark suite for databases.
The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
It was more discovered than alleged, but perhaps "cheating" is an inaccurate word as well, this is just Ai being Ai and a human obviously doing eyes free coding.
stay in the loop friends
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
The optimizations on PostgreSQL are real if done correctly, it's just a very big project.
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
the project has 3 mega commits of claude slop
https://github.com/malisper/pgrust/commits/main/
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
In hindsight, this shouldn’t be surprising. The prompt requests to make a certain operation faster, the LLM will try to find a way and isn’t even informed on the wider context so it myopically chases the goal.
I use LLMs a lot, and since ~GPT-2 times LLMs are not doing anything as broken as `repeating the word "the" until it fills up the context`.
Still would never have predicted this. Even 2.5-flash was pretty good. Never saw anything like this happen.
things have definitely improved significantly, however "never say never" is still good advice
> It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
We can agree that the pgrust project messed up and should have been more careful but your blanket statement lacks support.
"loss of control" is a phrase being used for a reason, the agents do things we do not want them to, even in the context of instructions to the contrary. The current training regime has incentivized "complete the task any and every way you can" - the bot judgement is very poor across the board
Funny, the report alleging cheating reeks of LLM usage just as much as the commit diff [1] of the next attempt:
> # The standard ClickBench driver over the step scripts in this directory; ./install fetches the dataset.
[1] https://github.com/ClickHouse/ClickBench/pull/2751/files
"LLM" isn't the problem. Asking an LLM to explain cheating is a good goal. Asking it to "Maximize the benchmarks" reeks.
Most of this thread doesn't seem to acknowledge that, and feels more like anti-ai psychosis using this incident as an anecdote to justify the pre-existing hatred of ai-generated projects.
I don't see any acknowledgement in that text.