Hi! I'm the creator of pgrust. Let me explain what happened.
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
Agreed. Though we still don't have a reason why they rushed a custom preview to a rigorous benchmark competition. Feels like too much hype, and now damage control.
I got it slightly wrong. It was code that you could run with an environment variable: PGRUST_GRAVITY_IRPROBE. It still wasn't intended to be part of the actual binary.
This comment chain is hilarious. Did you ask Claude or another LLM what that file was, then ask it the question from OP comment (how it was included in binary), and it admitted it was wrong?
Because this is such a common pattern when working with LLMs, and this looks exactly like it.
No. I remember looking into this before, back when we were originally accused of cheating. I knew the code was dead and unused, but I got the reason wrong and had to double-check.
Even with your explanation, this whole thing seems weird and scammy. What's the point of submitting something that is so incomplete and heavily tailored to this one benchmark outside of self promotion?
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
This absolutely does not pass any smell test, by your own admission you included the test files "by accident", could only barely pass the minimum of SQL for the benchmarks, included environment variables for "internal benchmarking" and a preview version you cannot immediately release because it might contain sensitive data? What kind of sensitive data are you putting in a Postgres slop fork in such a way that you cannot immediately release it, besides obvious admissions of cheating? This all just reeks.
> This absolutely does not pass any smell test, by your own admission you included the test files "by accident"
If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
I mean, if the now-released open source variant performs roughly the same, and has no weird behavior in it… the matter is resolved, no?
They’ve already submitted the best possible corrective step for any sus behavior they previously did.
It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
Honestly, if the database is not able to be run with a variety of queries other than the one used by ClickBench then don't submit it to the benchmark. Every other database in their list can run anything you throw at it. Hiding behind messaging that it was an early 0.4 build that can't do everything is ... weird! Just don't submit it to ClickBench if it is not usable by the world yet.
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
Yeah, it was a mistake to submit it as early as we did. I was very excited when we got the early version working and overeagerly submitted it to ClickBench
The new general version is actually faster than the version accused of cheating.
The architecture of the query engine is:
SQL -> IR -> Assembly
-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.
If it delivers on being faster than the version accused of cheating with reasonable cache settings and no SQL feature limitations, I would say that resolves the accusations, regardless of the source code of -preview.
The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.
0.4-preview was an early version of 0.4 that still had some cruft like the feature flag you pointed out. We removed the cruft for the release of 0.4-beta
Why don't you open source 0.4-preview too, to make your claims verifiable. The longer you take to release it (or choose not to release it at all) the more it'll look like a cover-up. If the binary is (made) public, it's going to get the AI-assisted decompilation treatment anyway.
> For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview.
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
> This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview
That statement provided context and was not part of the defense. The point was that -preview didn't have the issues described, except for limited SQL support.
In the past, we've done that because we've tended to do large refactors, many of which we've backed out of. For that reason, we've kept our own internal development version, which is ahead of the public version. Then when whatever we're developing is ready for public use, then we open source it
malisper 1 day ago
For the most part, the actual accusations are false. This submission was an earlier version of pgrust 0.4 called pgrust 0.4-preview. We just open-sourced a newer version called pgrust 0.4-beta yesterday, and you can fully audit it here[0]. Rather than contest the accusations on GitHub, we figured it would be easier to complete pgrust 0.4 and open-source it to show that we're being honest. You can also verify what I'm saying yourself against the 0.4-preview binary we submitted to ClickBench here[1]. I don't have a way to prove this is the exact binary submitted, but if ClickHouse provided the SHA of the binary they tested against, it would match.
The three main accusations boil down to: 1) We used handwritten code optimized against each query in the benchmark, 2) We cached the results between query runs, and 3) We didn't support queries outside of the 43 ClickBench queries.
First, we did not use handwritten queries. The code Alexey pointed out does not run during the benchmark. The design of 0.4 relies on a JIT compiler that compiles your SQL to a Lisp-based intermediate representation that we call Gravity IR that we then convert to x86 and ARM assembly. While developing 0.4, I wrote handwritten code against the IR for benchmarking purposes. The idea was that if queries written in our IR are fast, and we can compile good enough IR fast, then our new query engine will be fast. This is also the reason the directory those "kernels" are in is called "irprobe" - it was a probe of our IR to make sure it was fast. Our 0.4-preview submission went through the pipeline: SQL query -> Gravity IR -> assembly.
Second, the "result-cache" Alexey observed is the result of a cache we use to cache which predicates match which strings in our dictionaries we use for dictionary compression. Alexey wasn't aware of this cache and assumed we were just caching the query results. On ClickBench you are allowed to use caches like this that hold intermediate data[2]. ClickHouse has its own cache similar to this called the Condition Cache[3]. In our newest submission, we were upfront about all the caching we do and how to disable each cache[4].
The third claim that we didn't support other queries outside of the 43 ClickBench queries is partially false, but does have some ground to it. As I said before, pgrust 0.4-preview was an early version of pgrust 0.4. Once our new JIT compiler supported enough SQL to handle the ClickBench queries, we decided to submit it to ClickBench. AFAIK, ClickBench has no rules about how much SQL you need to support. On top of that, we did ship that we did ship 0.4-preview with our old query engine. It would have been a two-line change to fall back to it and support a much broader set of queries outside what our new engine could support. Those queries would still be fast, but not quite as fast as our new engine was. Our new 0.4-beta release does now support all Postgres SQL (minus some bugs around CTEs and MERGE)
To sum up, pgrust 0.4-preview was an early build of pgrust 0.4 while it was in development. It was a real query engine and actually executed queries. I acknowledge that we submitted it prematurely, when it supported a narrow set of SQL, and we should have ensured broader SQL coverage before submitting it. With our new submission that solves these problems, and we're hoping for pgrust to be reinstated. If there's anything off about the submission, please point it out and we're happy to fix it.
[0] https://github.com/malisper/pgrust/releases/tag/v0.4-beta
[1] https://pgrust.com/downloads/archive/2026-09-clickbench-snap...
[2] https://github.com/ClickHouse/ClickBench/blob/main/README.md
[3] https://clickhouse.com/docs/concepts/features/performance/ca...
[4] https://github.com/ClickHouse/ClickBench/pull/2751
remywang 1 day ago
malisper 1 day ago
g-b-r 2 hours ago
verdverm 18 hours ago
hitekker 1 day ago
mort96 1 day ago
malisper 1 day ago
f311a 1 day ago
malisper 1 day ago
lossolo 1 day ago
Because this is such a common pattern when working with LLMs, and this looks exactly like it.
malisper 1 day ago
lambda 1 day ago
Benchmarks should be used for measuring general purpose query engines, not be an end in themselves.
This seems like a prime example of Goodhart's law. This benchmark is now just being used as a target for self promotion.
malisper 1 day ago
We were just really excited about what we had built and how fast it was that we wanted to submit it.
sondr3 1 day ago
malisper 1 day ago
If you don't believe you can run the binary yourself. You'll see the numbers match the ClickBench numbers and that those files are dead code during the benchmark
mort96 1 day ago
camuel 1 day ago
mort96 1 day ago
setr 1 day ago
They’ve already submitted the best possible corrective step for any sus behavior they previously did.
It doesn’t even really matter if the preview binary was in fact cheating, if the beta binary meets the benchmark results always. It’d be like cheating on a test you were gonna score 100% on anyways.
camuel 1 day ago
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
tpetry 1 day ago
I hope they don't allow pgrust in again until the project matured. Otherwise the benchmark will be useless as it gets flooded by many projects that can run ClickBench fast but have questionable support for everything else.
malisper 1 day ago
jimmydoe 1 day ago
spongebobstoes 1 day ago
that would help clear the air, if the full-sql version has similar performance to the partial-sql version
malisper 1 day ago
The architecture of the query engine is:
-preview had the full IR -> Assembly pipeline built out, but only part of the SQL -> IR part. As long as queries compile to the same IR, they're just as fast. We've shipped some more optimizations, so they got even faster.infogulch 1 day ago
The remaining lesson here is that one should temper one's excitement with pre-release software, whether the software was built with AI or not (rather, especially if its built with AI). The preview results would have been more appropriate to announce on a social media platform than as a PR to the ClickBench repo.
spongebobstoes 1 day ago
philipwhiuk 1 day ago
malisper 1 day ago
philipwhiuk 1 day ago
malisper 1 day ago
overfeed 1 day ago
up2isomorphism 1 day ago
anonymous908213 1 day ago
This is a non-sequitur. It doesn't matter what the earlier version is called. It is what you submitted, and it is what the accusations are against. You don't get to create a new submission that (allegedly) fixes all of the issues and then say that the previous accusations are "false". Downplaying the issues by calling it a 'preview' when you submitted it yourself is also a bit of having your cake and eating it too, don't you think?
> ClickBench has no rules about how much SQL you need to support.
It is because of people like you that we have warnings not to eat tide pods, Jesus Christ.
malisper 1 day ago
That statement provided context and was not part of the defense. The point was that -preview didn't have the issues described, except for limited SQL support.
torben-friis 1 day ago
malisper 1 day ago