Having used D1 in production - it's not very reliable. There was a comment on HN from one of D1 devs [1] saying that D1 is a sql wrapper on top of durable objects that mimicks sqlite distributed like nature.
> I'll let you in on a sort of dirty secret:
It's almost always better to use Durable Objects storage, rather than D1. Even if you only want a single global database, it's better to implement that as a singleton Durable Object, than by using D1. Because that's all D1 itself actually is: a singleton Durable Object that exposes an API to its SQLite database. It's just a wrapper.
D1 also gives read replication and point in time recovery, but my biggest issue with it is you can’t create D1 dbs dynamically like you can with durable objects so things like tenancy are harder
Its funny, when I was reading part time parliament paper (paxos) and while learning zig and odin language created a pure paxos library and then the first application for that library was to build a multi-master sqlite. The two approaches I took are very different in case of zig I build zaxonlite (https://github.com/insanai/zaxonlite) and have a benchmark on it that includes rqlite. It uses WAL replication as strategy. The second attempt I tried in odin is sqlodin (https://github.com/insanai/sqlodin), in this the system does not use WAL but run sql query on each node, as result subset of sql query in sqlite are not supported.
Wow, a 155 page book for a relatively new project is interesting - what was your workflow for that? It looks mostly generated - is it used in its entirety as context to a top-level agent when working on new features, or are you directing it to just take relevant subsections? Why the PDF output (it does look nice)?
As someone who loved sqlite and the simplicity, the truth is that with agents I am finding it is extremely easy to manage postgres and it is realyl nice having all the SQL features of postgres -- sqlite is quite limiting in a lot of SQL. So if you are considering remote sqlite and all that, feels a better time than ever to just use a database server.
Of course, sqlite is amazing for local tools etc.
The problem is "remote" SQLite. Just run a "single" beefier box with in-process SQLite and Litestream backups to object storage and never have to think about RDBMS again. I'd wager there are 90%+ applications that would be better served, more performant, and cheaper with this design.
I've had the exact same experience. In the before-time I often reflected on how my love of sqlite was borderline irrational for what it is. Now, in a few projects I've let the LLM do postgres, and it's so nice having the pain if a db server mostly just handled. Still choose sqlite for anything embedded or without a hosted server.
I am the same way with MSSQL. I used to avoid it like the plague because of how the permissions/logins/everything worked. Now I can throw a T-SQL script at it in a few seconds to make it behave.
I feel like there should be a comparison table listing all of these engines and how they compare on their external dependencies, whether a single leader/any node can write, their shape (embeddable library vs server), etc.
IMHO support for transactions is a pretty important dividing line. Some engines support full SQLite transactions while others - like rqlite - don't officially support BEGIN/COMMIT [0], which matters if you rely on multi-statement transactions.
A friend/colleague has been building a pure-Python client for dqlite [1] (DB-API and SQLAlchemy), which may interest folks here. dqlite has full support for transactions, and like rqlite doesn't have external dependencies (ie no S3/Consul).
No knock on rqlite or any other engine - dqlite has warts too [2].
The comparisons between various projects, including between rqlite and Litestream, were interesting. (rqlite uses a consensus algorithm between nodes to keep them synced and allow changing the primary node, while Litestream just syncs itself to S3 so another node could take over with the data from there.)
I fully expected the article to get into D1 shortly and mention how that's about the same relationship between Cloudflare Durable Objects (which D1 is based on) and celld, an open source reimplementation of Cloudflare Durable Objects which writes to S3. But then the article treats D1 with a very sudden snarky disinterest, and then the final section feels very awkwardly defensive of rqlite and defeatist.
It's loaded with it to my eyes. (Pangram also agrees and flags the whole post as AI generated.)
# 2014: rqlite
The need was concrete.
# Before 2014, and alongside it
Two things deserve mention for honesty’s sake.
(the above line is extremely typical of Claude Opus 4.8 and 5.0)
[...] and the main difference is of one of packaging:
# What the twelve years show
Three things, I think.
The core was right. [...] Nobody has found a reason to stop doing it.
The design space has three corners, and they appeared in order.
(What order? The corners of this design space, which aren't spelled out until the following sentence, don't have any other natural ordering. This type of misapplied flourish feels very typical of LLMs.)
I envy you. For better or worse, Claude is the author I've read the most in this year. There's a handful of human authors I know well enough so that I would recognize a text that was written by them. With LLMs, it's much easier than with humans, since the current generation of LLMs seems utterly incapable of changing their style.
https://github.com/punnerud/mpedb is not there yet, focused on sync between sqlite3, Postgresql and Oracle. Shown that concurrent writes to sqlite3 is possible at the speed of Postgresql
(My own project)
It is a set of Java dependencies so a full data-bearing Ignite node will happily be an in-process part of your Java monolithic app or microservice. It has likewise bindings for .Net and some other languages but not so seamless as in case of JVM.
Last time I looked into it, you CAN bypass the rqlite raft layer, and hit the local sqlite directly, but you are not guaranteed consistency. Depending on your use case, that may be okay.
It's basically holding tenant and operator master registry that app nodes pull down resulting in very light load on rqlite. The architecture is resilient to control layer being down (just can't provision new tenants).
I maintain one of the client libraries for rqlite - I’ve found it an excellent concept really well implemented. One of my favourite bits is it gives you back json making Dapper-like mapping super easy in .NET and I suspect other languages.
piterrro 1 day ago
> I'll let you in on a sort of dirty secret: It's almost always better to use Durable Objects storage, rather than D1. Even if you only want a single global database, it's better to implement that as a singleton Durable Object, than by using D1. Because that's all D1 itself actually is: a singleton Durable Object that exposes an API to its SQLite database. It's just a wrapper.
[1] https://news.ycombinator.com/item?id=48611834
ruther4d 20 hours ago
unconscionable 23 hours ago
oblio 9 hours ago
ruther4d 8 hours ago
vikrantrathore 22 hours ago
thadt 18 hours ago
mmnfrdmcx 20 hours ago
mxey 15 hours ago
oblio 9 hours ago
rapind 10 hours ago
I'm able to handle around 200k/s on a sub $100 box. https://github.com/pairshaped/gleam_sqlite_benchmarks/blob/m...
smellf 14 hours ago
canadiantim 12 hours ago
bob1029 11 hours ago
gregsadetsky 10 hours ago
IMHO support for transactions is a pretty important dividing line. Some engines support full SQLite transactions while others - like rqlite - don't officially support BEGIN/COMMIT [0], which matters if you rely on multi-statement transactions.
A friend/colleague has been building a pure-Python client for dqlite [1] (DB-API and SQLAlchemy), which may interest folks here. dqlite has full support for transactions, and like rqlite doesn't have external dependencies (ie no S3/Consul).
No knock on rqlite or any other engine - dqlite has warts too [2].
[0] https://rqlite.io/docs/faq/#does-rqlite-support-transactions
[1] https://github.com/letsdiscodev/python-dqlite-dbapi
[2] https://github.com/canonical/dqlite/issues/882 opened by my friend, https://github.com/canonical/dqlite/issues/908 , https://github.com/canonical/dqlite/issues/912 opened by me
AgentME 20 hours ago
I fully expected the article to get into D1 shortly and mention how that's about the same relationship between Cloudflare Durable Objects (which D1 is based on) and celld, an open source reimplementation of Cloudflare Durable Objects which writes to S3. But then the article treats D1 with a very sudden snarky disinterest, and then the final section feels very awkwardly defensive of rqlite and defeatist.
klamann 23 hours ago
formerly_proven 21 hours ago
phrotoma 20 hours ago
AgentME 19 hours ago
bomewish 17 hours ago
klamann 17 hours ago
senderista 13 hours ago
dizhn 19 hours ago
Kinrany 1 day ago
- embeddable like sqlite
- clustered like dqlite
- has a standalone node like rqlite
- can be backed up to S3 like litesteeam
- optionally AP like cr-sqlite alongside CP
punnerud 1 day ago
thriftwy 21 hours ago
Kinrany 19 hours ago
thriftwy 18 hours ago
chasil 1 day ago
https://rqlite.io/
qaq 1 day ago
gizzlon 22 hours ago
jasonjayr 16 hours ago
pizzafeelsright 21 hours ago
it is great, fun, easy.
lukaslalinsky 21 hours ago
qaq 8 hours ago
bfren 1 day ago
gizzlon 23 hours ago
My understanding is that rqlite gives you "stored in more than place" guarantees, which is nice if you need it. Much slower than a 1 node db though.