Show HN: Uberblok – an interactive museum of ZFS internals (uberblok.storagebits.workers.dev)

124 points by storagebits 3 days ago

I've worked for a long time in the storage field during my career and ZFS is the piece of engineering I admire most in that space. I wanted a way to pay homage to it and to explore and better understand its internals. So I built Uberblok: a museum of ZFS. A deep dive into copy-on-write, txgs, vdevs, RAID-Z, snapshots, the ARC, metaslabs, the ZIL, send/receive with explanations and animated diagrams.

I'm sure people here know these internals better than I do, so corrections are very welcome.

I built it with my agent swarm organization, Algotitan.

I hope you'll enjoy the deep dive into Uberblok as much as I do.

Museum: https://uberblok.storagebits.workers.dev Repo: https://github.com/algotitanhq/uberblok

arlazar 2 minutes ago
Nice work on Show. I'd be curious how it behaves once the state is stale.
QuantumNomad_ 2 days ago
I absolutely love ZFS. I use it on FreeBSD for all disks on the whole system, including the ones that FreeBSD itself is installed on, and I use it with some disks on Linux and with most of my external disks with macOS.
In an alternative universe, Mac OS X would have gotten ZFS shipped with the base install back in 2007, and presumably we’d then still have that to this day. In our universe that ended up not happening even though it was supposed to :/ https://mjtsai.com/blog/2025/04/28/revisiting-zfs-for-mac/
I really would have loved to have the whole of macOS running on ZFS. But at least using ZFS from homebrew on macOS to have ZFS on external disks is the next best thing after that, and which we do have.
- ProllyInfamous 1 day ago
  I began learning ZFS (TrueNAS and the macOS-plugin) just two weeks ago. It is absolutely incredible technology.
  >I use it with ... most of my external disks with macOS
  How do you personally deal with the terrible macOS default SAMBA client (if using SMB shares)? I can get saturated single-file transfers, but even deleting 100+ small files takes a longgggggg time. It is my understanding the the macOS only allows single-threaded SMB access.
  I am considering using ExpanDrive (for NAS access only); openZFS is perfect for DAS (direct disk access), albeit not ideal (since the ARC takes away from system memory); is Thunderbolt linkage still impossible in TrueNAS?
  How do you connect your mac fastest to TrueNAS (not top speed, actual IO rate)?
  - QuantumNomad_ 21 hours ago
    >> I use it with ... most of my external disks with macOS
    > How do you personally deal with the terrible macOS default SAMBA client
    I almost never use SMB. The external disks in question are drives that I connect via USB3 or USB4/Thunderbolt to the MacBook Pro, including a couple 5-bay multi-drive DAS enclosures that have USB4/Thunderbolt.
- storagebits 2 days ago
  I forgot that Apple story, a great "what ifs". It's not in the history wing yet, thanks for the link, it might deserve its own panel!
cbarrick 2 days ago
Genuinely cool, but the LLM-speak started to wear on me.
An editing pass to make the prose less "punchy" would be great.
- MithrilTuxedo 2 days ago
  That's what that is! I was doing well until I got lost at Map of the Visit. I forgot I was pretending to be in a museum.
drewg123 2 days ago
I'd love ZFS a lot more if it weren't for the ARC. Using ARC means that ZFS caches stuff outside the kernel page cache. That in turn means that sendfile is no longer zero-copy when data originates on ZFS, making it a non-starter for content drives for our CDN workload.
- ahofmann 2 days ago
  Yes, but this is a Linux/licensing problem. Use one of the BSDs and the ARC doesn't fight with the system cache.
ashkankiani 2 days ago
Building a museum when you self admit you don't understand the internals that well... seems like a conflicting goal? If you're showing off to educate then how can you do that if you don't understand it?
We know the answer, obviously. I can't trust how much of this was something an AI said vs something you actually did seriously look over. At the very least, all of the text on this should've been in your own words, I think. Otherwise this is just show and tell of some prompts.
I wish I could enjoy tech demos anymore, but now they feel so hollow. Which is a real loss from the age of LLMs. I used to enjoy reading the dolphin EMU progress report for years. I wonder what they're up to.
- Hugsun 1 day ago
  Almost all of this was written by an LLM. No human writes like this.
unethical_ban 2 days ago
As a mere operator, I've been fascinated by ZFS ever since I first heard of it 15+ years ago. It seems so elegant, like it really is the last generalist filesystem one would ever need. Performant, reliable, fundamentals easy to comprehend. I've never tried managing LVM configs because zpools and raidz are just so damned easy. Native encryption. Datasets can be as simple or as complicated as you need them to be. send/recv dataset syncing at the block level vs having to decrypt and sync files individually.
A lot of this goes over my head as someone who isn't familiar with filesystem internals, but I appreciate the effort and the homage to a great tool.
- yjftsjthsd-h 2 days ago
  > Native encryption.
  I think maybe that's the one feature that's iffy? Theoretically great but bad history of bugs. (Unfortunately)
  - atmosx 16 hours ago
    I've been using encrypted datasets for years without problems. I don't used `send` and `receive` for backups though.
- gspr 2 days ago
  So many really brilliant people keep saying this, so I don't doubt that it's true. Can you explain to an ordinary user why ZFS isn't everywhere? It seems to be a marvel, yet ext4 lives on.
  - tom_alexander 2 days ago
    Everyone has mentioned the license, which is the reason ZFS isn't everywhere. I just wanted to add: Don't let the license scare you away. The CDDL is actually a nice compromise between copyleft (GPL) and copyfree (BSD). The CDDL is essentially the same as GPL except the GPL applies to everything it touches (include a single GPL *.c file in your project and now your entire project is subject to the GPL). Whereas the CDDL only applies to the CDDL-licensed files themselves. So you can include ZFS in your project without making your entire project subject to the CDDL.
    The issue is, the GPL wants to spread to everything it touches (including the CDDL-licensed ZFS files), which would make the ZFS files subject to both the GPL and the CDDL. But the CDDL and the GPL are incompatible, which is why Linux can't include ZFS whereas more permissively licensed projects like the BSDs can.
    So it's an unfortunate situation, but ZFS' license is not "bad". It's just the GPL and the CDDL are incompatible.
  - otterley 2 days ago
    Because ZFS is complicated to use and isn’t necessary for most workloads.
    The licensing issues don’t help. Consider that most Linux machines are in commercial settings (Google runs more of them than all individuals combined) and they don’t need Oracle on their backs.
    jjav 2 days ago
    > Because ZFS is complicated to use
    There is nothing complicated about having zfs be your filesystem. You can use its extra features like snapshots, send/receive, etc, but only if you wanted to. Otherwise it's just a filesystem, there is nothing to use as such, so nothing complicated.
    > and isn’t necessary for most workloads.
    Not sure what that means. What workload wouldn't benefit from a reliable filesystem?
    otterley 2 days ago
    The fact that ZFS isn’t massively popular after close to 30 years, and after nearly 20 years of availability via OSS, suggests that the benefits aren’t really that meaningful to most users.
    jjav 2 days ago
    Others have answered this: licensing
    Linux is the most common platform and many Linux distributions have an issue with license compatibility (or perceived one), which hinders adoption.
    unethical_ban 1 day ago
    I agree partially. In the age of cheap backups and cloud storage, having super-high-integrity filesystems with easy RAID is more of a niche, so the need for ZFS on desktop and app servers is low. ZFS is fine as a general purpose filesystem, too.
    In the end, I think it's momentum and licensing. AFAICT btrfs only exists because ZFS can't be shipped in Linux kernel. Defaults are insanely powerful, and ext4/xfs is good enough for a lot of use cases. If ZFS were BSD or GPL licensed, I think every Linux distro would ship with it as an option, if not a default.
  - storagebits 2 days ago
    I would say (in no particular order) that the CDDL license is not innocent, ext4 is good enough for many usages, cost ? ZFS is RAM hungry ... but ZFS is well present where it means most (NASes , Proxmox , backups to tell some)
  - unethical_ban 2 days ago
    Primarily licensing, IMO. Its license is incompatible with inclusion in the Linux kernel. Ubuntu ships with native support for it anyway; I guess they're not worried about legal issues. It's completely functional in Linux, you just have to take an extra step to install it, which is enough to keep it from mass non-techie adoption.
    Totally worth it for NAS. I think there are more systems that utilize its advanced dataset features in the BSD world.
- storagebits 2 days ago
  Thanks, that's exactly the reader I'd like to reach, and you're not the first to say it goes over your head. Honestly, I'm still trying to figure out who it doesn't go over the head of. Matt Ahrens, maybe? ^^ Next is a "start here" route: one plain question per room, answered before any code. Copy-on-write and snapshots are probably the best rooms to start with today.
ghm2180 2 days ago
One small suggestion: Make the code open to the right instead of a new window, that makes it easy to keep context without distraction.
- storagebits 2 days ago
  Good idea, thanks! A side panel is the way to go.
  - storagebits 2 days ago
    Shipped, thanks to you: the copy-on-write room now opens its sources in a side panel. Your comment went from here to production in about four hours. Others rooms to come. The rendering is far better!
ghostly_s 2 days ago
If you can't be bothered to copy edit the gratingly AI prose, I can't be bothered to read it.
AyyEye 2 days ago
> I built
You keep using those words. I don't think it means what you think it means.
- storagebits 2 days ago
  Fair hit. "Directed" is closer: agents wrote the code and most of the text, I chose the subject, steered and reviewed.
  - Brian_K_White 2 days ago
    Nobody wants to land a "hit". They were hit. The hit was being shown something as though it were valuable, only to discover it is not.
  - atmosx 16 hours ago
    And yet, if you didn't do all that steering, etc... this wouldn't exist. So let them "be" little sore ppl.
    ps. What I noticed is that those who created lots of projects before they'll be doing lots of projects after LLMs. Those who used to whine before, they'll keep whining after...
  - gspr 2 days ago
    You were a manager, in other words.
- ByteAtATime 2 days ago
  Inconceivable!
iberator 21 hours ago
ZFS is useless. No bit rot protection at all. ZFS just informs you about broken files (like FAT32 3 decades ago) but can't fix them... There should be some parity data added or some Salomon Reed forward correction of data... Waiting for such a feature for a decade now