GPU-unlock: FP/BF16 instruction issue rate on Nvidia 3000's cards (github.com)

Zagitta 14 hours ago

• aunty_helen 14 hours ago

  Thanks so much. 27b is a favourite model and last night my dev ran some benchmarks to achieve 133tks about 3.2x which changes the economics.

  • xenonite 13 hours ago

    great to hear, which quant + KV quant + KV size are you on? is it llama.cpp?

    • aunty_helen 1 hour ago

      Here's the writeup from the experiments that claude did: https://claude.ai/artifact/X6WJVtKdL1LrhYuCMqdXvG#b9a4d3e2-8...

• jacquesm 14 hours ago

  Super well done, I wished all mods took your approach. Patched drivers are a nuisance, especially if you have more than one kind of card. Thank you.

  edit: now do the 4090 ;)

  • Zagitta 14 hours ago

    Yeah I hate the patched drivers too, I have a CMP170 as well I want to see if I can find a way to get fully working the same way without or minimal driver patching.

    4000s is WIP, forks has already unlocked it but I'm doing some refactoring to better support different families and after that I'll port it over :-)

    • jacquesm 13 hours ago

      Can't wait, I have two 48G modded ada 4090's and 4 stock Blackwell 4000's (though I doubt those will be as easy). The 4090's alone would be an amazing thing to have boosted, they're my main workhorses and effectively any improvement there will immediately make my days more productive. If I can help out with testing, my email is in my profile.

      • Zagitta 13 hours ago

        Blackwell and hopper have a whole new boot process so those are unlikely to fall over anytime soon from what I've seen.

        If you can't wait for 4090 someone has a fork with a seemingly working unlock :D

        • jacquesm 13 hours ago

          Yes, but it's pinned to a driver version that I unfortunately can not use. That's precisely why I love your solution, it is the best way to do this.

          I'm running 8 GPUs attached to a single CPU and that driver is a pretty essential part of getting this to work with P2P between as many cards as possible. The modded 4090 is a problem because the BAR1 it advertises (32G) is smaller than the VRAM so not all software is able to use it (you need two memory segments rather than a single 48G one to make this work). That driver is part of the solution for this problem.

          • Tiberium 13 hours ago

            FWIW the gpu unlock ada repo doesn't pin to a driver version, Codex for me just compiled it, and it works with my 615.xx driver even though the fork is for 610.x. I think the repo just uses the firmware from that version. My GPU is 4070ti super.

            • jacquesm 12 hours ago

              Ok, trying it. I'm using a pretty heavily modified 590.48.01 to be able to use those 48G 4090s, they work but they do require a bit of handholding. The main problem is the BAR1 just isn't large enough, it's capped at 32G.

              Heh, got it working, thank you so much!!

              Overall 30% reduction in wall clock time, that's pretty neat (and I'm still not running this at full power but 450 W is a bit much for that puny connector).

              • SlavikCA 7 hours ago

                I have two of modded RTX 4090 with 48GB VRAM. But I'm using default drivers. I'm using models, which uses almost full 48GB.

                Now i'm curious - am I missing some settings? Do I need to check any of these BAR things?

                • jacquesm 7 hours ago

                  You're probably not using peer-to-peer then. Try the p2pbandwidthtest that comes with the NV development package to see if you are or not.

          • Zagitta 13 hours ago

            If they are forked from my repo whatever driver version they pin doesn't matter, the pin is just clanker stuff really. What it's used for is just extracting the right booter firmware and loading it, but the whole booter part is reset before handing over to nvidia driver so the versions dont have to match at all.

            • jacquesm 12 hours ago

              Apparently they were, it compiles and installs but I don't see any improvement just yet. I'll keep plugging away at it.

        • marcosscriven 8 hours ago

          https://github.com/JustAResearcher/gpu-unlock-ada

          • jacquesm 7 hours ago

            That's a fork of GP's work.

    • mietek 13 hours ago

      What about the 5090?

      • Zagitta 13 hours ago

        Hopper and blackwell have a very different security chain so those are out of scope for this approach at least.

        But I would suspect China has figured something out given the reports of them buying 5090s in droves and seemingly modifying them.

        • jacquesm 12 hours ago

          I've looked a lot at that and it seems to be mostly related to packaging rather than performance, as far as I know there are no 'improved' 5090's for sale in any kind of volume, and the ones that they made that I know of (a 64 and a 96 G version) never made it to production for some reason. I'd love for that to happen though.

          • Zagitta 12 hours ago

            Maybe they got hold of the vbios for those higher vram versions similar to the 48gb 4090s?

            • jacquesm 12 hours ago

              There is a lot of speculation about this but nobody seems to really have hard insight. As long as the hardware supports it it should in theory be possible to unlock it but like with those 48G 4090's that can lead to weird side effects. For instance, you would never know that these cards are modded until you start using p2p and then suddenly it all falls apart. The reason is that for p2p to work they map the memory space of one card into the memory space of another. But if the BAR1 is < than the total VRAM that trick will work only until you run past the TOP of that block and then you'll have a crash.

              I suspect that modding a 5090 in such a way that you can sell it is a lot harder than doing the same with a 4090.

            • az226 1 hour ago

              48gb 4090 uses stock vbios

  • marcosscriven 8 hours ago

    https://github.com/JustAResearcher/gpu-unlock-ada

• verdverm 8 hours ago

  Any idea if something like this is happening on GB10 (DGX Spark) and needs similar treatment?

  edit: seems you have provided some clarity that even if, unlikely soon unless China has and does the world a favor

  • jacquesm 2 hours ago

    DGX Spark is going to be very hard.

RachelF 11 hours ago

• jacquesm 11 hours ago

  Indeed, I already posted it earlier but this definitely deserves a top level comment.

wewewedxfgdf 2 hours ago

jacquesm 15 hours ago

• Zagitta 14 hours ago

  It's quite nuanced. In a nutshell the unlock just flips the 2 bits that halves how often the core can issue the specific instructions Nvidia locks to full rate on the professional and datacenter cards.

  Obviously doing more compute requires more power but what it does is shift the card down on the frequency curve to keep the wattage in check. So you're right that at stock power limits it hits a cap where the full issuing rate isn't reached but it depends heavily on the exact workload how much juice is left.

  The frequency curve also takes heat into account so in general you end up with a nice perf boost for a slightly hotter card.

  I've been running this on my 2 3090s for a month now with no issues but they are also water cooled :-)

  • jacquesm 13 hours ago

    Ah, that's a twist I didn't see coming, all my GPUs are aircooled. I can see how with watercooling you'd be able to get more performance out of the GPU before hitting the limits.

  • BoorishBears 12 hours ago

    There 3090 is from the era of overclocking, anything but the reference design should be fine even on air cooling (especially if you have a 3 connector model)

    My EVGA came with stock firmware that started at a 420W and allowed up 500W, of course 500W did nothing for gaming

• Tiberium 14 hours ago

  > - you will need to run your card hotter than you might be comfortable with

  Can you explain this one? On my GPU I didn't have to change the power limit. Obviously the card might spend more time at max power usage, but it's still within the limit.

  • jacquesm 14 hours ago

    If you increase the power limit on the card you will be able to get much more mileage out of this modification. At 450W the difference is much, much larger but that is not something I'm comfortable with doing for longer runs. Note that the stock settings are already pretty extreme for that cooler, and that ambient temp and free airflow are going to be impacting the resulting GPU temp in a very direct way. I have my 3090's in an external box (with its own separate PCIe backplane) so they can talk p2p and have 'fresh air' rather than in the enclosure.

    • Tiberium 14 hours ago

      Yes, but you can get up to a 40-50% increase in some workloads even with the same power limit, and some cards have it locked anyway. I just felt like it was a litle misleading for you to mention the power in your original post, since the unlock works regardless of that.

      • jacquesm 14 hours ago

        Interesting, I have 7 3090's here of different plumage and they all require a power increase to get > 25% gain with this modification, would you mind sharing the type of card you have and how it is cooled?

        Obviously, if there are 3090's out there that respond differently then I will be happy to retract that.

    • unsnap_biceps 13 hours ago

      Is your external box something you put together or is it a product you purchased?

      • jacquesm 13 hours ago

        I put it together myself. It was a lot of work and I learned a lot of nasty lessons along the way. The short version of the recipe:

        - get the largest PSU you can find (1500W at a minimum, more is better)

        - get the ADT PLX Quad PCIe backplane. Run 4 thin (single slot) GPUs inside your main enclosure, stick the ADT interface board in between two of the thin boards (don't stack them close to each other or they won't be able to breathe).

        - ensure you have as much or more RAM as you do VRAM

        - Ensure your CPU has enough PCIe lanes to run at least five boards in x16 (four for the GPUs in the main enclosure, another for the PLX PCIe switch)

        - In the BIOS set BAR1 to 'resizable' and > 4G decoding should be enabled.

        - ditch the cables that come with the ADT board, they may work but there are much better ones out there and they're not expensive, this will save you days of troubleshooting

        - Build your own enclosure rather than trying to adapt a standard one, it won't work anyway (the board pitch is wrong, as is the placement of the mounting holes).

        - Put another four double or triple boards on the backplane.

        - Add another 3x 140 mm fan to the front of the case so you get good airflow without the feeling of being next to a Harrier at take-off

        - I used 3D printed parts to mount the PSU and the board as well as the fans (see below).

        - Make sure the board is properly grounded to the enclosure

        - If you're running 3090's or other cards that are P2P capable that were hobbled by NV then look at this https://github.com/aikitoria/open-gpu-kernel-modules driver module.

        - Power on the ADT board, wait three seconds then power on the main machine. Enjoy your VRAM :)

        If you have more questions please email me and I'll be happy to help out.

• augment_me 14 hours ago

  These number claim 240W locked power draw on 4070:

  https://github.com/JustAResearcher/gpu-unlock-ada/releases/t...