The conversation examples are impressive but this made me say holy shit out loud:
"it runs locally on a MacBook Pro M4 Max or better. And we figured out how to train Dessa on only one NVIDIA GB200 NVL72."
If it really can replicate that low latency + that expressiveness on local devices, that's one heck of an accomplishment for "indie" AI.
Speech is in quite a weird spot when it comes to locally running it. Some of the best STT and TTS models run at faster-than-realtime speed on decent consumer hardware with quite small models (so also no huge RAM requirements), but the tooling and library ecosystem is much more fragmented, making it a lot harder to build actual working products around it. There is also a much more noticeable coverage / quality gap when it comes to languages other than English.
As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
> Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.
I don't know whether people are just ignorant but it is heavily likely say the S2S without any intelligence can run on a pro-sumer device. Just because it is "OpenAI" and "Google" and not public knowledge or will be open source/public weights doesn't mean you can't reasonably infer.
Probably Dessa started those demo conversations with some context, so maybe it could receive documents and speak about them. If those were random conversations started with no context, the people at Block probably recorded a lot of uninteresting chat and cherry picked the few good ones.
But the thing that makes this special, is explicitly cutting things out that make LLM's capable of that. As gpm's sibling comment points out, if you would add that back in, you would just end up in the converging to the same place (with the same downsides).
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
> be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.
This thing has the "the natural language nailed" to the same degree that a BERT has. That's a far cry from the intelligence level required to output text of any sensible length, and even farther from the structural adherence required for tool calls.
Highly-structured outputs (like what you'd need for tool calls) seems like it'd be easier to put out than arbitrary text, no? At the very least, even something like BERT could make the decision of “Would any of my available tools be useful here?” and delegate to some deeper-thinking model to structure that tool call.
> > one NVIDIA GB200 NVL72
That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).
that’s a good clarification, skimming some of the comments here I was thinking they trained it on a single GB200 gpu. THAT would be a huge win for indie AI
Edit: Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems. The claim of “we trained this on a single rack of gpus” is definitely much less impressive than “we trained this on a single gpu”
> Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems.
It's even more confusing than that :)
GB200 = 1 CPU + 2 B200 GPUs (packaged together)
NV72 = A rack of 72 B200 GPUs (or 32 GB200 superchips)
Think of GB200 as something like Apple's 'Ultra' chip. Not exactly, but similar. They fuse the chips together with a very fast interconnect (connection).
Also a speech to speech model it can't do any tool use or you can't guide it much am I mistaken? How will it be used for agentic uses is the part I don't get. Edit: this is response from Argon if anyone was also curious: You are not mistaken about Dessa: because it was trained solely on raw audio with zero text data, it cannot read text system prompts or output code for tool calls on its own. To use it as an agent, developers must either steer it purely through spoken voice instructions or pair it as a conversational "voice front-end" alongside a separate text LLM that listens to the audio and executes the tools in the background.
> they didn’t have to worry about hosting/billing/subscriptions or any of that yet
Or marketing, which they kinda get for free. So much research goes unseen, there’s huge competition for attention, and most research is very limited in that regard
I would say that’s the biggest factor that makes the team “not indie”
At the same time, like you said, still amazing accomplishment, very cool model, and great work
These bidi/duplex demos always come off as overly artificial to me. It's cool that the model can speak at the same time as you instead of taking turns, but training it to interrupt as much as possible isn't the way to do it. In the very first example, did anyone here find the laugh after the "winter" answer to be convincing? What was the need for it? Why did the human speaker not get to finish a single sentence or finish a thought before the model took over with a monologue?
I'm absolutely going to hate talking to these things as customer service agents, with fake vocal fry and odd amounts of intonation and emotion peppered in.
For sure there's gonna be people falling in love with them though! I find all the anthropomorphization creepy though.
I've been hearing people talk negatively about vocal fry for ages now with no idea what it meant. But this comment was finally the one that made me go look it up. And today I learned that I speak almost entirely in vocal fry.
I listened after reading your comment, and in the first half of the example, Dessa sounds drunk to me, or at least tipsy. So weirdly, the laugh made sense to me.
Maybe it's better with lower latency, but I get really annoyed by this with ChatGPT Voice. I almost always start my chat with "I'm gonna ramble, I'll tell you when it's your turn".
OpenAI ruined advanced voice mode for me with that. I’m in the middle of a sentence and it goes “uh huh” and it kills my whole train of thought. It does not make me feel like I’m talking to a real person, it makes me feel like I’m talking to a stupid stupid AI that I hate.
This is extremely valuable for language education.
This is like having on demand access to a conversational partner at all times. But there is no embarrassment factor, patience, or concern about wasting an instructors time. Because the model is trained entirely on audio and not a voice to text add on, you can provide a human extremely high quality feedback and training.
And it’s cheap to run.
It doesn’t need to be smart the way a SOTA LLM is smart. There’s tons of value here already. You can tack it onto another system for deep intelligence.
toddmorey 1 day ago
If it really can replicate that low latency + that expressiveness on local devices, that's one heck of an accomplishment for "indie" AI.
hobofan 19 hours ago
As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
yellowapple 8 hours ago
In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.
eadwu 11 hours ago
nextaccountic 4 hours ago
You could hook up a LLM to give it intelligence, if there were some interface
pmontra 18 hours ago
What prevents them from adding tool integration?
kenmacd 18 hours ago
I guess that's somewhat how we do it, so maybe it would work.
gpm 16 hours ago
Converging on what AI companies are already doing... just from the other direction.
xvector 18 hours ago
hobofan 15 hours ago
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
xvector 14 hours ago
It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.
hobofan 10 hours ago
yellowapple 8 hours ago
skhameneh 1 day ago
It's a nice accomplishment, but I think we have very different definitions of "indie".
https://en.wikipedia.org/wiki/Block,_Inc.#
sheepscreek 1 day ago
That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).
Many AI shops are funded well into tens of millions if not hundreds, for early rounds.
Also the team size is really small (< 10???). So yes, indie might be a stretch but still very impressive, relatively speaking.
Now taking the other side, they didn’t have to worry about hosting/billing/subscriptions or any of that yet. It’s a research demo at this point.
jedbrooke 1 day ago
that’s a good clarification, skimming some of the comments here I was thinking they trained it on a single GB200 gpu. THAT would be a huge win for indie AI
Edit: Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems. The claim of “we trained this on a single rack of gpus” is definitely much less impressive than “we trained this on a single gpu”
sheepscreek 23 hours ago
It's even more confusing than that :)
Think of GB200 as something like Apple's 'Ultra' chip. Not exactly, but similar. They fuse the chips together with a very fast interconnect (connection).michelsedgh 1 day ago
ctoth 1 day ago
nico 1 day ago
Or marketing, which they kinda get for free. So much research goes unseen, there’s huge competition for attention, and most research is very limited in that regard
I would say that’s the biggest factor that makes the team “not indie”
At the same time, like you said, still amazing accomplishment, very cool model, and great work
stevage 1 day ago
gnabgib 1 day ago
[0]: https://www.nvidia.com/en-us/data-center/gb200-nvl72/
[1]: https://www.tomshardware.com/tech-industry/artificial-intell...
dist-epoch 20 hours ago
tourmalinetaco 1 day ago
JohnHaugeland 21 hours ago
pdntspa 1 day ago
Open weights or GTFO
tgrowazay 12 hours ago
written-beyond 1 day ago
Still extremely impressive but not unheard of.
paxy 1 day ago
switchbak 1 day ago
For sure there's gonna be people falling in love with them though! I find all the anthropomorphization creepy though.
bityard 1 day ago
bityard 1 day ago
csmoak 1 day ago
preuceian 1 day ago
fy20 1 day ago
JoshTriplett 1 day ago
virtualcharles 1 day ago
iforgotmypasswo 16 hours ago
This is like having on demand access to a conversational partner at all times. But there is no embarrassment factor, patience, or concern about wasting an instructors time. Because the model is trained entirely on audio and not a voice to text add on, you can provide a human extremely high quality feedback and training.
And it’s cheap to run.
It doesn’t need to be smart the way a SOTA LLM is smart. There’s tons of value here already. You can tack it onto another system for deep intelligence.
karimf 14 hours ago
The last time I checked, open source full duplex model was still so far behind GPT-Live. So, I'm super excited about Dessa and I think it's very cool.
But I don't think it'll replace the classic cascade system for now for most use cases. A couple of reasons:
- Intelligence
- Tool calling
- System prompt?
- Language support other than English