I think the meaning of can't hallucinate in this model is that the type won't be hallucinated.
So if the generated schema is for a tool call for calculator, then the numbers will be valid numbers for sure (and not random words).
To me, it looks similar to BNF schema already introduced and implemented few years ago: generally speaking - it limits the next token that is allowed to be generated, probs are drawn from a subset tokens.
(tbh, I'm not sure why it didn't pick up as a more standard interface to LLMs, as it made a lot of sense back then, and now.)
Yeah, I thought about constrained generation as well. I've actually done something similar with local models before. And you can even get a "confidence" score by looking at the logits (something along the lines of logprob("YES") + logprob("Yes") + logprob("yes") - logprob("NO")...
There's also a cheeky "one of the models hallucinated a link" in the wiki jump example that most likely could have been avoided by properly using grammars. You can setup constrained gen so that only valid options (say from a list) can be outputted. Their own inference lib likely does that. So comparing to one that doesn't is a bit cheeky.
That being said, after a brief look at the site I could see this working. Especially if this can be ran locally, the speed and cost can enable some workflows where you have this as an "overseer" layer over say a cli agent. After each step you run through a list of "questions" ("is the task completed?" -> yes -> "does the edit touch files it shouldn't" / "does the edit follow our code writing policies") etc.
edit: extra points if the "question" rubric is also generated by a higher abstraction model. Say "/goal Build out auth" -> generate_rubrics(goal) -> "Is auth implemented on all endpoints" / "Has code touched anything else than auth" / "is this following the best practices" / ...
AFAICT it is the same interface as you describe, but the underlying inference algorithm is fundamentally different, hence the speed gains. There is an application I am currently working on right now where this typed output predictor is the performance bottleneck. I'd be very interested to see how this performs.
It did, no? If you send a JSON schema to an LLM endpoint, at least for OpenAI and Anthropic, they turn it into a grammar behind the scenes and use it for constrained decoding.
Worse, if you watch the GN video this article is based on you'll find the TV can figure out how to leverage other non obvious networks (threads, etc) to reach LG servers. Worth a watch to see how bad this is: https://www.youtube.com/watch?v=6IFVTcM28KA
So - in theory - something like a massive chat client, discord like, can be implemented via this solution?
And what would be the pricing of such a solution.
Cheap-serverless-discord
Thank you! I spent a decent chunk of time designing PicoMQ, so the documentation had a natural progression.
Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase.
The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes. For some raw numbers, assuming 1M messages/day, 200B per message, ~6GB/month, all-inclusive, it would be $30 to $150 a month, and storage would be the cheapest part.
Yeah, much of the base cost I mentioned was from the compute, networking and S3 write costs, which could sustain a lot more messages for sure. It also depends on the number of active streams/rooms, the throughput bursts, as opposed to a steady state.
But as we speak, I am in the process of running open-benchmarks on AWS with cost attribution. I'll be posting them in the docs with transparency soon enough.
Read and write through cache already exists today, which would work in favour of both cost, latency, and consumer reads fanout.
I like the HN Mirror over a Discord-like app for simplicity. No doubt that's where my weekend is going!
> They may only have bought themselves 6 more months of time given their purported burn rate
If they had only ~6 more months they (+auditor) had to issue a warning.
The 6 is not a hard number, AFAIK, but surely a point where it must be reported.
That was indeed my intent. Pre-IPO, SpaceX already had _some_ cash in hand, and was burning >4b per quarter IIRC; so presumably _some_ runway. That said, they also got the anthropic/openai monthly payments coming in soon (already started maybe?). The next earnings release will be the interesting one IMO.
Historically, vehicles actually did have completely interchangable motive components. They were called horses.
It would have been difficult given the state of other technology at the time for the inventors of the internal combustion engine to have supplied it as a drop-in horse replacement for your carriage, but you could kind of imagine that working with current technology.
I don't like the impulse to regulate, especially as a formative measure. Regulating new technology before industry has time to experiment seems like a recipe for high prices, few choices, and low competition.
Apparently there's an official reboot of the game:
https://www.reddit.com/r/airmash/comments/1whwxxg/the_offici...
reply