> And the rest of automakers offerings aren’t half as good.
This was true maybe even as recently as 5 years ago, but it certainly isn’t true now.
Tesla, at the top end, hasn’t been an attractive luxury proposition at least since the Hyundai Genesis & Mercedes EVs started rolling out. They had a shot at capturing the mid- to low-end market, but it looks like they’re in the process of blowing that as well.
> Hard to do in times of austerity [..]
The average American’s lifestyle is hardly austere — it _is_ precarious for very many (most?), but I don’t think that’s the same thing.
You can now get refurb EVs (e.g. a Hyundai Ionia) with <50k miles for <$15k, and that’s in a not-inexpensive part of the US (northeast).
Over the course of the next 10 years that used market is going to only grow, and I don’t think it’s unreasonable to expect that a battery swap will be less costly than the sorts of overhauls high-mileage gasoline cars require so there _will_ be a solid used market.
do not listen to someone like DHH, who loves to speak at length on topics that he is deeply unfamiliar with.
> Mortality in children, adolescents, and adults with attention deficit hyperactivity disorder: a nationwide cohort study
>
> ADHD was associated with significantly increased mortality rates. People diagnosed with ADHD in adulthood had a higher MRR than did those diagnosed in childhood and adolescence. Comorbid oppositional defiant disorder, conduct disorder, and substance use disorder increased the MRR even further. However, when adjusted for these comorbidities, ADHD remained associated with excess mortality, with higher MRRs in girls and women with ADHD than in boys and men with ADHD. The excess mortality in ADHD was mainly driven by deaths from unnatural causes, especially accidents.
reposting my comment from another time this discussion came up:
"Cosmopolitan has basically always felt like the interesting sort of technical loophole that makes for a fun blog post which is almost guaranteed to make it to the front page of HN (or similar) purely based in ingenuity & dedication to the bit.
as a piece of foundational technology, in the way that `libc` necessarily is, it seems primarily useful for fun toys and small personal projects.
with that context, it always feels a little strange to see it presented as a serious alternative to something like `glibc`, `musl`, or `msvcrt`; it’s a very cute hack, but if i were to find it in something i seriously depend on i think i’d be a little taken aback."
The problem with this take is that it is not grounded in facts that should determine if the thing is good or bad.
Logically having one thing instead of multiple makes sense as it simplifies the distribution and bytes stored/transmitted. I think the issue here is that it is not time tested. I believe multiple people in various places around the globe think about it and will try to test it. With time this will either bubble up or be stale.
i can't comment about sora specifically, however the architecture can support workloads beyond just LLM inference.
our demo booth at trade shows usually has StyleCLIP up at one point or another to provide an abstract example of this.
disclosure: i work on infrastructure at Groq and am generally interested in hardware architecture and compiler design, however i am not a part of either of those teams :)
i live in NYC and have traveled to plenty of other international cities.
none of the things that you're saying are true compared to my experiences (or those of my friends) in any way that i can think of as meaningful.
the only city i've been to that feels like it's captured the same "vibe" as NYC, for me, has been Paris.
Tokyo was more impressive in its sprawl and history (and obviously cleanliness), but there is a sense of Japanese monoculture that saturates everything in a way that is almost tactile. not in a bad way, but definitely such that i felt like something was "missing" during my visit.
Singapore gets really close to the same feeling, but for all of its heterogeneity there's an undercurrent of authoritarian sterility that made it very difficult to feel comfortable (Disneyland with the Death Penalty, indeed).
anyway this is already pretty long winded so i should probably stop talking, but NYC has a lot "going for it" besides the rest of the US just sort of being a suburban hellscape. at some point i'll move out, but living here has been a really comforting reminder that international views such as yours of American cities are incorrect.
I was born in Manhattan and lived in the city for over a decade and still own an apartment downtown. I know a thing or two about the place. It's cool that you get a vibe from being a transplant here for a couple years, that has literally nothing to do with anything I said. The lawlessness is also quite a different experience for women--I am guessing having random guys off the street try to force your door open and follow you into your building or corner you on a subway or follow you around riding a bike aggressively catcalling you is probably not something you are dealing with on a regular basis.
The day I left I moved out over a pool of dried blood from a stabbing in front of my door the night before. I've lived in over 20 countries since then and not experience anything similar except maybe in Canada, which has similar drug problems as the US.
With the caveat that I moved away (due to work) a little under a decade ago... what you describe doesn't match my experience with NYC at all. Maybe back in the 80s, before it was cleaned up... but I was less frequently there back then. Before you said you lived in the city, the message from your first post made me assume you were talking about the city as someone who learned everything they know about it from the news.
Visiting another city is not in any way comparable to living there. Or would you defer to the opinion of some tourist who visited NYC for a random weekend?
It's not fixed and our chip wasn't designed with LLMs in mind. It's a general purpose, low latency, high throughput compute fabric. Our compiler toolchain is also general purpose and can compile arbitrary high performance numerical programs without the need for handwritten kernels. Because of the current importance of ML/AI we're focusing on PyTorch and ONNX models as input, but it really could be anything.
We can also deploy speech models like Whisper, for example, or image generation models. I don't know if we have any MOE architectures, but we'll be implementing Mixtral soon for sure!
Will you be selling individual cards? Are you looking for use cases in the healthcare vertical (noticed its not on your current list)? Working in the medical imaging space and could use this tech as part of the offering. Reach out at 16bit.ai
I think we use a system with 576 Groq chips for this demo (but I am not certain). There is no DRAM on our chip. We have 220 MB of SRAM per chip, so at 576 chips that would be 126 GB in total.
Graphics processors are still the best for training, but our language processors (LPUs) are by far the best performance for inference!
Our language processors have much lower latency and higher throughput than graphics processors so we have a massive advantage when it comes to inference. For language models particularly, time to first token is hugely important (and will probably become even more important as people start combining models to do novel things). Additionally, you probably care mostly about batch size 1. For training, latency is not the key issue. You generally want raw compute with a larger batch size. Backpropagation is just a numerical computation so you can certainly implement it on language processors, but the stark advantage we have over graphics processors in inference wouldn't carry over to training.
Everything you say makes sense. Training is definitely more compute intensive than inference.
Training is both memory throughput and compute constrained. Much research in speeding up training goes into optimizing HBM to SRAM communication. The equivalent for your chips would be communication from the SRAM of one chip to the SRAM of another, where it sounds like your architecture has a major memory throughput advantage over GPUs. So I assume you don't have a proportional compute advantage?
By the way, it's great to see a non von Neumann architecture showing a major performance advantage in a real world application. And your chips are conceptually equivalent to chiplets; you should have a major cost advantage on bleeding edge process nodes if you scale up manufacturing. Overall very impressive!
I'm not an expert on the system architecture side of things. Maybe a Groqster who is can chime in. But the way I understand it is that you can't improve latency just by scaling, whereas you can improve throughput just by scaling, as long as it's acceptable to increase batch size. Increasing batch size is generally fine for training. It's a batch process! On the other hand, if someone comes up with a novel training process that is highly sequential then I'd expect Groq chips to do better than graphics processors in that scenario.
unless i'm misunderstanding `whisper.cpp` seems to support streaming & the repository includes a native example[0] and a WASM example[1] with a demo site[2].
have you tried it?
i mean for fun, it wouldn't hurt for sure and ggerganov is doing amazing stuff. kudos to him.
but whisper is designed to process audio files in 30-second batches if I'm not mistaken. it's been a while since whisper released, lol. These workarounds make the window smaller but it doesn't change the fact that they're workarounds. you can adjust, modify, or manipulate the model. You can't write or train it from scratch. check out the issues referring to the real-time transcription in the repo.
can you use it? yes
would it perform better than Deepgram? -although it's an API and probably not the best API- I am not sure.
would i use it in my money-generating application? absolutely not.
Especially when it comes to smuggling drugs across the border.
Of course, we could fix the drug smuggling by decriminalizing drug addiction and treating it like other public health issues. But the "War on Drugs" still rages on, after decades of failure.
This was true maybe even as recently as 5 years ago, but it certainly isn’t true now.
Tesla, at the top end, hasn’t been an attractive luxury proposition at least since the Hyundai Genesis & Mercedes EVs started rolling out. They had a shot at capturing the mid- to low-end market, but it looks like they’re in the process of blowing that as well.
> Hard to do in times of austerity [..]
The average American’s lifestyle is hardly austere — it _is_ precarious for very many (most?), but I don’t think that’s the same thing.
You can now get refurb EVs (e.g. a Hyundai Ionia) with <50k miles for <$15k, and that’s in a not-inexpensive part of the US (northeast).
Over the course of the next 10 years that used market is going to only grow, and I don’t think it’s unreasonable to expect that a battery swap will be less costly than the sorts of overhauls high-mileage gasoline cars require so there _will_ be a solid used market.