Someone mentioned that FPGAs though amazing to create bespoke chips that are fast, run into a fab capacity issue. Someone explained briefly to me that even if you could make an fpga that could do inference better than current GPUs it was economically possible to make them at scale. I can't recall whether that was for technology related issues or just because fab capacity is completely filled up.
With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
'FPGAs should only really be used for'
You're list is focused on using and FPGA as a processor but IME FPGAs are mostly used when you need massive precise IO along with precision timing . Or if you need any sort of reconfigurable glue logic or things like that.
> With FPGAs, you're paying for reconfigurability, which hurts in terms of cost, speed and power.
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
Isn't it like FPGAs use more transistors per equivalent transistor it simulates?
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
The overhead of FPGA Vs ASIC is in the order of 10x to 100x, depending if you count logic gates, chip area, routing overhead or power.
This compares for the same technology (chip feature size) But for your chip you might not be able to pay for the smallest nm chip generation. That might make up for some of the gap
Their logic is based on lookup tables of the logic block, rather than the direct physical logic gates! So it's HUGELY inefficient for die area, like a few thousand transistors each rather than, say, 6 for a OR gate! But, the software won't waste them on a single gate, instead combining many connected logic elements into one "cell". For example, Xilinx has 6 inputs and one out, with 64 element LUT, which still a terrible, more than a 50x increase in die area, and therefore cost.
A 6 LUT is well below a thousand transistors. Roughly 800 transistors or less. You need 64 bit for the SRAM which is 360 transistors. Double that for the multiplexer and various other features like fracturing or shift register features.
And that's a big LUT, more economical FPGA designs have smaller LUTs. For example a 5 LUT would only need 32 bits of SRAM which means it is half the size.
One issue is that FPGAs are very fine grained and they need to gaurentee an image the tool confirms works for a chip must work across all produced chips. There is a level of binning for speed and binning for some disabled portions of the chip but it is a lot less flexible then disabling shader cores, memory controllers, and portions of the cache on a GPU.
soon people like this won't be spending money on tokens because they will have synthesized their entire hardware, driver, software, inference engine and harness stack running at a large enough context window with high enough tokens/sec, with their chosen LLM modifying its drivers, software, inference engine, harness and its own weights for their use case, re-initializing itself and moving on
This is exciting, I have been wishing for good ternary models for a while as it seems more effective and combining it with FPGA I think this is a super promising path forward.
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
Investors / debtors, same as Uber / Lyft before they went public.
But also "cost" and "API price" are different. The actual costs of inference will likely rise once Anthropic / OpenAI burn through their cash pile and investors want profits instead of competition for market share.
API price is high and fixed. API users have inference resource priority over subscribers.
Or... it may just be a lot less underpriced than subscriptions.
Both Anthropic and OpenAI are believed to be heavily subsidizing inference as they compete for market share. Once their shareholders demand profits over market share, the pricing model will rise (or mix in other revenue streams like ads).
Over the week or so I have vibe coded a Lean-like language with
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.
I used to run a medium sized adwords company that managed landing pages, adwords, and lead generation for about 50 companies. I moved into engineering at a bigger company after a while, but since LLMs I have essentially recreated all of the page builder, custom domain hosting, lead gen, CRM tools, and adwords integrations I used to subscribe to as my own platform and started taking on customers again on a home grown platform and its WAY better. I say all this just to say that I agree, and that even if you know the domain it still works and verifiably so.
> The Slurm Workload Manager, formerly known as Simple Linux Utility for Resource Management (SLURM), or simply Slurm, is a free and open-source job scheduler for Linux and Unix-like kernels, used by many of the world's supercomputers and computer clusters.
But it helps, it’s shorter and you don’t have to know what exactly is making that thing slowl (slow owl).
In my Hobby project (cv generator) I don’t like the ui to be unresponsive during pdf generation and generally I want to have a fast system. So I set my cpu under heavy load (or put it into power saving mode where the cpu is running at 0.9ghz) and tell opus to make it faster.
This went from 45s under very heavy load to 5 sek
(Heavy load: almost 100 cpu constantly and 90% ram filled up, additional power saving mode on)
Claude often told me the system is busy and „this,that“-chat. But I could resist stopping saying „make it faster, don’t stop any processes. It’s your new reality. We’re now sharing out cpu with the neighbor university. Live with it“
And yes. Heavy load. Python. Running blizzfast now. It were like 10 millions of testing and code and git commits tokens and 5h to 10h to get it there. Somewhat 500 tests written for the speed optimization. With background knowledge I still would not know where to optimize and would need to tell it exactly what to test and to compose tests for
I checked and actually it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
> it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
Ok, let me know how that performs, especially for basically any bit manipulation task or true parallelism. How do you control the precise timing of execution?
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
FPGAs should only really be used for:
Pre-silicon verification
Designs where power/cost aren't important (a lot of high end industrial or military for example)
Or low quantity designs that won't make an ROI when taping out an ASIC.
That said, I'm currently designing one into a product because at the moment, the cost of an ASIC is too much of a gamble (and also makes it too much of a black box for functional safety certification)
I also got a couple of different LLM's running on a commodity FPGA, and hit 59,000tok/s with t=1, and 23,000tok/s with t=128. I say LLM, but the model was 3M params, so pretty dumb! But, to hit that speed, I couldn't use the DDR4 as it's bandwidth was the bottleneck.
I currently have an 8b model running with the DDR in the loop, but it's stuck at 14ish tok/s, but the model is usable.
Hbm really is the magic that makes it work (economically)
In that vein, what happened to HP's Field Programmable Nanowire Interconnect (FPNI)? At the time, the claim was that it had the potential to outperform ASICs in cost, speed and power.
And therefore only useful when the real CPU/GPU/ASICs don't exist or are highly inefficient for some reason, like you have to have a super wide bus signal to apply a known set of parameters in batch or something?
This compares for the same technology (chip feature size) But for your chip you might not be able to pay for the smallest nm chip generation. That might make up for some of the gap
In ye olde days, it was a universal gate an a few SRAM bits to configure each one.
And that's a big LUT, more economical FPGA designs have smaller LUTs. For example a 5 LUT would only need 32 bits of SRAM which means it is half the size.
Also, a LUT is more powerful than an OR gate...
It seems implementing transformers is the new "hello world" of AI assisted coding.
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
But that still not a breakthrough or anything for such a small active set of parameters that is quantized
Zero surprise it did it
But also "cost" and "API price" are different. The actual costs of inference will likely rise once Anthropic / OpenAI burn through their cash pile and investors want profits instead of competition for market share.
API price is high and fixed. API users have inference resource priority over subscribers.
Both Anthropic and OpenAI are believed to be heavily subsidizing inference as they compete for market share. Once their shareholders demand profits over market share, the pricing model will rise (or mix in other revenue streams like ads).
> As revenue has scaled over the last year, compute costs have fallen from USD 2.01 to just USD 0.48 per revenue dollar
https://global.morningstar.com/en-eu/stocks/where-anthropics...
- quantitative types
- a Rust-ish DSL that lowers into the base language
- a C, LLVM, WebAssembly backend and a Verilog backend for a subset of the language
- A RISC-V implementation written in the Verilog subset that runs Doom on my Arty FPGA.
- A category theory and real number library
- A graph storage engine with queries written using profunctor optics
- A datalog with an egglog in progress
- A Joern-like code analysis engine for C and C# using tree-sitter to parse
- A React style framework
- Editor plugins for my language
All at varying stages of completeness.
What you can do now is wild. My fear is a sort of Gell-Mann Amnesia effect where my lack of expertise in the area is causing me to miss major problems with it. But at least some part of it has to be real. I plan to announce it in a more official capacity once I've allayed those concerns (https://wiz.badlydrawnwizards.com for a sneak peak, but sorely need to update the website)
My bottleneck currently is CI capacity. I've turned every spare computer in my household into a Slurm node so I can run more testing locally to avoid wasting precious CI capacity.
https://en.wikipedia.org/wiki/Slurm_Workload_Manager
Common man, prompt at the level of someone with at least a little of background knowledge.
In my Hobby project (cv generator) I don’t like the ui to be unresponsive during pdf generation and generally I want to have a fast system. So I set my cpu under heavy load (or put it into power saving mode where the cpu is running at 0.9ghz) and tell opus to make it faster.
This went from 45s under very heavy load to 5 sek
(Heavy load: almost 100 cpu constantly and 90% ram filled up, additional power saving mode on)
Claude often told me the system is busy and „this,that“-chat. But I could resist stopping saying „make it faster, don’t stop any processes. It’s your new reality. We’re now sharing out cpu with the neighbor university. Live with it“
And yes. Heavy load. Python. Running blizzfast now. It were like 10 millions of testing and code and git commits tokens and 5h to 10h to get it there. Somewhat 500 tests written for the speed optimization. With background knowledge I still would not know where to optimize and would need to tell it exactly what to test and to compose tests for
Not with any degree of performance.
Checked what, an LLM?
> it turns out that it is easier simulate LUTs on a GPU than it is to directly simulate logic gates because LUTs are data and GPUs are SIMD, meaning LUT simulation lets you build and run one kernel for a whole batch of combinatorial logic.
Ok, let me know how that performs, especially for basically any bit manipulation task or true parallelism. How do you control the precise timing of execution?
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
Every big US+ and Versal chip in existence idles at more than 5W. Samsung Aquabolt HBM uses far more than 5W (even at 100GB/s.)
https://architectlabs.com/blog/frostbite
https://architectlabs.com/blog/redwood