Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 2 months agoSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squaremessage-squaremessage-square3linkfedilinkarrow-up118arrow-down12
arrow-up116arrow-down1message-squareSpent two weeks on a kernel that benchmarked 29x faster. End to end it's maybe 6-10%, and it's not even wired in yet.plus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 2 months agomessage-square3linkfedilink
Shifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 2 months agoBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squaremessage-squaremessage-square0linkfedilinkarrow-up15arrow-down11
arrow-up14arrow-down1message-squareBitNet b1.58-2B-4T at 36 tok/s on Xeon - pure C99, no Python/BLAS, single binary for ternary + GGUFplus-squareShifu@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 2 months agomessage-square0linkfedilink
