@System64fumo

🇺🇸🇮🇱 Icy fairy goofing around with embedded systems for fun!

 Israel
github.com/System64fumo
Joined May 2016
  • Tweets534
  • Following112
  • Followers94
61Media
Hello everyone! I'm pleased to publicly announce a project of mine that I've been working on for the past 2 months. github.com/System64fumo/kapp…

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
3
3
1
30
2,333
Compute is expensive.. Let's make it cheaper, And local!
1
3
82
I cooked hard, Here are some things that landed in kappai recently: GLM 5.3 Flash support OpenGL ES 3.1 backend Several new IQ quants TUI setup/run screen & many more There's also a new matrix and discord for this project, Come join us! github.com/System64fumo/kapp…

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
1
22
I don't actually use discord btw so if you want to contact me do so on matrix.
7
Holidays are over, Time to cook again. Where are all my cracked C devs at? 👀
19
Ok so i didn’t expect to get this much attention, But damn those are a lot of eyes. 👀 I’m out for holidays right now so sorry for the lack of updates. Thank you Bee fumo! 🐝
2
59
I have some pretty telling evidence that Union Alpha is some sort of GLM model. @Zai_org Can we please "pace the frontier"? /s And while you're at it could you kindly fix the glaring security hole? I wanted to work not dive down this rabbit hole, k? thx.
1
113
Happy #CirnoDay!
2
100
4
83
770
10,348
I don't know what others think but i find it quite amusing that i made a 12 CPU (CD8180) go toe to toe with a 20 core iGPU. (M4 Pro mac mini) Wish i had proof for this but the proper comparison with GLM 5.2 gave me: 0.38PP and 0.32TG Numbers from colibri:
1
147
Also please don't get me wrong, I'm not throwing shade at colibri or llama.cpp. They're great projects, I just can't benchmark properly since i don't have disk space nor memory for either. Sorry if that came off wrong. 😅
76
Humm maybe claiming to have a performance oriented inference engine followed by showing sub token performance on a pretty banger aarch64 system doesn't quite showcase the improvements.. Fine here are, Some direct 1:1 comparisons with llama.cpp on rk3588. 🧵
1
1
129
more replies
I also have an NPU backend that does indeed run here but unfortunately i can't publicly ship that code since my license is M.I.T and the NPU code is based on GPL.
1
31
Note: Just like my last post i had to re-create this post too, Again sorry!
26
Before i conclude i want to be crystal clear about some things. 1. I am not an AI/ML engineer, I am just a dude who likes to optimize things. 2. The codebase is primarily generated using @Zai_org's amazing models and i couldn't have done it without them.
1
1
127
PS: Had to re-upload the images and re-create the post because twitter's drafts made them super low resolution, Sorry about that! 😭
1
112
Load more