what does it mean to be human? explain with precision
bro i miss 2024, was shitposting all the time. anything i was getting my head into, what an idiot.
gotta do it more
ry retweeted
Accelerator companies focusing on inference and tok/s at the expense of fundamentals is a serious mistake. LLMs are just a small part of all the things that will be built with AI in the future.
ry retweeted
5.6tok/s,
alr I just wanna play around.
Just the idea that i'm able to run it locally feels thrilling!
recently got a new laptop,
i could finally use my old potato calculator to play now around. gonna make this rock think. THINK MONKEY THINK.
i've 2gb of vram, w 8gb of ram, 117ssd.
so would have to offload many layers before my gpu screams.
model: Qwen2.5 3B
initially thought I'd use airllm, I've some
old nvidia gpu (later found wasn't configured for cuda), so am using only cpu inference.
But it took around 0.073tok/s 🥀 (due to layer by layer streaming)
Tried it with ollama, 5.631tok/s
Every now and n then a new story comes along & i kinda miss Assange












