Ricklee retweeted
fantastic simple visualization of the self attention formula. this was one of the hardest things for me to deeply understand about LLMs.
the formula seems easy. you can even memorize it fast. but to really get an intuition of what the Q,K,V represent and interact, that’s hard.
Ricklee retweeted
Build your own NotebookLM!
A personal learning assistant that can:
- process multi-modal data
- scrape websites and YouTube videos
- create a unified knowledge base
- lets you do RAG over it
- remember every conversation
- generate a podcast 🎙️
100% open-source.
I just built an open NotebookLM clone!
Here's what it can do for you:
- Process multi-modal data
- Scrape websites and YouTube videos
- Create a unified knowledge base
- Lets you do RAG over it
- Remember every conversation
- Generate a podcast 🎙️
The idea here is not to reinvent the wheel but to understand how one of the most powerful tools for learning and research actually works, by building it step-by-step!
So by the end of this video, you'll learn how to:
↳ Process multimodal data (text, audio, video, URLs, and YouTube videos) into a format ready for LLMs
↳ Store everything in a vector database for fast retrieval
↳ Add a memory layer that remembers conversations and preferences for a personalized experience
↳ Chat with your knowledge base or generate podcasts using a fully open-source, locally running text-to-speech model
The podcast generation feature is my favorite part!
There's something powerful about turning written content into conversational audio that you can listen to while doing something else.
The entire code is 100% open-source. I've shared a link in the replies!
____
Don't forget to drop a like if you enjoy my videos. It shows me I should be making more content like this.
Cheers! :)
Ricklee retweeted
Replying to @Qianaegean
刚理解错了以为是一个 GPU 可以部署多少个实例,如果是算用户请求的话,可以这样算:
# 总显存需求 = 模型参数占用 + KV Cache占用 + 其他开销
# 模型参数占用 = 参数数量 × 数据类型字节数
KV Cache占用 = batch_size × seq_length × hidden_size × layers × 2 × 2
# 最大并发数 = (GPU显存容量 - 模型参数占用) / 单个请求的KV Cache占用
# 单个请求KV Cache = seq_length × hidden_size × layers × 2 × 2 bytes
RTX 4090 (24GB),部署 Llama-7B INT4:
模型占用:3.5GB
可用显存:20.5GB
单个请求(512 tokens):约100MB (用户上下文长度、连续对话会显著增加 kv 占用)
理论并发数:~200个请求
Ricklee retweeted
if you’re confused about how large language models work this new animation from google deepmind should clear a lot of things up
Ricklee retweeted
Things Every Developer Should Know: Concurrency is 𝐍𝐎𝐓 parallelism.
In system design, it is important to understand the difference between concurrency and parallelism.
As Rob Pyke(one of the creators of GoLang) stated:“ Concurrency is about 𝐝𝐞𝐚𝐥𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 lots of things at once. Parallelism is about 𝐝𝐨𝐢𝐧𝐠 lots of things at once." This distinction emphasizes that concurrency is more about the 𝐝𝐞𝐬𝐢𝐠𝐧 of a program, while parallelism is about the 𝐞𝐱𝐞𝐜𝐮𝐭𝐢𝐨𝐧.
Concurrency is about dealing with multiple things at once. It involves structuring a program to handle multiple tasks simultaneously, where the tasks can start, run, and complete in overlapping time periods, but not necessarily at the same instant.
Concurrency is about the composition of independently executing processes and describes a program's ability to manage multiple tasks by making progress on them without necessarily completing one before it starts another.
Parallelism, on the other hand, refers to the simultaneous execution of multiple computations. It is the technique of running two or more tasks or computations at the same time, utilizing multiple processors or cores within a computer to perform several operations concurrently. Parallelism requires hardware with multiple processing units, and its primary goal is to increase the throughput and computational speed of a system.
In practical terms, concurrency enables a program to remain responsive to input, perform background tasks, and handle multiple operations in a seemingly simultaneous manner, even on a single-core processor. It's particularly useful in I/O-bound and high-latency operations where programs need to wait for external events, such as file, network, or user interactions.
Parallelism, with its ability to perform multiple operations at the same time, is crucial in CPU-bound tasks where computational speed and throughput are the bottlenecks. Applications that require heavy mathematical computations, data analysis, image processing, and real-time processing can significantly benefit from parallel execution.
--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/bbg-social
Ricklee retweeted
Je me remets en condition pour flag du app - sys. Elle était pas mal cette cheat sheet sur les Buffer Overflow ! :P
Ricklee retweeted
Info sheet: binary representations of numeric values commonly used by hardware architectures (esp. x86) and programming languages. Unsigned integers, two's complement signed integers and IEEE 754 floating-point.
#programming #coding #learning #cheatsheet #numbers #hardware
Ricklee retweeted
Preliminary block diagram for Zen 5 by AMD.
Still awaiting on some details that AMD didn't clarify, such as the ITA BTB, Store & Load Queue sizes, etc.
μarch Block Diagrams: bit.ly/32qLLew
μarch Cheat Sheet: bit.ly/2JTplfJ
#x86 #AMD #CPU
Ricklee retweeted
Kubernetes Native Sidecar Containers Explained 🔽
Kubernetes Pods support several types of containers:
- init
- regular
- ephemeral
- ...and since 1.28, "sidecars"
Learn about the difference between them and when to use the new "native" sidecar type 👉 labs.iximiuz.com/tutorials/k…
Ricklee retweeted
How does Docker work?
A comparison of Docker-based and non-Docker-based development is shown below.
With Docker, we can develop, package, and run applications quickly:
🔹 The developers can write code locally and then build a Docker image and push it to a dev environment. In this way, the development work is shared within the team. The tests are run automatically in the dev environment.
An image is a read-only template with instructions for creating a Docker container. We use a Dockerfile to define the steps to create the image and run it.
🔹 The above process can run incrementally when bugs are found or improvements are needed.
🔹 When dev testing is complete, the Docker image is pushed to the production environment (often on the cloud).
Compared with traditional development without Docker, Docker is quite lightweight and fast, because only the changed part of Dockerfile is rebuilt every time we make a change.
Over to you: Where are Docker images stored?
–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/3KCnWXq
Ricklee retweeted
How To Pick The Right Docker Image For Your Node.js App 👇
- Use an LTS release
- Avoid bloated images
- Balance simplicity/security
All mainstream image candidates with the usage advice on a single diagram:
Ricklee retweeted
I've just discovered this amazing document showing super clearly the relation between the opcode and the instruction 🤯
pnx.tf/files/x86_opcode_stru…
































