Typical torch.compile:
getting a PR merged in transformers,
making torch.compile work,
... be like: youtu.be/Trkxjp0qERg?t=112
and the list could go one
Living in Europe I've always been jealous of all the cool hackathons happening in the US around ML systems & optimization - thankfully AMD is hosting one in Paris in two weeks: lu.ma/fmvdjmur, come say hi (and maybe win some €€, and definitely meet cool folks)! 🤗
Is dynamic indexing not supported in torch.cuda.CUDAGraph API?
CUDA complains about `operation not permitted when stream is capturing` when using dynamic indexes in this example, but is happy with static indexes. Sounds weird to me :o
Meta models user experience: "Your request to access this repo has been rejected by the repo's authors."
DeepSeek user experience: `huggingface-cli download deepseek-ai/DeepSeek-R1` let's goooo
I'll be presenting today at 5:00 pm CET the work we have been doing at HF on TGI x ROCm, feel free to drop by: streamyard.com/watch/iMZUvJn…
On the next Hugging Cast, see how AMD works with @huggingface to run open source models in the cloud with Instinct GPUs on Azure and locally on Ryzen AI PCs
efxmarty retweeted
Beautiful doc by @efxmarty to monitor your LLM deployments (Prom + Grafana)
huggingface.co/docs/text-gen…
Does H100 actually have 85GB HBM? cudaMemGetInfo gives me 85.17 GB, and nvidia-smi gives me 85.52 GB. Yet the GPU is named "NVIDIA H100 80GB HBM3"
I'll be in Seattle next week attending Microsoft Build, if you're around and would like to chat feel free to drop me a message!
huggingface.co/chat/ with llama 3 70B + web search feature is pretty useful IMO, straight to the point, filtering out the bullshit search results, quoting precise sources you can ctrl+F. Next for me is to use an equivalent directly in VSCode for debugging :)
getting a PR merged in transformers,
making torch.compile work,
... be like: youtu.be/Trkxjp0qERg?t=112
and the list could go one
yep so according to this commit chromium.googlesource.com/ex… it is NVIDIA/ARM/Intel with 7 exponent bias vs Graphcore/AMD/Qualcomm with 8 exponent bias (arxiv.org/pdf/2209.05433.pdf vs arxiv.org/pdf/2206.02915.pdf)
AutoGPTQ 0.7.0 is released and includes @elias_frantar's Marlin kernel for int4*fp16 matrix multiplication on Ampere GPUs. Check out github.com/AutoGPTQ/AutoGPTQ… - This is usable with any int4 quantized Transformers model (symmetric quantization, no act-order) directly from the Hub!🧵
Something interesting to note is that comparing Exllama and PyTorch/Marlin is not very fair, as exllama uses cublasHgemm that accumulates on fp16 while PyTorch/Marlin use fp32 accumulation, which impacts latency & the comparison is then not apple to apple.
I wish I were as good at CUDA & profiling than @elias_frantar is, this is awesome work!
30%+ faster training, 80%+ memory savings? You heard it right, 🤗 Transformers now uses @PyTorch scaled_dot_product_attention by default and automatically dispatches to flash attention, mem-efficient attention, with no other package required than `torch`! github.com/huggingface/trans…
