coding . ml . math . writing . books . https://xitter.cf/t.co/SJXxlX5JeT
🇯🇵⇄🇧🇷
Joined December 2009
- Tweets7K
- Following352
- Followers9.9K
- Likes12.2K
📝 I hope with this new post, you can steal some ideas, and insights, and put them into practice in your life. This is my reflection about reading 47 books in the first 6 months of 2023 and how I am focusing on reading less + applying them in my life.
iamtk.co/i-read-47-books-in-…
✨ Hands-On Machine Learning with Scikit-Learn and TensorFlow — Full Review: goodreads.com/review/show/65…
2026 has been the year of biographies and technical books so far. In the last couple of years, I've learned to read fewer books, choose better, and focus on the signals (still wip).
Here are the ones I read so far: goodreads.com/readingchallen…
> AI and ML for Coders in PyTorch: A Coder's Guide to Generative AI and Machine Learning
> Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications
> Deep Learning for Biology: Harness AI to Solve Real-World Biology Problems
> Decoding Greatness: How the Best in the World Reverse Engineer Success
> Nós: O Atlântico em solitário
> Practical Statistics for Data Scientists: 50 Essential Concepts
> The 5 Types of Wealth: A Transformative Guide to Design Your Dream Life
> Machine Learning System Design: With end-to-end examples
> Build a Large Language Model
> The Art of Doing Science and Engineering: Learning to Learn
> AI Engineering: Building Applications with Foundation Models
---
I'm currently reading The Infinity Machine (DeepMind) and The Idea Factory (Bell Labs). I have just finished AI Engineering and am looking for a new technical book.
📝 I hope with this new post, you can steal some ideas, and insights, and put them into practice in your life. This is my reflection about reading 47 books in the first 6 months of 2023 and how I am focusing on reading less + applying them in my life.
iamtk.co/i-read-47-books-in-…
I am taking time off from work for 3 weeks starting today. Because I don't have a planned trip, It's time to focus and keep pushing myself. It's a huge privilege to not think about anything else but only my ML research, writing, running, and reading books.
Watching the DeepMind doc at breakfast gave me a huge boost of motivation. Hopefully, I'll have great outputs from this time.
One of the best articles I've read these days. A very practical guide for ML research. I should re-read it every now and then to improve my workflow.
→ joschu.net/blog/opinionated-…
📝 Finetuning Foundation Models
A week ago, I wrote about building the GPT foundation model from scratch [0] [1], and while writing it, I set one section apart for a separate article. This part is about finetuning the foundation model.
The example is similar to what was shown in the Build a Large Language Model book [2]: Spam or Ham.
The first step was to create a dataset based on the UCI SMS Spam Collection. We download the data, preprocess it, and wrap it into a PyTorch Dataset to handle items, and then into a PyTorch Dataloader to handle batches and workers. The dataset is also split into train, validation, and test sets.
One important note about foundation model datasets is the nature of the input text: they can come in different sizes, so one strategy is to encode the texts (into token IDs) and pad to the longest encoded length, so every input text has the same size.
For the finetuning part, I used the previous model implementation [1] as the backbone, but because it isn't pretrained on a large amount of data, we use the weights from another more powerful model (GPT2). This part requires:
> Load the parameters from GTP2 into our model
> Repurpose the model for a classification problem: replace the output head (binary classification)
> Freeze the parameters to preserve what was learned in pretraining (enable gradients only for a few parameters)
After that, we move to the standard approach to Neural Network [3] training [4]:
> Compute loss
> Compute gradients
> Optimize parameters
> Build evaluation metrics across epochs and steps
The article finishes with some simple tests on spam and ham texts.
----
Here's the link to the article.
→ Link: iamtk.co/finetuning-foundati…
----
[0] Self-Attention, Foundation Models, and the GPT Architecture from Scratch: iamtk.co/self-attention-foun…
[1] LLM from Scratch Repo: github.com/imteekay/llm
[2] Build a Large Language Model: manning.com/books/build-a-la…
[3] Building A Neural Network from Scratch with Mathematics and Python: iamtk.co/building-a-neural-n…
[4] Mastering PyTorch: From Linear Regression to Computer Vision: iamtk.co/mastering-pytorch-f…
𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗚𝗣𝗧 𝗠𝗼𝗱𝗲𝗹
For the past few weeks, I've been reading about Foundation Models [0] and decided to work on the implementation of the GPT architecture [1] to understand its building blocks and how it works under the hood.
Here are the concepts I worked on in this implementation:
Tokenization → Embeddings → Self-Attention → Multi-Head Attention → Transformer Block → GPT Model → Pretraining.
— The tokenization part was focused on building tokens from the input text and transforming them into token IDs; Then using a BPE tokenizer algorithm [2]
— Embeddings: representing tokens with a simple scalar value (ID) is too simplistic. Embeddings come to build richer representations. I built small embeddings for learning purposes and then increased the representation to scale that
— Multi-Head Self-Attention: this was one of the most interesting parts, creating attention scores and building relationships between tokens to produce context vectors
— Transformer blocks have the attention heads, dropout, layer norm, and the feed-forward network
— Pretraining is a standard training process used for deep learning models. But in this case, we update the weights end-to-end, from the embeddings to the attention layer to the feedforward network
The implementation was highly inspired by the Language Modeling from Scratch course [3] and the Build a Large Language Model book [4]. It's still very rudimentary, but very useful if you plan to learn these concepts in depth.
🔗 Article Link: Self-Attention, Foundation Models, and the GPT Architecture from Scratch: iamtk.co/self-attention-foun…
---
In the future, I plan to write about finetuning (using foundation models and finetuning for other tasks) and optimizations (attention blocks optimization, GPU and kernel optimization).
[0] Foundation Models at Nubank: xitter.cf/wordsofteekay/status/2…
[1] LLM implementation repo: github.com/imteekay/llm
[2] Tokenizers lecture: youtube.com/watch?v=SQ3fZ1sA…
[3] Language Modeling from Scratch: youtube.com/playlist?list=PL…
[4] Build a Large Language Model: manning.com/books/build-a-la…
This resonates a lot with my experience. My record was 60 books a year (not 80 in 6 months tho). Because I'm curious about a lot of things, many topics get my attention, so the "Parallelize" (books) tip is a really effective way to read more book. I read 3-4 at the same time, a bit every day, consistently. It turns out it is much easier to do, and in the long-term, I accomplish more.
Reading a lot also made me rethink about which books I choose to read (reading less → reading better books: iamtk.co/i-read-47-books-in-…). And because I usually read technical and non-fiction books, it's great to re-read them, take notes, and think in way to apply the ideas in my life (iamtk.co/essays/applying-boo…).
"How To Read More" by Borretti: borretti.me/article/how-to-r…
As an applied ML engineer who is learning more about research and theory, I found two interesting resources I read this week that are worth sharing.
The first one is the "On Research Taste"¹ article by Albert Ying. I liked how he defines what 'taste' really is: "the ability to find the node that would affect the largest number of other nodes [...] over a network", where the graph is a collection of "hypotheses and analyses you could pursue". I think the missing part of this short article is "how to develop 'taste'".
The second one is the "An Unofficial Guide to Prepare for a Research Position Application"² by Sakana AI. That was the most insightful blogpost I've read this year. It lays down all the core principles to be a great researcher, how to approach ideas, the importance of clear communication, and having a good balance between technical ability (engineering skills) and creativity.
The post is more than how to prepare for their interview. It's their way of doing great research.
¹ kejunying.com/blog/research-…
² pub.sakana.ai/Unofficial_Gui…
𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗚𝗣𝗧 𝗠𝗼𝗱𝗲𝗹
For the past few weeks, I've been reading about Foundation Models [0] and decided to work on the implementation of the GPT architecture [1] to understand its building blocks and how it works under the hood.
Here are the concepts I worked on in this implementation:
Tokenization → Embeddings → Self-Attention → Multi-Head Attention → Transformer Block → GPT Model → Pretraining.
— The tokenization part was focused on building tokens from the input text and transforming them into token IDs; Then using a BPE tokenizer algorithm [2]
— Embeddings: representing tokens with a simple scalar value (ID) is too simplistic. Embeddings come to build richer representations. I built small embeddings for learning purposes and then increased the representation to scale that
— Multi-Head Self-Attention: this was one of the most interesting parts, creating attention scores and building relationships between tokens to produce context vectors
— Transformer blocks have the attention heads, dropout, layer norm, and the feed-forward network
— Pretraining is a standard training process used for deep learning models. But in this case, we update the weights end-to-end, from the embeddings to the attention layer to the feedforward network
The implementation was highly inspired by the Language Modeling from Scratch course [3] and the Build a Large Language Model book [4]. It's still very rudimentary, but very useful if you plan to learn these concepts in depth.
🔗 Article Link: Self-Attention, Foundation Models, and the GPT Architecture from Scratch: iamtk.co/self-attention-foun…
---
In the future, I plan to write about finetuning (using foundation models and finetuning for other tasks) and optimizations (attention blocks optimization, GPU and kernel optimization).
[0] Foundation Models at Nubank: xitter.cf/wordsofteekay/status/2…
[1] LLM implementation repo: github.com/imteekay/llm
[2] Tokenizers lecture: youtube.com/watch?v=SQ3fZ1sA…
[3] Language Modeling from Scratch: youtube.com/playlist?list=PL…
[4] Build a Large Language Model: manning.com/books/build-a-la…
[Paper Reading: Your Spending Needs Attention]
I've just finished reading the "Your Spending Needs Attention" paper by Nubank, and not only are the results impressive, but the ML and engineering approach is also very interesting. It shows the power of self-supervised representation learning to automatically understand user behavior from raw (transaction) data, which made me think about how many insightful representations we are missing by not using it, and why (engineering and money trade-offs come to mind).
Here's the research breakdown: causal self-attention + tabular feature embedding + fine-tuning for RecSys.
Transformer-based model:
> Text is All You Need: Individual transactions are tokenized, concatenated into a transaction string, and fed through a Transformer [0] to produce a transaction sequence embedding.
> No Positional Embeddings (NoPE) [1]: drop the temporal information
> FlashAttention [2] + NoPE = Efficient Long Contexts (transaction = ~14 tokens — the sequence gets large very fast): the model can train on much larger context lengths
Tabular Features:
> Feature embeddings for numerical and categorical variables
> LightGBM: gradient-boosted tabular modeling
> Deep Cross Network V2 (DCNv2) [3]: learn feature interactions
Fine-Tuning — classification task for RecSys:
> Low-Rank Adaptation (LoRA) [4]: injecting trainable low-rank matrices into attention layers to handle the "overfitting and catastrophic forgetting" issues.
> Late Fusion: freeze the transformer embeddings and use them as static features passed into LightGBM or DCNv2 independently.
> Joint Fusion (nuFormer): keep the transformer embeddings trainable end-to-end alongside the tabular features.
It's very insightful how joint fusion trains the entire system end-to-end using a DNN, so gradients can flow through the embeddings compared to GBT.
Other insightful ideas from the paper:
> Context window problem: adding more data sources (e.g. financial products) can lead to worse results because each data source will "compete" for the available tokens for a fixed context window.
> Scaling laws: larger model size, context lengths, and data volume lead to improved performance.
There are still many interesting avenues they will explore, especially scaling laws and scaling the application to other products. It was also insightful how they are not just following the state of the art, but doing research to find new ideas [5].
---
Paper: arxiv.org/abs/2507.23267
---
[0] arxiv.org/abs/1706.03762
[1] arxiv.org/abs/2305.19466
[2] arxiv.org/abs/2205.14135
[3] arxiv.org/pdf/2008.13535
[4] arxiv.org/abs/2106.09685
[5] open.spotify.com/episode/11v…
✨ I worked on this article the whole day and made a lot of progress. I'm almost there.
A lot of work, with many experiments, but it's getting traction. "Make Something Wonderful" inspired me to keep building and sharing.
[ML Grind]
Finished:
> Foundation Models: finished transformer-based model implementation from scratch + finetuning
> Finished reading the Attention-based model in the industry paper: interesting insights about context length, scaling laws, and joint fusion
Have been working on:
> ML monitoring + alerting system for ML models
> AI agent for business flow: interesting engineering learnings (agent/prompt refinements <> MCP <> backend + infra)
> Real estate liquidity model: interesting learnings about temporal splits, model calibration, model optimization, and dataset exploration
Plan for today:
> Continue writing the blog post about the foundation model implementation
> Continue the "Language Modeling from Scratch" course by Stanford
> Read a new ML paper
📚 Started a new book today.
I'm on the first few pages, and the way it was written already caught my attention.
"Teachers should prepare the student for the student's future, not for the teacher's past. Most teachers rarely discuss the important topic of the future of their field, and when this is pointed out, they usually reply: 'No one can know the future'. It seems to me the difficulty of knowing the future does not absolve the teacher from seriously trying to help the student to be ready for it when it comes."
Excited to be educated on styles of learning and thinking, and then get back to training, applying those principles.
Many people have already pointed out, but this course by Stanford is remarkable. It's been part of the first hour of my morning. Watching the lecture, taking notes, spawns new tabs with different papers mentioned, and coding to build the intuition behind each lecture.
Mixture of experts was a nice lecture, but the one I liked the most so far was about PyTorch and resource accounting and how to make sense of CPU/GPU, memory, runtime/compute (FLOPs), etc., from first principles.
🔗 link: youtube.com/playlist?list=PL…
My notes on the repo: github.com/imteekay/machine-…
Even though most of my notes are written in my physical notebook. Still lacking time to move all to the repo.
notes: github.com/imteekay/machine-…
