Open-Source Interpretability for Generative Language Models 🔎 🐛
- Tweets57
- Following607
- Followers336
- Likes94
ALT Granular visualization of an attention weight matrix for Gemma 2B on a summarization task. The slider on the bottom allows to visualize scores for individual heads without aggregation.
ALT Generated text with probability highlights (in green). For every generated token, source and target attribution scores can be visualized by opening the collapsed section (scores for token "colore" shown in figure)
ALT Output visualization of the attribute_context command for a chain-of-thought reasoning task with output-side context. The token "pairs" in the generated sentence "20 pairs of legs." is found to be context-sensitive and its prediction over the non-contextual alternative "horses" is motivated mainly by the presence of "pairs" in the output context.
ALT Granular visualization of an attention weight matrix for Gemma 2B on a summarization task. The slider on the bottom allows to visualize scores for individual heads without aggregation.
ALT Generated text with probability highlights (in green). For every generated token, source and target attribution scores can be visualized by opening the collapsed section (scores for token "colore" shown in figure)
ALT Output visualization of the attribute_context command for a chain-of-thought reasoning task with output-side context. The token "pairs" in the generated sentence "20 pairs of legs." is found to be context-sensitive and its prediction over the non-contextual alternative "horses" is motivated mainly by the presence of "pairs" in the output context.
ALT A Google Colab notebook that loads the pretranied Pythia-1B model from HuggingFace and then visualizes it with Treescope.
ALT A Google Colab notebook that converts the pretranied Pythia-1B model from HuggingFace to a Penzai model and then visualizes it with Treescope.
![[points to Pyvene]
Inseq: You there, what is your profession?
Pyvene: I do trainable inference-time interventions... sir.
Inseq: [points to another library] And you, TransformerLens, what is your profession?
TransformerLens: I provide a unified hookable architecture to support mechinterp analyses on Transformer models, sir.
Inseq: I see...
[turns to a third soldier]
Inseq: You?
SAELens: I enable the training of sparse autoencoders to mitigate superposition in latent features.
Inseq: [turns back shouting] INSEQ USERS! What is OUR profession?
Inseq Users: HA-OOH! HA-OOH! HA-OOH!
Inseq Users: We do feature attribution of generative LMs! 😇](https://pbs.twimg.com/media/GMlicq3X0AM_NNG.jpg?name=orig)
ALT [points to Pyvene] Inseq: You there, what is your profession? Pyvene: I do trainable inference-time interventions... sir. Inseq: [points to another library] And you, TransformerLens, what is your profession? TransformerLens: I provide a unified hookable architecture to support mechinterp analyses on Transformer models, sir. Inseq: I see... [turns to a third soldier] Inseq: You? SAELens: I enable the training of sparse autoencoders to mitigate superposition in latent features. Inseq: [turns back shouting] INSEQ USERS! What is OUR profession? Inseq Users: HA-OOH! HA-OOH! HA-OOH! Inseq Users: We do feature attribution of generative LMs! 😇
ALT Figure 1 of the paper showing the main workflow to identify and mitigate gender bias in machine translation.
