Testing LLMs (and prompts) like we test software: towardsdatascience.com/testi…
TL;DR: (1) You should, (2) How to test: specific properties, evaluate these with LLMs (perception is easier than generation), (3) What to test: get the LLM to help you figure it out.
Marco Tulio Ribeiro retweeted
Microsoft open-sources a new AI library that connects to open-source GPTs, not just OpenAI. github.com/microsoft/guidanc…
Marco Tulio Ribeiro retweeted
Also highly relevant: guidance from microsoft
"Guidance programs allow you to interleave generation, prompting, and logical control"
Also internally handles subtle but important tokenization-related issues, e.g. "token healing".
github.com/microsoft/guidanc…
Marco Tulio Ribeiro retweeted
been reading the readme for github.com/microsoft/guidanc…, kind of galaxy brain
tl;dr they made a whole prompt engineering language
Blog post: playing with Vicuna-13B, ChatGPT (3.5), MPT-7B-Chat on harder stuff medium.com/@marcotcr/explori…
TL;DR: We think ChatGPT is still way ahead, but sometimes the extra control from open source models is worth it.
My intern is close to writing a paper, so I wrote her this blog post on writing (part 1 of 2):
medium.com/@marcotcr/writing…
I never tweet, but here is a blog post I wrote for an intern, may be useful for others too...
Part 1: medium.com/@marcotcr/coming-…
Part 2: medium.com/@marcotcr/organiz…
"Why Should I Trust You?" Explaining the Predictions of Any Classifier.
Promo video: youtube.com/watch?v=hUnRCxny… #kdd2016 @guestrin @sameer_
Marco Tulio Ribeiro retweeted
Great work from @marcotcr , @sameer_ on explaining any machine learning model (20 newsgroup, deep net). homes.cs.washington.edu/~mar… @guestrin