PhD student in PL/ML at MIT. Working towards interpretable ML using symbolic-program-inspired structures. Formerly at UC Berkeley. they/them
Cambridge, MA
Joined April 2020
- Tweets198
- Following56
- Followers459
- Likes161
Pinned Tweet
Hi! I made this account to separate out my academic posting from my non academic posting!
"Duality" is one of those terms that just kinda means a completely different thing in every context it's used in in math. Very annoying
Some CEO in 2027 is going to be talking about how 2022-2024 era RLHF had a systematic issue with reducing stylistic variance in the output distribution and be treated by a genius despite this having been obvious for years
This quoted post is unavailable.
Legitimately unsure what the point of giving reviewers 2 weeks to bid and 4 weeks to do reviews is, everyone knows it takes not nearly that long
Kavi Gupta retweeted
Day 2 of the @neurosym summer school started with a continuation of the tutorial by @akavidemic, @atharva_sehgal and me on neurosymbloc programming. Now @wellecks is telling us about how LLMs can help with formal mathematical reasoning.
Kavi Gupta retweeted
The @neurosym summer school is off to a great start!
In the morning, @atharva_sehgal, @akavidemic, and I gave the first part of our tutorial on neurosymbolic programming. Slides at drive.google.com/file/d/1lgk….
Now @AI4Code is telling us about the Scallop framework for neurosymbolic relational programming: scallop-lang.org/.
My extremely unpopular hot take is that mathematical language works really well for the kind of thing that people complain about like sigma notation but increasingly poorly for more complicated stuff like in this image
Like you will not convince me that a single person who has a similar amount of experience using sigma notation and for loops actually thinks the for loop is easier. That's just blatant familiarity bias. But formally specified theorems using Lean or Coq >>>>
What is the complexity of this function? (only allowed to be called on non-negative ints)
def f(x):
if x == 0: return 1
return x * f(x - 1)
This algorithm computes factorial. We can conceptualize it as steps 1 to x
At step k, it computes k * factorial(k-1). Factorial(k-1) has Theta(k log x) digits, and k has Theta(log(k)) digits, so the computation time is Theta(k log² k)
xitter.cf/WeinerZen/status/17961…
A significant failure of scientific communication in machine learning is the fact that a lot of people seem to think that next token prediction is inherently incompatible with true understanding of a problem
P(x) = P(x[0]) P (x[1:] | x[0]) means this definitively isn't the case
There are a lot of potential problems with next token prediction, but they all relate to relatively fiddly technical aspects of training functions. As an inference algorithm, next token prediction is more commonly known as "writing a sentence one word at a time"
My hot take here is that this isn't a failure of NTP it's just a failure of the model at being a reasonable world model. Being able to answer variant riddles correctly would increase not decrease its score on NTP xitter.cf/kjw_chiu/status/179602…
Wait is Google doing character level tokenization??? Why is it getting "New York Bay" wrong as "New York Bar"????
Is there a named concept describing a weaker version of independence? Like
For all y1, y2; P[X | Y = y1 ] ≥ k * P[X | Y = y2]
Obvs for k=1 you recover independence but for k =0.5 you'd have like a softer version
I personally would suggest that CS is not about P=NP, as it is mostly about true statements
