Bits Chat is now in @SlackHQ 👀 Alert fires → @datadoghq investigate → find the root cause → generate a fix with Bits Code → open a PR → resolve. From alert to resolution, without leaving the incident channel ↓ datadoghq.com/blog/bits-chat…

Some raw / unfiltered thoughts around the Jev model. Thinking out loud: - Lots and lots of problems in business are classification or regression problems. Lots and lots of companies don't know that the types of problems they have are solvable by classical ML methods. They often solve them with people and process instead of technology. - With the emergence and popularity of LLMs, more companies are thinking 'Maybe we can use AI for that!' and are solving classification and regression problems with LLMs. - This is good in some ways, because companies are potentially automating some manual work, but also bad in some ways, because it's often the wrong tool for the job, and possibly not as good as the classical ML methods for what they are trying to do.. - If anything, the emerging popularity of 'AI' (which for most folks is synonymous w/ LLMs at this point) has caused them to try to solve their problems with that type of 'AI' (even if classical methods are more appropriate if you knew about the available tools / techniques) - There are very established 'classical ML' patterns to solve the problems they have (e.g. XGBoost is often all you need). AutoML works pretty well at this point for the 80% of problems here too, as long as your data is clean. - But the classical techniques require you to label your data, train a model, and host that model somewhere. They aren't as easy to use compared to 'calling an LLM API' - and it requires you and your org to be aware of those techniques, and capable of investing in them. - Classical ML is now much easier to do because of LLMs. I've built a bunch of models in the last 2 years using LLMs to build the classical ML code, do data cleaning, etc. - that would have taken me *much longer* if I didn't have an LLM to help - but still harder than shoving the problem at a text-based model. - There are some great patterns / techniques which exist around LLMS to constrain output for classification / regression, and make things cheaper / more performant (Structured Outputs, DSPy, GEPA, etc.) - still not super-accessible to the broader audience (I can count on one hand the number of practioners I know who have experimented with DSPy / GEPA -- maybe a few more who use Structured Outputs) - I am hopeful that these types of things become more adopted, but maybe the technical hurdles are too high in the short-term. - With Jev, maybe they've nailed the 'UX' for this. Yes, it looks a lot like structured outputs, but it is more opionated and directly targets classification / regression -- in a way that probably more folks will understand vs. Structured Outputs / DSPy / GEPA - I do wonder how effective it will be. It feels very adjacent to zero-shot Timeseries stuff like TimesFM. Like yes, it is very cool you can do predictions without training, but when you look at stuff like TimesFM, it seems to be not very good compared to supervised training of TS models (again, requires expertise to do so). Maybe you can put your labeled data in context, and this will work - But then my question becomes -- if it takes a bunch of know-how to use this type of zero-shot classifier / regressor model, at what point is it better to just use LLMs to help you build classical ML models? Thank you for reading my rant. I know it is disorganized and probably not fully coherent. Either way, it's super-cool to see new ways to piece together this new tech that has emerged over the last few years. Excited to track where Jev goes, and I think it prob. will be useful for some. I plan on trying it out for sure.

A couple of days ago, I did a quick explainer on Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be interesting to go into a bit more detail when explaining how it works. So, instead of the usual text article, I recorded a little lecture on the topic (to change it up a bit from my usual articles). It ended up a bit longer than intended, but I hope it clarifies a lot of things: - Sampling the next token in an LLM and pseudorandom number generators - How watermarking relates to the regular LLM sampling process - Whether watermarking makes text "worse" - How to remove watermarks - Tournament sampling - How new text is checked for watermarks without rerunning the LLM I ended up with ~50 slides, but I hope that these explain it well, though! Happy watching!

I ran a simplify task on every file in tldraw's main apps/packages. +10K -22K LOC. Tests pass, no API changes, mostly superficial stuff (but a LOT of it)

failures of “monitoring” as a general solution to ai safety: monitoring, like all other software, runs on flaky and mortal infrastructure with some amount of downtime. does a momentary blip in monitoring open Pandora’s box? will people accept fail closed monitoring on all systems? monitoring produces false positives. people get tired of reading fake or minor issues and stop looking. monitor fatigue is a real problem, and “solve all precision recall issues” is as hard a problem as any these are the prosaic failures. the more exotic failures include models and their monitors colluding etc monitoring is by no means perfect panacea. nothing short of actually aligning model will work ultimately

1) Probably a useful engineering practice if you have AI doing anything important. 2) I *strongly* endorse most companies having a similar Batphone available to all employees.

good context engineering is just reading the entirety of the agent’s context at the moment something bad happened and reflecting on whether or not a human could’ve made the right choice given that particular set of information. the hard part is looking at the data.