Some exciting new features just dropped for @SQLMesh v0.100! Snowpark's DataFrame api is now available to use in Python models! Now you can leverage the power of Python and Snowpark to run advanced transformations that run directly on Snowflake! Python models are super easy to write and debug with SQLMesh because they run wherever you run SQLMesh. The driver code runs on your laptop while the distributed code (dataframe api) runs in the warehouse. It's the best of both worlds! Another really cool feature we've added is the new Incremental by Partition model kind / materialization. These are special models that will incrementally replace partitions while keeping existing partitions intact. For example, if you have a dataset that is partitioned by country, and every day, you want to only refresh a set of countries. sqlmesh.readthedocs.io/en/st… sqlmesh.readthedocs.io/en/st…

I'm really glad to see unit testing is becoming normal in the analytics engineering workflow with the release of dbt's version of unit tests. It's something we at @TobikoData cared a lot about during our days at Netflix and Airbnb and we knew it had to be part of @SQLMesh on day 1. But we certainly understand that testing data is a relatively new concept and seen as a challenging chore. But hopefully with dbt's new addition, together we can make testing data as widespread as testing software. We've done a lot of thinking about how to make unit testing data easier and less painful. Here are some unique features that SQLMesh unit tests provides. 1. We can run your tests on DuckDB even if your project normally runs on a warehouse like Bigquery or Snowflake. This is made possible by SQLGlot. It's way faster, cheaper, and easier to setup then running tests against your actual warehouse. If your SQL cannot be automatically translated, you can of course run your tests on any warehouse you like sqlmesh.readthedocs.io/en/st… 2. No one likes to define the input data for tests, so we added a tool to create tests for you. Just write a query and the results will form your initial test which you can then tweak! sqlmesh.readthedocs.io/en/st…

Last night I was working with a partner with a large dbt deployment. They were struggling with an insanely tangled spider web DAG. They wanted to understand how two tables were connected, but dbt wasn't able to load a vis. We used SQLMesh's powerful selector notation to figure it out. sqlmesh dag --select-model "db.parent_model+ & +db.child_model" dag.html