Marc Garcia retweeted
The early design decisions for the Categorical type were under strain because of our streaming engine. Every data chunk carried its own mapping between the categories and their underlying physical values, forcing constant re-encoding. The global StringCache we built to solve it caused lock contention and wasn't designed for a distributed architecture.
The new Categories object, released in 1.31, solves this, and gives you:
• Control over the physical type (UInt8/16/32)
• Named categories with namespaces
• Parallel updates without locks
• Automatic garbage collection
When you know the categories up front you can use Enums. They're faster because of their immutability and allow you to define the sorting order of values.
The StringCache is now a no-op, but the code will keep working how it used to (with global Categories). You can also migrate by replacing it with explicit Categories where needed.
The result is a Categoricals data type that works well on the streaming engine without performance degradation, and is compatible with a distributed architecture.
Read the full deep dive: pola.rs/posts/categoricals-r…
Marc Garcia retweeted
DataFusion 52 is released datafusion.apache.org/blog/2…
Marc Garcia retweeted
We're happy to announce the release of #pandas 2.3.0. You can install it with `pip install pandas` or `conda install -c conda-forge pandas`. Thanks to all contributors and sponsors who made this release possible! The release notes can be found at: pandas.pydata.org/docs/whats…
Marc Garcia retweeted

unusual_whales@unusual_whales22 Nov 2024
North America
North AmericaConnected via North America App StoreAccount-level information, not a live location or per-post device.
Tech occupations have been among the hardest hit by steep declines in hiring since 2018, per BI;
Marc Garcia retweeted

Interesting things@awkwardgoogle16 Nov 2024
South America!
South AmericaConnected via South America App StoreLocation may be affected by a proxy or VPN.Account-level information, not a live location or per-post device.
Possibly the greatest single male athletic performance of all time
Marc Garcia retweeted
Today we are launching the first open Crash Course training sessions with a limited time discount. These instructor-led sessions are open to everyone looking to get up and running with Polars.
Find a date and sign up via our Academy: pola.rs/academy/
Marc Garcia retweeted
I've written a blog about Tonbo's research on async Rust and io_uring:
tonbo.io/blog/async-rust-is-…
We need to be careful to avoid the cancellation problem when using async Rust and io_uring together.
Marc Garcia retweeted
.@jimmy_dore: "Barack Obama made sure the bankers got their bonuses while he kicked 5 million families out of their homes. He got more money from Wall Street than John McCain did. There was a Citigroup email with a list of people they wanted in his cabinet. Every person on that list ended up in his cabinet. That got released by Wikileaks, and that's why they've been trying to kill Julian Assange ever since.
Obama wasn't a departure from George W. Bush. Just his skin color. He took us from two wars to seven. He gave us Mitt Romney's right-wing health care plan that came out of the Heritage Foundation, which is the same people that gave you Project 2025. He dropped more bombs than George Bush, and nobody noticed, and that's why the donor class wanted to have Kamala Harris.
They thought we'd have a female Barack Obama, dropping bombs, stealing resources, and doing neoliberal and neoconservative policies, but then she couldn't get a vote in the primary because she has no political talent. So then they forced her on Joe Biden as the Vice President, and then, of course, they did a coup just like Jim Gaffigan said at the Al Smith dinner.
After Occupy Wall Street and the Tea Party, major newspapers' mentions of racism and white supremacy went through the roof. That's because, as Chomsky taught us, newspapers aren't there to inform you. They are there to manufacture consent, and one of the great ways to manufacture consent for the establishment is to divide and conquer. That's what identity politics is."
Marc Garcia retweeted
One day we are going to wake up, after years of the consistent and pernicious pollution of critical information ecosystems, and deeply regret this era of thoughtless LLM deployment.
A devastating loss of accuracy, clarity and conciseness -- for what?
The internet has been progressively diluted with AI-generated slop. Are medical records headed for the same fate?
🧵 I just published a perspective in @NEJM with @AdamRodmanMD and Arjun Manrai on why rushing AI into medical documentation could be a mistake.
Marc Garcia retweeted
Scikit-Learn just crossed 100000 citations! I am no longer part of the core team, but I am proud to have been part of the project since the beginning. A big thank you to all maintainers, contributors, and users for their continuous vision, support and help! 👏 @scikit_learn
























