Vitre Experiment #4. Winter is coming, so I brought a bit of spring into the hallway by adding synthetic 3DGS assets to the scene. Still running smoothly. #3DGS #GaussianSplatting #ComputerVision

Finally got around to trying Spirula Studio, and I'm really impressed! It simplifies the process of preparing datasets from 360° footage enormously. Trained this 3D Gaussian Splatting scene from footage I captured back in 2022 with an Insta360 ONE RS 1-Inch 360. #3DGS #GaussianSplatting #SpirulaStudio

I added a ToF sensor to the equation... an Femto Bolt. Both the captured Gaussian Splat and the Femto Bolt depth data are registered to the physical space and rendered with the same off-axis projection… so now people passing through the hallway are captured and inserted into the 3DGS scene. Still rough around the edges… but I like it. This time I needed some help with the demo. Someone had to be the tracked POV while I strangely walked around the hallway :) #3DGS #GaussianSplatting #ComputerVision

This time the tracking runs with a 120 fps global-shutter camera. Getting twice as many position updates, with much fewer gaps when tracking faster movements… It feels much better. The window is still rendering from a position roughly 20–24 ms in the past, but I compensate for some of that by predicting the position slightly forward. It still works best when tracking the phone, though. #3DGS #GaussianSplatting #ComputerVision

When I need a break from my screen, I make it disappear. Vitre, experiment #2. This time I’m using one of the monitors on my desk. It happens to sit right in front of a glass wall looking into the hallway and the rest of the office, which makes for a much more interesting view. Press the button and Vitre takes over the monitor, rendering a 3D Gaussian Splat I previously captured of the space behind it. The 3DGS is registered to the physical space, and the view is rendered in real time from the tracked viewer’s position... making the monitor vanish. I’ve been iterating on the tracking and motion-to-photon latency since the first experiment. Still a few things I want to explore, but the illusion is getting better. #3DGS #GaussianSplatting #ComputerVision

A few months ago, @kcimc sent me a message with a simple idea: capture my workspace as a Gaussian Splat, put a monitor back into that same space, track the viewer and make the monitor disappear. It’s been on my to-do list ever since... I finally built a POC… Working name: Vitre. It turns a screen into a window onto a 3D capture of the space it is actually in. My office space was captured as a 3D Gaussian Splat and registered back into the physical space. A camera tracks the viewer’s position, and the scene is rendered from that point of view in real time. Move around the screen and the perspective changes with you, as if you were looking through a window. Still a POC. The calibration of the whole system needs to be more precise, and there’s still some noticeable latency between the tracking and the render. #3DGS #GaussianSplatting #ComputerVision

Quick look at The Great Eye, our in-house tool for handling and merging sensor data to make spaces interactive. Here it’s running with 6 Orbbec Femto Mega cameras and 3 Hokuyo 2D LiDARs, all feeding into the same system. We’ve been developing and evolving this tool for a while now, mostly as the sensor layer behind our interactive installations. Built with @openframeworks #openFrameworks #ComputerVision #LiDAR #DepthCamera

A new Dormant Memories exploration, this time using Carveout.... Coiffure Mile End. One of the reasons I built Carveout was to explore 3D captures as both places and data. Explore this Dormant Memory yourself ↓ #GaussianSplatting #3DGS #ComputerVision #segmentation

Carveout is available here: github.com/smallfly/Carveout

Carveout is out. I started Carveout as a research project to explore how objects in existing 3D Gaussian Splatting scenes could be identified, labelled Carveout renders synthetic views from an existing 3DGS, uses SAM 3 to segment objects in those views, then lifts the 2D masks back onto the individual Gaussians. The result is a set of labelled 3D objects with positions, bounding boxes, supporting views and their own Gaussian subsets. No retraining of the scene, and no original capture images are needed. A big part of the project was also about the process itself: learning how all these pieces fit together, while developing Carveout from the beginning with Claude Opus 5.1 and Fable, and later bringing in OpenAI Astra as a challenger. I have tested it on rooms, shops, objects and outdoor captures, at different scales and orientations. It definitely won’t work perfectly on everything though, and that is part of why I’m releasing it. The code is now available for anyone who wants to experiment with it, break it, or take it somewhere else. Link in the thread. #GaussianSplatting #3DGS #ComputerVision #segmentation

New scene in the Dormant Memories series. A bouquet, some light, depth and time. #GaussianSplatting #3DGS #pointcloud #memories

I’ve been testing Carveout on a much wider range of 3DGS captures. Different scales, tilted scenes, low ceilings, interiors, objects, exteriors… each exposed assumptions that were fine on my original test scenes but didn’t generalize well enough. So over the last few days I’ve been doing my best to fix those properly rather than adding scene-specific workarounds. The main changes: - Scene scale is now handled explicitly, including captures with unknown or incorrect scale. - Tilted captures can be levelled without modifying the original 3DGS. - Camera placement and view selection are more robust across very different spaces. - The VLM can be selected per scene, with a smaller option when VRAM is limited. - The UI now does a better job of guiding each stage and showing what is running or needs attention. The bigger gain is that Carveout is becoming much less dependent on how a scene was captured or exported. Though it’s still better to start with a scene that is properly oriented and scaled 😶‍🌫️ 3DGS capture by Seeget3D, shared on SuperSplat. #GaussianSplatting #3DGS #ComputerVision

Taking a little more time before releasing Carveout. I’ve been testing it across very different types of 3DGS captures, fixing issues as they come up and making the pipeline more capable and flexible along the way. Getting close. Here it's running on a great garage capture by Simon Bethke. #GaussianSplatting #3DGS #ComputerVision

Carveout is almost ready to be released. Here is a accelerated run through the current version. The goal of Carveout is to bring object-level understanding to existing 3D Gaussian Splat scenes by propagating segmentation from rendered views back down to the individual Gaussians. It renders views of the scene, segments and identifies objects, accumulates that evidence across views, then groups the labeled Gaussians into 3D instances with bounding boxes and metadata. A lot has changed since I last shared it: the pipeline is now fully local, running SAM 3 + Qwen 3.8 27B, with a completely reworked web UI and a much more complete workflow. Planning to release it in the next couple of days. #GaussianSplatting #3DGS #ComputerVision #segmentation

Try it here: smallfly.com/dormant_memorie…

New scene in the Dormant Memories series. A 1937 Dodge, captured at a roadside car show during a quick impromptu scan this summer.  ⁣ ⁣ #GaussianSplatting #3DGS #pointcloud #memories

Art Morph, adapted from my Hands-on Hallucinations explorations, is one of the many installations in the new ArtLens gallery at the Cleveland Museum of Art. I love seeing outputs from the installation live beyond the experience itself, as a mosaic on the gallery’s large LED wall. Many thanks to Design I/O and @ClevelandArt for this amazing collaboration. #ai #art #realtime

The reimagined ArtLens exhibit is now open at the Cleveland Museum of Art, and I’m really happy to have contributed to one of its installations: Art Morph. Art Morph evolved out of my ongoing Hands-on Hallucinations explorations, developed in collaboration with Design I/O for the exhibition. Many thanks to Design I/O and the @ClevelandArt for making us part of this. I’ll be sharing a few videos from the installation over the coming days.