Skip to content

Add Gaussian Splatting for visualizations - #89

Open
amacati wants to merge 17 commits into
mainfrom
feat.splats
Open

Add Gaussian Splatting for visualizations#89
amacati wants to merge 17 commits into
mainfrom
feat.splats

Conversation

@amacati

@amacati amacati commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

This PR adds Gaussian Splatting for rendering in the simulation.

We use splax to be able to include the splatting in JAX. This also allows us efficient differentiation, batched rendering, support for sliced transforms etc.

OOP vs Functional

As usual, we support two APIs: One that is object oriented, where we do not bother optimizing the rendering function to its max. The other is the functional API, which fuses all static params into a newly generated function. Because MuJoCo structs are large and JAX does runtime validation checks, this happens to matter a lot for smaller inputs. Once we deal with sufficiently high-res images or a large number of parallel envs, the disadvantage becomes insignificant.

Color vs Depth

We support both RGB and RGBD. The latter must convert semi-transparent pixels into distances. We thus let users choose an alpha threshold and return the expected depth for everything less transparent than the threshold.

Viewer

This update brings the Viser viewer with it. The viewer has some real perks. It is web-based and thus allows crazyflow to be streamed from remote machines via port forwarding (which mostly happens automatic). We could consider moving to this viewer in the long term, even for the regular visualization.

Data

We must store the splats somewhere. Since these are large binary blobs, we pull them from huggingface for now and use splax's caching.

Examples

We have quite a few examples with this PR. One shows the new viewer, one the rendering of parallel splats in parallel worlds for parallel drones, one the RGBD images, and one the differentiability for pose optimization.

@amacati
amacati requested a review from ratheron August 3, 2026 22:46
@amacati amacati added enhancement New feature or request sensors labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request sensors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants