Add Gaussian Splatting for visualizations - #89
Open
amacati wants to merge 17 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds Gaussian Splatting for rendering in the simulation.
We use splax to be able to include the splatting in JAX. This also allows us efficient differentiation, batched rendering, support for sliced transforms etc.
OOP vs Functional
As usual, we support two APIs: One that is object oriented, where we do not bother optimizing the rendering function to its max. The other is the functional API, which fuses all static params into a newly generated function. Because MuJoCo structs are large and JAX does runtime validation checks, this happens to matter a lot for smaller inputs. Once we deal with sufficiently high-res images or a large number of parallel envs, the disadvantage becomes insignificant.
Color vs Depth
We support both RGB and RGBD. The latter must convert semi-transparent pixels into distances. We thus let users choose an alpha threshold and return the expected depth for everything less transparent than the threshold.
Viewer
This update brings the Viser viewer with it. The viewer has some real perks. It is web-based and thus allows crazyflow to be streamed from remote machines via port forwarding (which mostly happens automatic). We could consider moving to this viewer in the long term, even for the regular visualization.
Data
We must store the splats somewhere. Since these are large binary blobs, we pull them from huggingface for now and use splax's caching.
Examples
We have quite a few examples with this PR. One shows the new viewer, one the rendering of parallel splats in parallel worlds for parallel drones, one the RGBD images, and one the differentiability for pose optimization.