Skip to content
BranchesTags

Repository files navigation

Face3D

Next-gen 3D face reconstruction from phone video.
Samsung Galaxy S25 Ultra → 2D Gaussian Splatting → Animated 3D Face

Capture a 20-second orbit video. Get a photorealistic, animatable 3D face model.

What It Does

Face3D takes a video of someone's face (shot on a Samsung Galaxy S25 Ultra) and reconstructs a full 3D Gaussian Splat model with:

  • Photorealistic rendering from any angle via 2D Gaussian Splatting
  • Clean triangle mesh with 4K UV texture via SuGaR extraction
  • Facial animation driven by FLAME expression parameters (smile, surprise, etc.)
  • Compressed export — Draco compression, .splat web format, glTF 2.0 support
  • Desktop app — Tauri 2.0 + React app with immersive pipeline visualization

Optionally feed in Expert RAW photos and Samsung Sensor Logger data alongside video.

Pipeline

                        Samsung Galaxy S25 Ultra
                     Video (.mp4) + Photos (.jpg/.dng)
                                  |
                    +-------------+-------------+
                    |                           |
            Frame Extraction              Photo Processing
            Motion-based sampling         Expert RAW / DNG
            HW-accelerated HEVC          EXIF lens matching
            S-Log3 color correction       High-res anchors
                    |                           |
                    +-------------+-------------+
                                  |
                        Quality Filtering
                        Blur / exposure / face detection
                        Smart frame selection (farthest-point)
                                  |
                    +-------------+-------------+
                    |                           |
              Depth Anything 3            COLMAP (fallback)
              Depth + Poses + Confidence  Structure-from-Motion
              Single inference pass       Face-optimized settings
              Replaces COLMAP             SuperPoint/SuperGlue
                    |                           |
                    +-------------+-------------+
                                  |
                      Face Reconstruction
                      MediaPipe 478 landmarks
                      FLAME parametric fitting
                      Face segmentation + GrabCut
                                  |
                    +-------------+-------------+
                    |                           |
          FLAME-Bound Gaussians          Free Gaussians
          30K on mesh triangles          10K for hair/ears
          Animatable via FLAME           Move freely
          Barycentric coordinates        No mesh binding
                    |                           |
                    +-------------+-------------+
                                  |
                      2D Gaussian Splatting
                      Normal consistency loss
                      Distortion regularization
                      LPIPS perceptual loss
                      Depth supervision
                      Appearance embedding
                      Progressive training
                                  |
                    +-------------+-------------+
                    |             |             |
              Gaussian PLY   SuGaR Mesh   Animation
              + compressed   + 4K texture  Smile, surprise
              15x smaller    UV-baked      Head turn
                             .obj + .mtl   FLAME-driven
                                  |
                          Quality Report
                          HTML with comparisons
                          Turntable GIF
                          Stage timing

Quick Start

1. Environment Setup

# Create conda environment
conda env create -f environment.yaml
conda activate face3d

# Install CUDA-dependent packages
pip install gsplat pycolmap
pip install git+https://github.com/bytedance-seed/depth-anything-3.git --no-deps
pip install addict evo moviepy==1.0.3 einops huggingface_hub safetensors

# Download FLAME model (required, free registration)
# https://flame.is.tue.mpg.de → Download → place in Models/Flame/

2. Capture

Record a 20-second orbit video of the subject's face with your S25 Ultra:

  • Pro Video mode, 4K or 8K, 30fps
  • Slowly orbit from ear to ear (frontal → left profile → back → right profile → frontal)
  • Keep ~1.5m distance, maintain consistent height
  • Even diffuse lighting (overcast or ring light)
  • Subject keeps a neutral expression and stays still

Optionally take 5-10 Expert RAW photos at key angles (front, 3/4, profile).

3. Run

# Auto-detect all inputs from a folder (recommended)
python scripts/run_pipeline.py --content-dir content/New --session my_face

# Video only
python scripts/run_pipeline.py --video path/to/face_video.mp4

# Resume from a specific stage
python scripts/run_pipeline.py --content-dir content/New --session my_face --start-stage 13

# Desktop app (Tauri)
cd face3d-app && npm run tauri dev

4. Output

data/output/{session}/
├── gaussians.ply                # Full Gaussian splat model (15 MB)
├── gaussians_compressed.ply     # Quantized (7.5 MB, 2x smaller)
├── gaussians_draco.gs3          # Draco compressed (6.6 MB, 2.3x smaller)
├── mesh.obj / mesh.ply          # SuGaR Poisson mesh
├── mesh_textured.obj            # UV-textured mesh + material
├── texture.png                  # Face texture map
├── renders/                     # 30 turntable views + MP4 video
├── previews/                    # Depth maps, landmarks, masks, turntable GIF
├── report.html                  # Self-contained quality report
├── pipeline_timing.json         # Per-stage performance profiling
└── metrics.jsonl                # Per-stage metrics

Architecture

Modules

Module Purpose
src/capture/ Video frame extraction, Expert RAW processing, quality filtering, smart frame selection
src/sensors/ Samsung IMU parsing, Madgwick orientation filter, COLMAP rotation priors
src/depth/ Depth Anything 3 unified (depth + poses), DA2 fallback, scale alignment
src/reconstruction/ COLMAP SfM, MediaPipe landmarks, FLAME fitting, face segmentation
src/splatting/ 2DGS training, FLAME-bound initialization, SuGaR mesh, texture baking, animation
src/utils/ COLMAP I/O, camera profiles, quality reports, visualization

Key Technologies

Component What Why
Depth Anything 3 Depth + camera poses in one pass Replaces COLMAP, 35% better pose accuracy
gsplat 2D Gaussian Splatting Flat-disc Gaussians for better surface reconstruction
FLAME Parametric face model 5,023 vertices, expression-driven animation
MediaPipe 478-point face landmarks Bridge between 2D detection and 3D FLAME fitting
COLMAP Structure-from-Motion Fallback when DA3 is unavailable

Desktop App (Tauri 2.0)

Feature Description
Pipeline View Immersive 15-stage timeline with live progress, animations, and stage descriptions
3D Viewer Real-time PLY point cloud viewer with React Three Fiber
Gallery Multi-category image browser with lightbox and before/after comparison
Sensor Panel IMU/quaternion visualization with animated 3D orientation cube
Metrics Training curves, stage timing bars, quality dashboard
libSQL Database Turso-powered session tracking, training metrics, vector search for ML prior

Training Innovations

  • 2D Gaussian Splatting — flat disc primitives for better face surfaces
  • 6 loss functions — L1, D-SSIM, normal consistency, distortion, LPIPS, depth supervision
  • FLAME-bound initialization — Gaussians on mesh triangles with barycentric coordinates
  • Joint FLAME optimization — GaussianSwap-inspired temporal consistency (shared identity, per-frame expression)
  • Progressive training — 3-stage resolution (1/4 → 1/2 → full) with staged loss introduction
  • Photo-weighted sampling — Expert RAW views sampled 3-5x more during training
  • Draco compression — Industry-standard 3D compression (2.3x file size reduction)

Performance & ML

  • Numba JIT kernels — 6.5x faster color correction, 131x faster Madgwick filter
  • Pipeline profiler — Per-stage timing, GPU memory tracking, throughput metrics
  • Experience replay buffer — Each scan improves future reconstructions via ML prior
  • Face prior network — 50K-param MLP predicts initialization from FLAME params
  • Taichi physics — Physics-aware mesh refinement (Laplacian smoothing, anatomical constraints)
  • 570+ tests — Syrupy snapshots, Pandera contracts, Mutmut mutation testing, property-based tests

Hardware Requirements

Component Minimum Recommended
GPU NVIDIA GTX 1080 (8GB) RTX 3080+ (16GB)
VRAM 8 GB 16 GB
RAM 16 GB 32 GB
Storage 10 GB free 50 GB free
OS Windows 10/11 Windows 11
CUDA 11.8+ 12.x
Phone Any with video Samsung Galaxy S25 Ultra

Configuration

All parameters are in config/pipeline.yaml:

# Key settings to tune
capture:
  max_frames: 300
  filtering:
    blur_threshold: 30.0       # Lower = accept softer frames
    require_face: true

splatting:
  training:
    iterations: 7000           # 3000 = quick test, 30000 = max quality
    max_num_gaussians: 500000  # VRAM cap for 16GB GPU
  export:
    texture_resolution: 4096   # UV texture map size
    turntable_views: 60

Error Handling

The pipeline is designed to be resilient:

  • OOM Recovery — automatically reduces Gaussian count and retries (up to 3x)
  • NaN Detection — restores last good checkpoint, halves learning rate
  • Gradient Clipping — prevents training explosions
  • Stage Validation — checks outputs between stages, fails fast with helpful messages
  • Graceful Fallbacks — DA3 → COLMAP, 2DGS → 3DGS, FLAME → sparse points, SuGaR → TSDF

Project Structure

Face3D/
├── config/pipeline.yaml           # Master configuration (all hyperparameters)
├── scripts/run_pipeline.py        # Pipeline orchestrator (15 stages)
├── src/
│   ├── capture/                   # Video frame extraction, Expert RAW, quality filtering
│   ├── sensors/                   # Samsung Sensor Logger parsing, orientation fusion
│   ├── depth/                     # DA3 unified (depth + poses), AnyDepth fallback
│   ├── reconstruction/            # COLMAP, MediaPipe, FLAME, face segmentation
│   ├── splatting/                 # 2DGS training, export, compression, animation
│   ├── validation/                # Pandera data contracts for stage-to-stage validation
│   ├── ml/                        # Experience replay, face prior network, Taichi physics
│   └── utils/                     # COLMAP I/O, Numba kernels, Triton kernels, timing
├── face3d-app/                    # Tauri 2.0 desktop app
│   ├── src/                       # React + TypeScript frontend (25+ components)
│   ├── src-tauri/                 # Rust backend (7 command modules + libSQL)
│   └── package.json
├── tests/                         # 570+ tests (pytest, Hypothesis, Syrupy, Pandera, Mutmut)
├── Models/Flame/                  # FLAME model files (not in repo)
├── data/                          # Pipeline I/O (not in repo)
└── LICENSE                        # GPL-3.0

License

This project is licensed under the GNU General Public License v3.0 — see LICENSE for details.

Third-party dependencies have their own licenses:

  • FLAME — Requires separate registration at flame.is.tue.mpg.de
  • Depth Anything 3 — Apache 2.0
  • gsplat — Apache 2.0
  • MediaPipe — Apache 2.0
  • COLMAP — BSD 3-Clause

Built by Nickalus Brewer

About

No description, website, or topics provided.

Resources

Readme
GPL-3.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

1 fork
Report repository

Releases

Packages

Contributors

Languages