fix(replay): Release MediaMuxer when no frames are encoded (#5583)

* fix(replay): Release MediaMuxer when no frames are encoded

The MediaMuxer is created when the video encoder is constructed, but its
release() was reachable only on the happy path. Two cases leaked it:

- createVideoOf returned early when frameCount was 0 without releasing the
  encoder.
- SimpleMp4FrameMuxer.release() called muxer.stop() before muxer.release().
  stop() throws if the muxer was never started (no frame ever muxed), so
  release() was skipped.

This surfaced as a CloseGuard "resource was acquired but never released"
warning. Guard stop() behind the started flag so release() is always
reached, and release the encoder on the no-frames return path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* changelog

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: runningcode

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## Unreleased
+
+### Fixes
+
+- Release `MediaMuxer` when a replay segment has no encodable frames to avoid a resource leak ([#5583](https://github.com/getsentry/sentry-java/pull/5583))
+
 ## 8.44.1
 
 ### Fixes

sentry-android-replay/src/main/java/io/sentry/android/replay/ReplayCache.kt

@@ -199,6 +199,10 @@ public class ReplayCache(private val options: SentryOptions, private val replayI
 
     if (frameCount == 0) {
       options.logger.log(DEBUG, "Generated a video with no frames, not capturing a replay segment")
+      encoderLock.acquire().use {
+        encoder?.release()
+        encoder = null
+      }
       deleteFile(videoFile)
       return null
     }

sentry-android-replay/src/main/java/io/sentry/android/replay/video/SimpleMp4FrameMuxer.kt

@@ -67,7 +67,11 @@ internal class SimpleMp4FrameMuxer(path: String, fps: Float) : SimpleFrameMuxer
   }
 
   override fun release() {
-    muxer.stop()
+    // stop() throws if the muxer was never started (e.g. no frame was ever muxed), so we guard it
+    // to ensure release() is always reached and the underlying resources are freed
+    if (started) {
+      muxer.stop()
+    }
     muxer.release()
   }