🎙️ StarLink

Real-Time Simultaneous Translation for Wireless Earphones & Bluetooth Audio

Translate ambient English, Japanese, and Chinese speech into Korean — streamed directly to your AirPods, Galaxy Buds, Bluetooth earbuds, or MFi hearing devices with sub-second latency.

🌐 English | 한국어

주변의 영어·일본어·중국어 말을 한국어로 통역해 무선 이어폰·보청기로 들려주고 화면에 자막을 띄우는 iOS 앱과 통역 에이전트. iOS app and Python agent that turn nearby English, Japanese and Chinese speech into Korean audio and captions.

작품 페이지: https://ljhljh0703-cmd.github.io/StarLink/portfolio.html

실행

# 통역 에이전트 (Python)
cd backend
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
python agent.py dev        # 통역 에이전트
python server.py           # 접속 토큰 서버(선택)

# iOS 앱 (macOS · Xcode · XcodeGen)
cp StarLink/Secrets.plist.example StarLink/Secrets.plist
xcodegen generate          # StarLink.xcodeproj 생성 → Xcode에서 실기기로 실행
./run_simulator.sh         # 시뮬레이터 빌드·실행 (블루투스 경로는 실기기에서만 확인됨)

필요한 값: backend/.env에 LIVEKIT_URL · LIVEKIT_API_KEY · LIVEKIT_API_SECRET · GOOGLE_API_KEY, Secrets.plist에 LIVEKIT_URL · LIVEKIT_TOKEN · TOKEN_SERVER_URL.

시험

cd backend
python -m unittest test_agent -v    # 언어 감지 · 문장 자르기 · 자막 상태 추적

2026-10-21 컨테이너에서 돌린 결과: 17개 통과. iOS 쪽 StarLinkTests(XCTest)는 Xcode에서 돌린다.

구조

StarLink/Managers/     오디오 경로(블루투스·보청기) · LiveKit 연결
StarLink/Views/        SwiftUI 자막·설정 화면
backend/agent.py       Gemini Live 통역 에이전트, 언어 감지 · 자막 동기화
backend/server.py      접속 토큰 발급 서버
backend/test_agent.py · StarLinkTests/   에이전트 · 앱 시험
project.yml            XcodeGen 프로젝트 정의

누가 무엇을

내가 정한 것: 기획·설계·판정 (이주형)
AI로 짠 것: 코드 작성, AI 코딩 도구(Claude)와 함께. 커밋 9개 중 7개(첫 공개 커밋 포함)에 Co-Authored-By: Claude 표기
내가 확인한 것: 검증 (이주형)

📖 Overview

StarLink is an iOS application and backend agent system designed for hands-free, real-time simultaneous translation through wireless audio devices.

By pairing a mobile device (iPhone) with any Bluetooth earphone (such as AirPods or Galaxy Buds) or MFi (Made for iPhone) hearing aids, the system captures ambient foreign speech, streams it through a low-latency WebRTC connection to a backend agent powered by Gemini 3.5 Live Translate, and delivers natural-sounding Korean speech directly into the user's ears. It also displays real-time synchronized subtitles on-screen with automatic input language detection.

🎬 Demo & Portfolio

📹 Demo video — Watch on YouTube
🔗 Interactive portfolio — ljhljh0703-cmd.github.io/StarLink

🎧 Supported Audio Devices

StarLink works with any wireless audio output — AirPods, Galaxy Buds, generic Bluetooth earbuds, and MFi (Made for iPhone) hearing aids are all handled identically. No device-specific configuration is required; translated audio is routed to whatever wireless device is connected.

Note for non-hearing-aid users: While a wireless device is connected, forcing output to the iPhone's built-in speaker is intentionally disabled (a routing-protection measure for MFi devices). To play translation through the phone speaker, disconnect the wireless device first.

⚡ Core Engineering Challenges Solved (Portfolio Highlights)

This project addresses several complex real-time audio engineering and WebRTC challenges in iOS:

1. Bluetooth & MFi Audio Routing Control

Challenge: Audio can default to the iPhone's internal receiver (earpiece) instead of connected Bluetooth earphones or MFi hearing devices. Manipulating AVAudioSession directly fights LiveKit's WebRTC session management, causing async audio-thread locks and freezes.
Solution: Delegated the audio-session lifecycle to the LiveKit SDK (isAutomaticConfigurationEnabled = true) to avoid async audio-thread deadlocks, and controlled the output path through LiveKit's official API (AudioManager.shared.isSpeakerOutputPreferred) rather than raw AVAudioSession overrides. When a hearing device is connected, speaker preference is forced off and a short Bluetooth settle delay is inserted to avoid AUIOClient_StartIO (-3001) deadlocks.

2. Acoustic Feedback Loop Prevention

Challenge: Translated audio playing through the earphones can leak back into the iPhone's microphone, causing the AI agent to hear its own translated voice and re-translate it (an echo loop). Physically muting/unmuting the mic track to prevent this risks WebRTC audio-device locks and freezes.
Solution: Instead of physically toggling the mic track (which destabilizes the WebRTC audio device), feedback and back-channel interception are suppressed on the backend by disabling interruption handling in the AgentSession ("interruption": {"enabled": False}). This keeps the audio output path alive while preventing the agent from re-translating its own output.

3. Voice-Subtitle Modality Synchronization

Challenge: The Gemini Live voice-to-voice model generates raw audio packets. In its default state, it does not send the corresponding transcription stream, leaving on-screen subtitles stuck in a loading state.
Solution: Wrapped the Gemini session initialization with custom AudioTranscriptionConfig parameters on both input and output streams. This forces the model to emit simultaneous transcription tokens, which are captured, parsed, and pushed to the SwiftUI view in sync with the audio playback.

4. Zero-Latency Input Language Detection

Challenge: Translating multi-language conversations requires showing the source language (e.g. English, Japanese, Chinese) to the user. Using external translation APIs to detect language introduces unacceptable network latency.
Solution: Built a lightweight, 0ms-latency Unicode range block analyzer (detect_language) on the backend. It maps character sets (Hangul, Hiragana/Katakana, CJK Hanzi, Latin) of streaming segments, packages the language code, and transmits it via the LiveKit data channel to render glassmorphic language badges ([영어], [일본어], etc.) in real time.

🏗️ Architecture & Data Flow

┌─────────────────────────────────────────────────────────────────┐
│                        iPhone (StarLink App)                    │
│                                                                 │
│  ┌──────────┐    ┌──────────────┐    ┌───────────────────────┐  │
│  │ Mic      │───▶│ LiveKit SDK  │◀──▶│ Audio Session Manager │  │
│  │ (Ambient)│    │ (Room Client)│    │ (MFi/Bluetooth Route) │  │
│  └──────────┘    └──────┬───────┘    └──────────┬────────────┘  │
│                         │                       │               │
│                         │ WebRTC (Audio Track)  │ Bluetooth     │
│                         │                       │               │
│                         ▼                       ▼               │
│                  ┌──────────────┐        ┌──────────────┐       │
│                  │ Caption View │        │ AirPods /    │       │
│                  │ (SwiftUI UI) │        │ Earphones    │       │
│                  └──────────────┘        └──────────────┘       │
└─────────────────────┬───────────────────────────────────────────┘
                      │
                      │ WebRTC (Bi-directional Audio & Data)
                      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    LiveKit Cloud / Server                       │
│              (Low-latency Media Routing Engine)                 │
└─────────────────────┬───────────────────────────────────────────┘
                      │
                      │ WebRTC (Audio & Data Streams)
                      ▼
┌─────────────────────────────────────────────────────────────────┐
│                   Backend Agent (Python)                        │
│                                                                 │
│  ┌─────────────────┐    ┌──────────────────────────────────┐    │
│  │ livekit-agents   │───▶│ Gemini 3.5 Live Translate       │    │
│  │ (Room Subscriber)│◀───│ (Audio-to-Audio / EN-JA-ZH->KO)  │    │
│  └─────────────────┘    └──────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Audio Capture: The iPhone microphone captures ambient foreign speech.
Uplink Streaming: The LiveKit SDK publishes the local audio track to the room.
AI Translation: The Python agent receives the audio track, forwards it to Gemini 3.5 Live, and translates it into Korean.
Subtitles Delivery: The agent parses the source language and publishes transcription data to the LiveKit data channel under the topic caption.
Downlink Playback: The iOS client receives the translated audio track, and the LiveKit-managed audio session delivers it to the connected wireless device (AirPods, earphones, or hearing aid).
UI Render: SwiftUI decodes the JSON payload, rendering captions alongside their detected language badge (e.g. [영어], [일본어]).

📂 Project Structure

StarLink/
├── project.yml              # XcodeGen project specification
├── README.md                # This file
├── LICENSE                  # MIT Open-source License
├── .gitignore               # Git exclude rules
├── StarLink/                # iOS App Sources
│   ├── Info.plist           # App configuration & permissions
│   ├── Config/              # App Config (AppConfig.swift)
│   ├── Secrets.plist.example # App credentials template
│   ├── Secrets.plist        # Local private keys (git-ignored)
│   ├── Managers/            # Audio Session & LiveKit managers
│   ├── Models/              # Data models (CaptionEntry)
│   └── Views/               # SwiftUI Views (CaptionScrollView, StatusBar)
└── backend/                 # Python Backend
    ├── agent.py             # Translation Agent (Gemini Live)
    ├── server.py            # Token Server (Dynamic Token Issuer)
    ├── generate_token.py    # Local token generation utility
    ├── requirements.txt     # Python dependencies
    ├── .env.example         # Environment template
    └── .env                 # Local credentials (git-ignored)

⚙️ Setup & Installation

1. Prerequisites

Xcode 16.0+ (iOS 17 SDK)
XcodeGen (brew install xcodegen)
Python 3.11+
LiveKit Cloud account (cloud.livekit.io)
Google Gemini API Key (aistudio.google.com)
Physical iOS Device (AirPods/Bluetooth audio routing cannot be tested on Simulator)

2. Configure Credentials

iOS Client Setup (Static / Pre-configured Mode)

Duplicate the credentials template:

cp StarLink/Secrets.plist.example StarLink/Secrets.plist

Open StarLink/Secrets.plist and input your LiveKit WebSocket URL:
- LIVEKIT_URL: wss://your-project.livekit.cloud
- LIVEKIT_TOKEN: (Optional) Static JWT token for development.
- TOKEN_SERVER_URL: (Optional) The URL of your deployed Token Server (e.g., https://your-backend.com/api/token). If defined, the app will dynamically fetch short-lived tokens on startup.
Generate the Xcode project:
```
xcodegen generate
```
Open StarLink.xcodeproj, set your Development Team under Target -> Signing & Capabilities, and build/run on your physical iPhone.

🔑 Bring Your Own Key (BYOK) Mode (For Public Distribution)

To avoid server hosting and API usage costs for a public release, the app includes a Settings UI (accessible via the Gear icon in the top status bar) enabling runtime configuration:

Tap the Gear icon in the status bar.
Input your own LIVEKIT_URL, static LIVEKIT_TOKEN, or dynamic TOKEN_SERVER_URL.
Tap Save Settings to persist the configuration in UserDefaults.
Note: When configured, user-defined inputs override default compile-time keys. Tap Reset to Defaults to clear these overrides.

Backend Setup

Navigate to the backend directory:
```
cd backend
```

Create and activate a virtual environment:

python3 -m venv venv
source venv/bin/activate

Install dependencies:
```
pip install -r requirements.txt
```
Duplicate the environment variables template:
```
cp .env.example .env
```

Open backend/.env and add your LiveKit credentials and Gemini API Key:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
GOOGLE_API_KEY=your_google_api_key

3. Run the System

A. Start the Translation Agent

The agent subscribes to the room, connects to Gemini, translates incoming speech, and publishes the audio/subtitles.

cd backend
source venv/bin/activate
python agent.py dev

B. Start the Token Server (Optional)

To run in a secure production mode, launch the token server to issue 1-hour expiration JWTs:

cd backend
source venv/bin/activate
python server.py

By default, the server runs on http://localhost:8080.

C. Run the iOS App

Connect your AirPods or Bluetooth earbuds to your iPhone, launch the StarLink app, and tap the translation toggle to begin.

🛠️ Troubleshooting

iOS App Issues

Audio routes to Phone Speaker / Earpiece: Make sure your earphones are paired and active. If routing fails, toggle Bluetooth in iOS Settings or ensure Bluetooth permission is granted to StarLink.
Captions stuck loading: Make sure the backend agent is running and has a stable connection to LiveKit and Gemini. Check that the Gemini API Key has sufficient quota.
SPM Package errors: Reset Xcode package cache: File -> Packages -> Reset Package Caches.

Backend Agent Issues

GOOGLE_API_KEY not found: Ensure your .env contains the correct key and is loaded.
Agent connects but stays silent: Check that you are speaking clearly in a supported language (English, Japanese, Chinese). Low-quality microphone inputs or loud background noise might be filtered by the agent's noise isolation rules.

📄 License

This project is licensed under the MIT License - see the file for details.

Built for hands-free accessibility and real-time cross-language communication.

Navigation Menu

Repository navigation

About

Repository files navigation

🎙️ StarLink

📖 Overview

🎬 Demo & Portfolio

🎧 Supported Audio Devices

⚡ Core Engineering Challenges Solved (Portfolio Highlights)

1. Bluetooth & MFi Audio Routing Control

2. Acoustic Feedback Loop Prevention

3. Voice-Subtitle Modality Synchronization

4. Zero-Latency Input Language Detection

🏗️ Architecture & Data Flow

📂 Project Structure

⚙️ Setup & Installation

1. Prerequisites

2. Configure Credentials

iOS Client Setup (Static / Pre-configured Mode)

🔑 Bring Your Own Key (BYOK) Mode (For Public Distribution)

Backend Setup

3. Run the System

A. Start the Translation Agent

B. Start the Token Server (Optional)

C. Run the iOS App

🛠️ Troubleshooting

iOS App Issues

Backend Agent Issues

📄 License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
StarLink		StarLink
StarLinkTests		StarLinkTests
backend		backend
case		case
.gitignore		.gitignore
LICENSE		LICENSE
README.ko.md		README.ko.md
README.md		README.md
portfolio.html		portfolio.html
project.yml		project.yml
run_simulator.sh		run_simulator.sh

Latest commit

History

Folders and files

Repository files navigation

🎙️ StarLink

📖 Overview

🎬 Demo & Portfolio

🎧 Supported Audio Devices

⚡ Core Engineering Challenges Solved (Portfolio Highlights)

1. Bluetooth & MFi Audio Routing Control

2. Acoustic Feedback Loop Prevention

3. Voice-Subtitle Modality Synchronization

4. Zero-Latency Input Language Detection

🏗️ Architecture & Data Flow

📂 Project Structure

⚙️ Setup & Installation

1. Prerequisites

2. Configure Credentials

iOS Client Setup (Static / Pre-configured Mode)

🔑 Bring Your Own Key (BYOK) Mode (For Public Distribution)

Backend Setup

3. Run the System

A. Start the Translation Agent

B. Start the Token Server (Optional)

C. Run the iOS App

🛠️ Troubleshooting

iOS App Issues

Backend Agent Issues

📄 License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages