DataDot

A powerful RAG (Retrieval-Augmented Generation) application for intelligent document processing and AI-powered conversations

---

📋 Table of Contents

• Overview
• Features
• Architecture
• Tech Stack
• Getting Started
  • Prerequisites
  • Installation
  • Configuration
• Running the Application
  • Development Mode
  • Production Mode
  • Docker Deployment
• Project Structure
• Development
• Testing
• Environment Variables
• Contributing
• License

---

Overview

DataDot is a comprehensive RAG application that combines a modern Next.js frontend with powerful Python backend services to provide:

• Multi-LLM Support: Integration with OpenAI, Anthropic, Google Gemini, Cohere, and Ollama
• Smart Document Processing: Convert and process PDFs, Word docs, Excel files, images, audio, and more
• Intelligent Chat Interface: Real-time streaming conversations with AI agents
• Workspace Management: Organize documents and conversations by project
• Vector Search: Multiple vector database support (LanceDB, ChromaDB, Pinecone, Qdrant, Weaviate)
• MCP Server: Model Context Protocol server for AI agent integration
• Customizable Themes: Dark/light mode with custom branding
• Internationalization: Multi-language support via i18next

---

Features

Frontend

• Built with Next.js 16 (App Router) and React 19
• Real-time chat with streaming responses (SSE + WebSockets)
• Responsive design with mobile support
• Custom theme system with CSS variables
• JWT-based authentication
• Drag-and-drop file uploads
• Text-to-speech and speech-to-text capabilities
• Data visualization with Recharts

Backend Server

• FastAPI with async support
• Multi-user mode with workspace isolation
• Secure authentication with JWT tokens
• Unified LLM interface for multiple providers
• Vector database operations for RAG
• Background job scheduling
• Comprehensive API documentation (OpenAPI/Swagger)

Document Collector

• Support for 9+ file formats (PDF, DOCX, XLSX, EPUB, MBOX, images, audio)
• Audio transcription with OpenAI Whisper
• OCR for images using Tesseract
• Web scraping with depth control
• GitHub/GitLab repository ingestion
• YouTube transcript extraction
• Obsidian Vault processing
• Automatic token counting and text chunking

MCP Server

• Model Context Protocol (MCP) server for AI agent integration
• SSE (Server-Sent Events) transport for real-time communication
• Dynamic configuration updates at runtime
• API key authentication
• Multiple LLM provider support
• Document upload and processing tools
• Vector database query tools
• Workspace management tools

---

Architecture

DataDot consists of four main components:

┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
│   Frontend      │         │  Backend Server │         │   Collector     │
│   (Next.js)     │◄───────►│   (FastAPI)     │◄───────►│   (FastAPI)     │
│   Port 3000     │         │   Port 3001     │         │   Port 8888     │
└─────────────────┘         └────────┬────────┘         └─────────────────┘
        │                            │                           │
        │                            │                           │
        │                   ┌────────┴────────┐                  │
        │                   │   MCP Server    │                  │
        │                   │   (FastAPI)     │                  │
        │                   │   Port 8080     │                  │
        │                   └────────┬────────┘                  │
        │                            │                           │
        v                            v                           v
  User Interface          ┌─────────────────┐         Document Processing
  Chat & Workspaces       │ Vector Database │         File Conversion
  Theme System            │  SQLite/Postgres│         OCR & Transcription
  AI Agent Integration    │   Embeddings    │         Web Scraping
                          └─────────────────┘

Communication Flow

1. User uploads document → Frontend → Collector → Processes and returns metadata
2. User sends chat message → Frontend → Backend → LLM Provider → Streaming response
3. Backend queries documents → Vector DB → Retrieves relevant context → Enhances LLM prompt
4. AI agents connect → MCP Server → Backend → Access documents and workspaces

---

Tech Stack

Frontend

• Framework: Next.js 16 with App Router and Turbopack
• Language: TypeScript 5
• UI: React 19, Tailwind CSS, Radix UI
• State: React Context API
• Streaming: Server-Sent Events (SSE), WebSockets
• Charts: Recharts
• Icons: Phosphor Icons, Lucide React

Backend

• Framework: FastAPI 0.110
• Language: Python 3.9+
• Database: SQLAlchemy (SQLite/PostgreSQL)
• Authentication: JWT with passlib/bcrypt
• LLMs: OpenAI, Anthropic, Google Gemini, Cohere, Ollama
• Vectors: LanceDB, ChromaDB, Pinecone, Qdrant, Weaviate
• Background Jobs: APScheduler

Document Processing

• Framework: FastAPI 0.115
• PDF: PyPDF2, PyMuPDF
• Office: python-docx, openpyxl, python-pptx, odfpy
• OCR: pytesseract
• Audio: OpenAI Whisper, ffmpeg
• Web: httpx, BeautifulSoup4, Playwright
• Tokens: tiktoken

---

Getting Started

Prerequisites

• Node.js 18+ and npm
• Python 3.9+
• Git
• Tesseract (for OCR) - Optional but recommended
• FFmpeg (for audio processing) - Optional but recommended
• Docker (for containerized deployment) - Optional

Installing System Dependencies

Windows (using Chocolatey):

choco install tesseract ffmpeg

macOS (using Homebrew):

brew install tesseract ffmpeg

Linux (Ubuntu/Debian):

sudo apt-get update
sudo apt-get install tesseract-ocr ffmpeg

Installation

1. Clone the repository

git clone https://github.com/meetpatel006/datadot.git
cd datadot

2. Install Frontend Dependencies

npm install

3. Install Backend Server Dependencies

cd server
python -m venv venv

# Windows
venv\Scripts\activate
# Linux/Mac
source venv/bin/activate

pip install -r requirements.txt
cd ..

4. Install Collector Dependencies

cd collector
python -m venv venv

# Windows
venv\Scripts\activate
# Linux/Mac
source venv/bin/activate

pip install -r requirements.txt

# Install Playwright browsers
playwright install chromium

cd ..

Configuration

1. Frontend Environment - Create .env.local in the root directory:

NEXT_PUBLIC_API_BASE=http://localhost:3001/api
NODE_ENV=development

2. Backend Server Environment - Create .env in the server/ directory:

# Server Configuration
PORT=3001
HOST=0.0.0.0
WORKERS=1
RELOAD=true
LOG_LEVEL=INFO

# Database
DATABASE_URL=sqlite:///./datadot.db

# Security
JWT_SECRET_KEY=your-secret-key-change-this
JWT_ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=10080

# Multi-user Mode
MULTI_USER_MODE=true

# LLM Providers (add your API keys)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
COHERE_API_KEY=...

# Collector Integration
COLLECTOR_API_URL=http://localhost:8888
COLLECTOR_API_KEY=your-collector-key

# Vector Database (choose one)
VECTOR_DB=lancedb  # or chromadb, pinecone, qdrant, weaviate

3. Collector Environment - Create .env in the collector/ directory:

# Server Configuration
PORT=8888
HOST=0.0.0.0
NODE_ENV=production
LOG_LEVEL=INFO

# Security
COLLECTOR_API_KEY=your-collector-key

# AI Services
OPENAI_API_KEY=sk-...  # For Whisper audio transcription

# External Services (optional)
GITHUB_TOKEN=ghp_...
GITLAB_TOKEN=glpat-...

---

Running the Application

Development Mode

You'll need to run all three services in separate terminals:

Terminal 1 - Frontend:

npm run dev
# Access at http://localhost:3000

Terminal 2 - Backend Server:

cd server
venv\Scripts\activate  # Windows
# source venv/bin/activate  # Linux/Mac
python main.py
# API at http://localhost:3001

Terminal 3 - Document Collector:

cd collector
venv\Scripts\activate  # Windows
# source venv/bin/activate  # Linux/Mac
python run.py
# API at http://localhost:8888
# Docs at http://localhost:8888/docs

Production Mode

Frontend:

npm run build
npm start

Backend Services:

# Use production ASGI server with multiple workers
cd server
uvicorn app.main:app --host 0.0.0.0 --port 3001 --workers 4

cd collector
uvicorn app.main:app --host 0.0.0.0 --port 8888 --workers 2

Docker Deployment

The easiest way to run the entire stack:

cd docker

# Copy and configure environment
cp .env.example .env
# Edit .env with your configuration

# Build and start all services
docker compose -f docker-compose.combined.yml up --build

# Or run in detached mode
docker compose -f docker-compose.combined.yml up -d --build

# View logs
docker compose -f docker-compose.combined.yml logs -f

# Stop services
docker compose -f docker-compose.combined.yml down

Services will be available at:

• Frontend: http://localhost:3000
• Backend API: http://localhost:3001
• Collector API: http://localhost:8888
• MCP Server: http://localhost:8080
• API Documentation: http://localhost:8888/docs

---

📁 Project Structure

Datadot/
├── app/                      # Next.js App Router pages
│   ├── Main/                 # Dashboard and home
│   ├── workspace/            # Workspace pages
│   ├── login/                # Authentication pages
│   └── layout.tsx            # Root layout
├── components/               # React components
│   ├── WorkspaceChat/        # Chat UI components
│   ├── Sidebar/              # Navigation
│   └── Modals/               # Modal dialogs
├── context/                  # React Context providers
│   ├── AuthContext.tsx       # Authentication state
│   ├── ThemeContext.tsx      # Theme management
│   └── I18nProvider.tsx      # Internationalization
├── models/                   # API client layer
│   ├── workspace.ts          # Workspace API
│   ├── system.ts             # System API
│   └── admin.ts              # Admin API
├── utils/                    # Helper functions
│   ├── paths.ts              # Route definitions
│   ├── request.ts            # HTTP utilities
│   └── constants.ts          # App constants
├── hooks/                    # Custom React hooks
├── types/                    # TypeScript definitions
├── i18n/                     # i18next configuration
├── lib/                      # Shared libraries
├── public/                   # Static assets
├── server/                   # Python FastAPI backend
│   ├── app/
│   │   ├── api/              # API endpoints
│   │   ├── core/             # Configuration
│   │   ├── crud/             # Database operations
│   │   ├── models/           # SQLAlchemy models
│   │   ├── schemas/          # Pydantic schemas
│   │   ├── services/         # Business logic
│   │   └── utils/            # Utilities
│   ├── main.py               # Entry point
│   └── requirements.txt      # Dependencies
├── collector/                # Document processor
│   ├── app/
│   │   ├── converters/       # File converters
│   │   ├── extensions/       # External loaders
│   │   ├── processors/       # Processing logic
│   │   ├── routes/           # API routes
│   │   └── utils/            # Utilities
│   ├── run.py                # Entry point
│   └── requirements.txt      # Dependencies
├── mcp/                      # MCP Server
│   ├── app/
│   │   ├── tools/            # MCP tools
│   │   ├── prompts/          # MCP prompts
│   │   ├── config.py         # Configuration
│   │   ├── utils.py          # Utilities
│   │   └── server.py         # MCP server setup
│   ├── main.py               # Entry point
│   └── requirements.txt      # Dependencies
├── docker/                   # Docker configuration
│   ├── docker-compose.combined.yml
│   └── .env.example
├── package.json              # Frontend dependencies
├── next.config.ts            # Next.js config
├── tailwind.config.js        # Tailwind config
└── tsconfig.json             # TypeScript config

---

💻 Development

Code Style

Frontend:

• Follow existing TypeScript patterns
• Use Tailwind utility classes
• Prefer functional components with hooks
• Use "use client" directive for client-side features

Backend:

• Follow PEP 8 style guide
• Use type hints for all functions
• Write docstrings for public APIs
• Use async/await for I/O operations

Linting

# Frontend
npm run lint

# Backend (if configured)
cd server
ruff check .

cd collector
ruff check .

Adding Features

New Frontend Page:

1. Create page in app/ directory
2. Add route helper to utils/paths.ts
3. Update navigation in components/Sidebar/

New API Endpoint:

1. Add route in server/app/api/
2. Create service logic in server/app/services/
3. Add schema in server/app/schemas/

New Document Converter:

1. Create converter in collector/app/converters/
2. Add MIME type mapping in utils/constants.py
3. Register in processor routing

---

🧪 Testing

Frontend

No test framework currently configured. Test manually:

npm run dev
# Test in browser at http://localhost:3000

Backend

cd server
pytest                          # Run all tests
pytest -v                       # Verbose output
pytest --cov=app tests/        # With coverage

cd ../collector
pytest                          # Run collector tests

---

🔐 Environment Variables

Frontend (.env.local)

Variable	Description	Default
NEXT_PUBLIC_API_BASE	Backend API URL	http://localhost:3001/api
NODE_ENV	Environment mode	development

Backend Server (server/.env)

Variable	Description	Required
DATABASE_URL	Database connection string	✅
JWT_SECRET_KEY	Secret for JWT tokens	✅
OPENAI_API_KEY	OpenAI API key	⚠️
ANTHROPIC_API_KEY	Anthropic API key	⚠️
COLLECTOR_API_URL	Collector service URL	✅
MULTI_USER_MODE	Enable multi-user	❌

Collector (collector/.env)

Variable	Description	Required
COLLECTOR_API_KEY	API authentication key	✅
OPENAI_API_KEY	For Whisper transcription	⚠️
GITHUB_TOKEN	GitHub API access	❌
GITLAB_TOKEN	GitLab API access	❌

⚠️ = Required for specific features

---

🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and test thoroughly
4. Commit your changes: git commit -m 'Add amazing feature'
5. Push to the branch: git push origin feature/amazing-feature
6. Open a Pull Request

Contribution Guidelines

• Write clear, descriptive commit messages
• Add tests for new features
• Update documentation as needed
• Follow existing code style and patterns
• Ensure all tests pass before submitting PR

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Acknowledgments

• Next.js team for the amazing framework
• FastAPI for the excellent Python web framework
• All the open-source libraries that make this project possible

---

Support

For issues, questions, or contributions:

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Built with ❤️ using Next.js, React, and FastAPI