promptdiff — lint, diff, and score LLM prompts (with Ollama A/B testing)

[HadiFrt20]

Sharing a tool I built for managing LLM prompts as code.

If you are building agents or RAG pipelines with LlamaIndex, your system prompts are production-critical but probably have no quality gates.

promptdiff parses .prompt files into typed sections (persona, constraints, examples, format) and applies:

• 10 lint rules: conflicting constraints, vague language, missing examples, no injection guard, role confusion, word limit mismatches
• Semantic diff: "word limit tightened 150→100, high impact" not "+1 -1 line"
• Quality score: 0-100 across 5 dimensions
• A/B compare: run two prompt versions through Ollama and score outputs against constraints
• Prompt composition: extends and includes for DRY prompt management
• MLflow integration: log prompts and scores as experiment runs

Everything local, 3 deps, 217 tests.

GitHub: https://github.com/HadiFrt20/promptdiff

npm install -g promptdiff