Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ jobs:
- name: Install dependencies
run: npm ci

- name: Run typecheck
run: npm run typecheck

- name: Run lint
run: npm run lint

Expand Down
4 changes: 2 additions & 2 deletions capability.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Capability

Web-page front-end quality audit. Kilotest tests public web pages for front-end quality, namely accessibility, usability, and standards conformity, and makes structured findings available for retrieval.
Web-page front-end quality audit. Kilotest tests web pages for front-end quality, namely accessibility, usability, and standards conformity, and makes structured findings available for retrieval.

## Purpose

Expand Down Expand Up @@ -36,7 +36,7 @@ A list of available reports.

## Side effects

Kilotest reads the specified public web page during testing. It does not modify the target site. Testing is asynchronous and may take up to approximately one day before reports are available for newly requested pages.
Kilotest reads the specified web page during testing. It does not modify the target site. Testing is asynchronous and may take up to approximately one day before reports are available for newly requested pages.

## Cost

Expand Down
6 changes: 3 additions & 3 deletions docs/AI-TOOLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The external features that support the use of Kilotest as a collection of AI too
- A [pull request](https://github.com/public-apis/public-apis/pull/6346/changes) to add Kilotest to the list of public APIs in the `public-apis` repository.
- An [issue](https://github.com/APIs-guru/openapi-directory/issues/2677) to add Kilotest to `openapi-directory`.
- A [pull request](https://github.com/w3c/wai-evaluation-tools-list/pull/1153) to add Kilotest to the WAI evaluation tools list.
- Registration of Kilotest as an active server in the [official MCP Registry](https://registry.modelcontextprotocol.io/v0/servers?search=kilotest) (`io.github.jrpool/kilotest`), maintained by the [Model Context Protocol](https://modelcontextprotocol.io/) project. The [PulseMCP](https://www.pulsemcp.com/servers) directory and the VS Code MCP gallery both crawl this registry, so Kilotest is discoverable through them with no separate submission.
- Registration of Kilotest as an active server in the [official MCP Registry](https://registry.modelcontextprotocol.io) (`io.github.jrpool/kilotest`), maintained by the [Model Context Protocol](https://modelcontextprotocol.io/) project. The [PulseMCP](https://www.pulsemcp.com/servers) directory and the VS Code MCP gallery both crawl this registry, so Kilotest is discoverable through them with no separate submission.
- Registration of Kilotest with the [Smithery](https://smithery.ai/servers/pool/kilotest) MCP server registry.
- Registration of Kilotest with the [Glama](https://glama.ai/mcp/connectors/com.kilotest/kilotest) MCP server registry, including a claim of ownership of the MCP connector by the maintainer via a `TXT` DNS record for `_glama-claim.kilotest.com`, with the value `glama_claim_gVV4HrPdcXr9TEfj1p3icEUBJ3_OVJGX`.
- Registration of Kilotest with [Awesome MCP Servers](https://www.tensorblock.co/mcp/servers/github-jrpool-kilotest-f518ba36).
Expand Down Expand Up @@ -107,7 +107,7 @@ Other future work:
Capability: web-page-quality-audit

Purpose:
Test a public web page for accessibility,
Test a web page for accessibility,
usability, HTML and web-standards problems.

Interfaces:
Expand All @@ -123,7 +123,7 @@ Other future work:
rule-engine diagnoses, WCAG mappings

Side effects:
Reads the specified public web page.
Reads the specified web page.
Does not modify the target site.

Cost:
Expand Down
2 changes: 1 addition & 1 deletion docs/SERVICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,7 @@ Kilotest records basic counts of how it is used, so the maintainer can answer qu

- The generic `.html` GET dispatch in `index.ts`, once per successfully served web page, recorded under the `pageViews` category by page name (for example `tutorialWeb`, `listReports`), unless the page is a manager-only page (see "Manager page activity" below), in which case it is recorded there instead.
- The home page (`/` and `/index.html`), served by its own branch in `index.ts` rather than through the generic `.html` dispatch, recorded under `pageViews` as `index`.
- Each of the 8 MCP tool handlers in `mcp.ts`, once per successful tool call, recorded under the `mcpToolCalls` category by tool name (for example `listReports`, `requestTest`).
- Each of the 9 MCP tool handlers in `mcp.ts`, once per successful tool call, recorded under the `mcpToolCalls` category by tool name (for example `listReports`, `requestTest`).
- The `/api/*` GET and POST service branches in `index.ts`, once per call, recorded under the `apiOperations` category by operation name.

`db/metrics.json` also stores a `since` time stamp, set when the file is first created, so the counts can be read as "since this date" rather than assumed to cover Kilotest's entire history. `getMetrics()` backfills any category absent from an existing `db/metrics.json` (for example one written before a category such as `managerActivity` existed), so an older file remains readable rather than causing every subsequent request to error.
Expand Down
2 changes: 1 addition & 1 deletion llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Kilotest is an application that performs ensemble testing of web pages for front

## What Kilotest does for language models

Kilotest is deployed on the public Internet as a service at `https://kilotest.com`. Kilotest can test only web pages that can be accessed from the public Internet and that are not protected by authentication or other access controls. Kilotest has not implemented any mechanism for testing private, internal, password-protected, or otherwise restricted pages.
Kilotest is deployed on the public Internet as a service at `https://kilotest.com`. That deployment of Kilotest can test only web pages that can be accessed by its Testaro worker with a URL. Generally those are pages on the public Internet that are not protected by authentication or other access controls.

Without calling any tool, an LLM can, under some conditions, load web pages and use its knowledge to assess successes and failures of front-end quality. But such an assessment is almost always:

Expand Down
2 changes: 1 addition & 1 deletion llms.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Kilotest

Kilotest helps language models and AI platforms to give thorough, accurate and truthful answers when users ask about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages. Kilotest integrates an ensemble of twelve independent rule engines to test any public web page. Kilotest creates a structured report of the results and saves the report for future reference. Kilotest API endpoints and MCP tools can provide multipage, single-page, and element-level test results.
Kilotest helps language models and AI platforms to give thorough, accurate and truthful answers when users ask about the front-end quality (i.e. accessibility, usability, and standards conformity) of web pages. Kilotest integrates an ensemble of twelve independent rule engines to test a web page. Kilotest creates a structured report of the results and saves the report for future reference. Kilotest API endpoints and MCP tools can provide multipage, single-page, and element-level test results.

## API

Expand Down
45 changes: 41 additions & 4 deletions mcp.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import http from 'node:http';
import {mcpPath, createMCPServer, handleMCP} from './mcp.ts';
import {metricsPath} from './util.ts';
import fs from 'node:fs';
import {Client} from '@modelcontextprotocol/sdk/client/index.js';
import {InMemoryTransport} from '@modelcontextprotocol/sdk/inMemory.js';

// CONSTANTS

Expand All @@ -35,11 +37,12 @@ test('mcpPath is /mcp', () => {
assert.equal(mcpPath, '/mcp');
});

test('createMCPServer registers all 8 tools', () => {
test('createMCPServer registers all 9 tools', () => {
const server = createMCPServer();
const toolNames = Object.keys((server as any)._registeredTools);
assert.equal(toolNames.length, 8);
assert.equal(toolNames.length, 9);
assert.deepEqual(toolNames, [
'getKilotestOverview',
'listReports',
'listIssues',
'listViolators',
Expand All @@ -56,10 +59,41 @@ test('each tool has a description and a handler function', () => {
const tools = (server as any)._registeredTools;
for (const [name, tool] of Object.entries(tools) as [string, any][]) {
assert.ok(tool.description, `${name} has a description`);
assert.ok(
tool.description.includes('front-end quality'),
`${name} description includes the shared domain context`
);
assert.equal(typeof tool.handler, 'function', `${name} has a handler function`);
}
});

test('getKilotestOverview handler returns overview text and records a metric', async () => {
const server = createMCPServer();
const countBefore = getToolCallCount('getKilotestOverview');
const result = await (server as any)._registeredTools.getKilotestOverview.handler({});
assert.ok(result.content);
assert.equal(result.content[0].type, 'text');
assert.ok(result.content[0].text.includes('front-end quality'));
assert.equal(getToolCallCount('getKilotestOverview'), countBefore + 1);
});

test('client receives instructions, server description, and the overview resource', async () => {
const server = createMCPServer();
const [clientTransport, serverTransport] = InMemoryTransport.createLinkedPair();
await server.connect(serverTransport);
const client = new Client({name: 'test-client', version: '1.0.0'});
await client.connect(clientTransport);
assert.ok(client.getInstructions()?.includes('front-end quality'));
assert.ok(client.getServerVersion()?.description?.includes('front-end quality'));
const resources = await client.listResources();
assert.deepEqual(resources.resources.map((r: any) => r.uri), ['docs://kilotest/overview']);
const read = await client.readResource({uri: 'docs://kilotest/overview'});
assert.equal(read.contents[0]?.mimeType, 'text/plain');
assert.ok((read.contents[0] as any).text.includes('front-end quality'));
await client.close();
await server.close();
});

test('listReports handler returns content and structuredContent', async () => {
const server = createMCPServer();
const countBefore = getToolCallCount('listReports');
Expand Down Expand Up @@ -258,23 +292,26 @@ test('handleMCP responds to initialize with server info', async () => {
assert.equal(res.statusCode, 200);
const result = parseSSEResult(res.body);
assert.equal(result.result.serverInfo.name, 'Kilotest');
assert.ok(result.result.serverInfo.description.includes('front-end quality'));
assert.ok(result.result.instructions.includes('front-end quality'));
assert.equal(result.result.protocolVersion, '2025-06-18');
}
finally {
await closeMCPServer(server);
}
});

test('handleMCP lists all 8 tools via tools/list', async () => {
test('handleMCP lists all 9 tools via tools/list', async () => {
const server = await startMCPServer();
try {
const port = server.address().port;
const res = await sendMCPRequest(port, 'tools/list', {}, 2);
assert.equal(res.statusCode, 200);
const result = parseSSEResult(res.body);
const toolNames = result.result.tools.map((t: any) => t.name);
assert.equal(toolNames.length, 8);
assert.equal(toolNames.length, 9);
assert.deepEqual(toolNames, [
'getKilotestOverview',
'listReports',
'listIssues',
'listViolators',
Expand Down
Loading
Loading