feat: Add repo indexer - #502
Conversation
|
|
Codex reviewThe new repo indexers are generally coherent, but clone errors can leak credentials and git-backed discovery/chunk metadata have correctness gaps. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| const result = spawnSync('git', args, { encoding: 'utf8' }) | ||
| if (result.status !== 0) { | ||
| const detail = (result.stderr || result.stdout || 'git clone failed').trim() | ||
| throw new Error(`Failed to clone ${url}: ${detail}`) |
There was a problem hiding this comment.
BLOCKING ```ts
throw new Error(Failed to clone ${url}: ${detail})
This formats the raw remote URL into exceptions. Private-repo callers often use tokenized HTTPS remotes, so a clone failure would put credentials in logs. Redact `username/password` from the URL before building the error, scrub the git stderr/stdout detail as well, and mirror that in the Python clone helper.
| if (options.respectGitignore && isGitWorkTree(root)) { | ||
| const tracked = gitListFiles(root) | ||
| if (tracked !== null) { | ||
| return tracked |
There was a problem hiding this comment.
CONSIDER ```ts
if (tracked !== null) {
return tracked
}
In git worktrees this returns `git ls-files` results before applying `excludeDirs`, so defaults like `node_modules`, `dist`, and `vendor` are ignored whenever those paths are unignored or tracked. Filter tracked paths by path components against `options.excludeDirs` before returning, or apply the exclusion after candidate collection; the Python `_list_candidates` path needs the same fix.
| } | ||
|
|
||
| export function splitLines(content: string): string[] { | ||
| return content.split(/\r?\n/) |
There was a problem hiding this comment.
CONSIDER ```ts
return content.split(/\r?\n/)
For files ending with a newline, this adds a synthetic final empty line, so line metadata can extend past the last real line and diverges from the Python `splitlines()` behavior. Return `[]` for empty content and drop exactly one trailing empty element after splitting while preserving interior blank lines.
| def _is_git_work_tree(root: Path) -> bool: | ||
| if not (root / ".git").exists(): | ||
| return False | ||
| result = subprocess.run( |
There was a problem hiding this comment.
CONSIDER ```py
result = subprocess.run(
["git", "-C", str(root), "rev-parse", "--is-inside-work-tree"],
This probe raises `FileNotFoundError` when a local repo has a `.git` entry but the runtime image does not include `git`, so `discover_files()` fails instead of falling back to `_walk_filesystem`. Check `shutil.which("git")` or catch `FileNotFoundError` in the git probes and return `False`/`None`.
|
@sonikagutha looks good just fix the GitHub actions and then also sign the CLA |
3 participants
Pull Request Checklist
Please ensure that your PR meets the following requirements:
Description
Adds library packages that clone or walk a repository, chunk Markdown and source code, and upload DocumentInfo rows to a Moss index for semantic codebase search.
Packages
packages/moss-repo-indexer (Python)
packages/moss-repo-indexer-js (@moss-tools/repo-indexer, TypeScript)
Behavior
Resolve a local path or shallow-clone a git URL
Discover files with include globs / exclude dirs (respects git ls-files when present)
Chunk Markdown (heading-aware) and code (sliding window + symbol heuristics)
build_documents / sync with dry-run, recreate, or upsert upload
Shared document metadata contract (path, language, type, start_line, end_line, symbol, navigation, etc.)
Fixes #382
Type of Change