Skip to content

About

Agent skill to prevent the LLM models read the raw names of people, business from markdown files

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

Pseudonymizating-markdown-skill

한국어 | English

Requirements

  • Python 3.10 or later.
  • No external packages.

Usage

Convert a single Markdown file

python scripts/pseudonymize.py "input.md" "manifest.md" -o "result.md"

Convert many Markdown files under a folder -> use a config.yaml file

python scripts/pseudonymize.py -c config.yaml

Options

Option Mode Description Constraints
input.md manifest.md Single file Markdown to convert and the pseudonymization dictionary Cannot be combined with -c
-o, --output Single file Output file path Default: <input>.pseudonymized.md. Cannot be combined with -c
-c, --config Batch Convert many files under a folder using config.yaml Cannot be combined with the input, the manifest or -o
--log-dir Both Folder for the run log (relative to the current folder) Batch: defaults to log_dir in the config. Single file: a log is written only when given
--overwrite Both Overwrite existing result files Inputs, the manifest and the config are always refused
-f, --force Both Includes --overwrite, and also overwrites or deletes results edited by hand Matters most with --sync and --prune
--sync Batch Re-convert only the files that changed Needs -c and a fixed output_dir
--prune Batch Delete the results of inputs that no longer exist Needs --sync. Results edited by hand are deleted only with -f
--restore Batch Restore new files from the pseudonymized folder into input_dir Needs -c. Cannot be combined with --sync, --prune, --overwrite or -f
--from Batch Pseudonymized folder to restore from (relative to the current folder) Needs --restore. Defaults to output_dir (required when it is empty)
--version Both Print the version from git tags

config.yaml

Example: config.yaml

Key Description etc
manifest Pseudonymization dictionary
input_dir Input folder
output_dir Output folder If omitted: ./output/<run time>
log_dir Folder for run logs If omitted: ./logs
manifest_log_dir Folder for the manifest log. It stores names before pseudonymization, so be careful. If omitted: log_dir
include Patterns of files to process
except Patterns of files to exclude
  • Paths can also be written in Windows form (G:\folder\file, .\folder).
    • On non-Windows environments (WSL, etc.), G:\... is read as /mnt/g/... and \ as /.
  • --log-dir and --from on the command line are relative to the current folder.
  • Values in the config are relative to the config folder.
  • Only files that match include (default **/*.md) and do not match except are processed.
    • except takes priority, and a matching folder excludes everything below it.
    • glob: *, ?, **, [a-z], [!x], {a,b}. * and ? do not cross /.
    • A pattern ending with /, such as draft/, means everything below that folder. Windows-style patterns with \, such as docs\**\*.md, are read with /.
  • log_dir and manifest_log_dir must not be the same as input_dir or output_dir (for restore, the --from folder).
    • Logs and state/ contain text from before pseudonymization. They are rejected with an error so that they are not converted again as input or mixed into the result folder.
  • Results are saved in output_dir with the same subfolder structure and file names. Subfolders are created automatically.
  • If output_dir is empty or omitted, the results are saved in output/<run time>/ (for example output/20261102-090410/).
    • The folder name is the same as the log file name.
  • YAML supports only the following.
    • key: value, - item lists, comments and quoted strings
    • Wrap values that start with * or { in quotes.
  • The manifest, the config, the output and log folders, and symbolic links are excluded from the input.
  • If some files fail, the rest are still processed and the exit code is 1.

Run log

Example: logs/example.log

  • Every batch run creates logs/YYYYMMDD-HHMMSS.log (plain text, permission 0600).
    • A single-file run creates one only when --log-dir is given.
  • If a file name contains a real name, it is also recorded in the log.
    • The log warns the agent not to share or print it without a user request.
  • Original text, pseudonyms and Keys are not recorded in the run log.
    • It lists only the number of removed lines and replacements per file, and the match count per manifest row number.

Sync only changed files under a folder (--sync)

python scripts/pseudonymize.py -c config.yaml --sync

  • After editing the sources a little and running again, only the changed files are re-converted.
    • Even with many files, unchanged files are not read; only stat is checked.
    • Use it only when output_dir is fixed in config.yaml.
      • If output_dir is empty (including a value with only a comment, such as output_dir: #default), the run ends with an error before any conversion.
      • No log or state file is created in that case, so nothing appears in logs/.
      • Set output_dir to an existing folder, or run without --sync.
  • The sync state is saved in <log_dir>/state/<hash>/<hash>.sync.json.
    • Default permission 0600
    • Records per-file size, modification time and content hash, and the manifest hash
    • Created separately for each input/output folder pair
  • How each input file is judged
    • Same size and modification time: unchanged
    • Only the modification time changed and the content is the same: unchanged
    • Content changed: ok (updated)
    • Result file is gone: ok (restored)
  • If the manifest file changed, all files are converted again.
    • Changing only the note column does not count.
  • With the -f option, sync is forced even when a pseudonymized file was edited and no longer matches.
    • Without this option, a failed: output was edited ... error occurs.
  • With the --overwrite option, both edited and unedited files are overwritten.
  • Results of files that no longer exist in the input are not deleted by default. They are listed as kept in [orphan] of the log.
  • With the --prune option, only results that are exactly as made by the last sync are deleted, and emptied subfolders are cleaned up.
    • Results edited by hand are deleted only with -f.
    • So run --sync alone first, check the [orphan] list, and then use it.
  • The sync state file contains file paths and content hashes, so it is as private as the logs.
    • If you delete it, the next run becomes a first sync.

Restore (--restore)

python scripts/pseudonymize.py -c config.yaml --restore [--from <pseudonymized folder>]

  • The agent reads only pseudonymized files, writes a report and saves it in the pseudonymized folder (output_dir). The new Markdown file is reverse-mapped using the manifest and saved at the same relative path in the original folder (input_dir).
    • The agent does not need to see the original text.
  • If output_dir is empty (a new folder every run), specify the folder with --from output/<time>.
  • It cannot be combined with --sync, --prune, --overwrite or -f.
  • How each file is judged
    • If it is in the restore data and the pseudonymized file is unchanged: unchanged.
      • For --sync files that are not in the restore data, the same judgment uses the result hash in the state file.
    • If it is in the restore data but the pseudonymized file was edited: skipped (edited existing file ...).
      • Restoring edits to existing files is not supported yet (the original is left untouched).
    • A new file that is not in the restore data is restored.
      • If a file already exists at the original path, it is not overwritten and the result is failed.
  • Restore turns the Value of the manifest table back into the Key.
    • The aliases of a row are restored to the first Key.
    • When pseudonyms overlap, the longer one takes priority.
    • Text removed with remove_text or remove_line is not in the new file, so it is not restored.
  • Round-trip check: the restored text is pseudonymized again.
    • If it equals the pseudonymized file, it is registered in the restore data and the sync state (unchanged on the next --sync).
    • If it differs, it is marked ok (roundtrip differs) with a [warning].
  • The log is <log_dir>/YYYYMMDD-HHMMSS.restore.log
    • Example (fake data): logs/example.restore.log
    • Permission 0600
    • Original text and pseudonyms are not recorded.
    • No manifest log is created.

Tool-only files (state/)

logs/
    20261102-090410.log
    20261102-090410.manifest.log
    state/                       (permission 0700)
        <hash>/                  (permission 0700, one per input/output folder pair)
            <hash>.sync.json     (permission 0600)
            <hash>.restore.json  (permission 0600)
  • <hash> is a 10-character hash made from the input and output folder paths.
    • Folder paths and names are not used.
  • While a run is in progress, <hash>.lock briefly appears in the same folder.
  • They contain paths and parts of the original text, so the agent and other people must not read or share them.

<hash>.sync.json (sync state)

  • Used by --sync.
  • Records the manifest hash and, per file, the size, modification time, input hash and result hash.

<hash>.restore.json (restore data)

  • On every (batch) pseudonymization, per-file records are saved in this single file.
  • Contents
    • Top level: manifest hash (rules_sha), script hash (script)
    • Per file: input and result hash, number of original lines, the number and full text of removed lines, replacement positions (line, column, original, pseudonym)
    • It is a partial copy of the original text, so it is as private as the manifest log.
  • With --sync, only the records of re-converted files are updated.
  • When --prune deletes a result, the record of that file is deleted too.
  • Synced files without a record are filled in automatically when the hashes match.
  • If the manifest changed, the old records are discarded.
  • If it cannot be read, a conversion rebuilds it and notes it in the log [warning], while --restore stops with an error.
  • Pseudonym collision warnings
    • If a Value in the manifest is part of another Value, or a result contains pseudonym text that no rule produced, it is reported to stderr and in the log [warning].
    • It means that restoring may become ambiguous.

How logs are written

  • The two logs (run log, manifest log) are opened before conversion starts and appended to after each file.
    • Even if the run is interrupted, what was recorded up to that point remains.
  • The result: line in [result] at the top of the run log is the run status.
    • While running, or after a forced kill: running
    • Finished normally: N ok, M failed
    • Ended by an exception or Ctrl+C: aborted (N ok, M failed so far)
    • The field has a fixed width (padded with spaces) and is overwritten when the run ends.
  • Layout: warning, header (files: = number of files to process), [result], [file] (one line per file), [manifest] (match count per row number)
    • [manifest] is written at the end.
    • The manifest log keeps a table format, so judge completion from the run log.
  • If a log cannot be created (for example a rerun in the same second), nothing is converted and the run ends with an error.

Manifest log (private)

Example (fake data): logs/example.manifest.log

  • Every run also creates YYYYMMDD-HHMMSS.manifest.log with the same time as the run log.
    • The location is manifest_log_dir in the config.
    • If it is empty or omitted, it is saved in log_dir together.
  • It records the original text (Key) and pseudonym (Value) as they are for each converted file.
    • It is as private as the manifest.
    • Like the run log, its permission is 0600 (owner only); this does not apply on Windows.
    • .gitignore excludes *.manifest.log in any folder.
  • As the warning at the top says, the agent and other people must not read or share it.
  • The format is a single table: output row action count key value (tab-separated)
    • output is the absolute path of the converted result file.
    • There is one line per matched (result file, rule); files with no match have no line.
  • row is the line number in the manifest file and equals the row number in the [manifest] section of the run log.

Incident handling summary

For the detailed table, see 사고 대응 (incident handling) in SKILL.md.

  • Basic principle: do not overwrite or delete, and when in doubt, stop or record a failure.
  • Mixed-up paths
    • Overlapping input_dir/output_dir/log_dir are rejected.
    • A --from without pseudonymization records is refused for restore.
  • Changes during a run
    • A run keeps using the config and manifest it read at the start.
    • --restore after the manifest changed is refused; pseudonymize again with --sync.
    • Check which code ran by the script: hash in the log.
  • Deletion accidents
    • If results or logs disappear during a run, this is detected at the end and reported as a failure (exit code 1).
    • If the log folder was deleted, rebuild the records with --sync --overwrite.
  • Concurrent runs
    • Only one run per folder pair is allowed, using state/<hash>/<hash>.lock.
    • The lock of a dead process is taken over automatically.
  • Interrupted writes
    • Result files are written completely to a temporary file and then renamed, so no truncated file remains.

Tests

python -m unittest discover -s tests -v

  • Fake data is used in temporary folders, so real documents and manifests are never touched.
  • Covers glob, the YAML parser, path conversion, log formats, --overwrite/-f/--sync/--prune, restore and incident handling.

Version

python scripts/pseudonymize.py --version

Manifest format

  • manifest.md table: Key | Value | note | action
  • If the action column or cell is missing, it is replace.
    • note is used only as a description.
  • action types
    • replace: replace Key with Value (Value required)
    • remove_text: delete only the Key string (Value left empty)
    • remove_line: delete the whole line containing the Key (Value left empty)
  • A row starts with |.
    • Key and Value are the first two columns; note/action are optional columns.
  • The Key cell can hold several aliases separated by commas or <br>, and all are handled as the same Value.
    • Write a literal comma as ,.
    • Empty aliases caused by repeated or trailing separators are ignored.
    • Spaces inside an alias are kept.
  • Rows with all cells empty are skipped.

Troubleshooting

  • If a Key is duplicated, only a warning is shown and the rule of the last row read is used.
  • If different rows use the same Value, the run is aborted.
    • Gather the originals that should share a pseudonym as aliases in one row.
  • A Key with no alias, an invalid action or a wrong number of columns is an error.
  • Error and warning messages show only row numbers.

Priority

  • Line removal is done first, then the remaining text is replaced or deleted in one pass.
  • At the same position, the longer string wins, and replaced text is not replaced again.
  • Matching is exact and case-sensitive, with no word-boundary or particle adjustment.

Others

  • UTF-8 input, UTF-8 (no BOM) output.
  • LF/CRLF is preserved.
  • Originals, the manifest and existing outputs are not overwritten by default.
    • --overwrite: overwrites existing result files only.
      • Originals, the manifest and the config are always refused.
      • With --sync, existing results unknown to the state file are overwritten too.
    • -f (--force): includes --overwrite and also goes past the safeguards.
      • With --sync, it overwrites results edited by hand.
      • With --prune, it also deletes results edited by hand.
    • Files are written to a temporary file and then replaced, so an existing result is not broken by a mid-way failure.
      • Files behind hard links or symbolic links are not changed.
      • Logs are never overwritten, whatever the options.
    • --prune deletes only regular files inside output_dir.
      • Paths outside it, symbolic links and folders are not deleted even with -f.
  • The parent folder of the output must exist.
  • If -o is omitted, <source name>.pseudonymized.md is created.
  • On Windows you can also use py, and on WSL python3.
  • Personal information that is not in the manifest, and image or external link targets, are not detected automatically.
  • It applies to the whole text, including tables, code and links, so check the document structure after conversion.
  • Keep the manifest and any result you share separately.
  • Old JSON rules are still read for compatibility, but cannot be mixed with a table in one file.

About

Agent skill to prevent the LLM models read the raw names of people, business from markdown files

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages