한국어 | English
- Python 3.10 or later.
- No external packages.
Convert a single Markdown file
python scripts/pseudonymize.py "input.md" "manifest.md" -o "result.md"
Convert many Markdown files under a folder -> use a config.yaml file
python scripts/pseudonymize.py -c config.yaml
| Option | Mode | Description | Constraints |
|---|---|---|---|
input.md manifest.md |
Single file | Markdown to convert and the pseudonymization dictionary | Cannot be combined with -c |
-o, --output |
Single file | Output file path | Default: <input>.pseudonymized.md. Cannot be combined with -c |
-c, --config |
Batch | Convert many files under a folder using config.yaml |
Cannot be combined with the input, the manifest or -o |
--log-dir |
Both | Folder for the run log (relative to the current folder) | Batch: defaults to log_dir in the config. Single file: a log is written only when given |
--overwrite |
Both | Overwrite existing result files | Inputs, the manifest and the config are always refused |
-f, --force |
Both | Includes --overwrite, and also overwrites or deletes results edited by hand |
Matters most with --sync and --prune |
--sync |
Batch | Re-convert only the files that changed | Needs -c and a fixed output_dir |
--prune |
Batch | Delete the results of inputs that no longer exist | Needs --sync. Results edited by hand are deleted only with -f |
--restore |
Batch | Restore new files from the pseudonymized folder into input_dir |
Needs -c. Cannot be combined with --sync, --prune, --overwrite or -f |
--from |
Batch | Pseudonymized folder to restore from (relative to the current folder) | Needs --restore. Defaults to output_dir (required when it is empty) |
--version |
Both | Print the version from git tags |
Example: config.yaml
| Key | Description | etc |
|---|---|---|
manifest |
Pseudonymization dictionary | |
input_dir |
Input folder | |
output_dir |
Output folder | If omitted: ./output/<run time> |
log_dir |
Folder for run logs | If omitted: ./logs |
manifest_log_dir |
Folder for the manifest log. It stores names before pseudonymization, so be careful. | If omitted: log_dir |
include |
Patterns of files to process | |
except |
Patterns of files to exclude |
- Paths can also be written in Windows form (
G:\folder\file,.\folder).- On non-Windows environments (WSL, etc.),
G:\...is read as/mnt/g/...and\as/.
- On non-Windows environments (WSL, etc.),
--log-dirand--fromon the command line are relative to the current folder.- Values in the config are relative to the config folder.
- Only files that match
include(default**/*.md) and do not matchexceptare processed.excepttakes priority, and a matching folder excludes everything below it.- glob:
*,?,**,[a-z],[!x],{a,b}.*and?do not cross/. - A pattern ending with
/, such asdraft/, means everything below that folder. Windows-style patterns with\, such asdocs\**\*.md, are read with/.
log_dirandmanifest_log_dirmust not be the same asinput_diroroutput_dir(for restore, the--fromfolder).- Logs and
state/contain text from before pseudonymization. They are rejected with an error so that they are not converted again as input or mixed into the result folder.
- Logs and
- Results are saved in
output_dirwith the same subfolder structure and file names. Subfolders are created automatically. - If
output_diris empty or omitted, the results are saved inoutput/<run time>/(for exampleoutput/20261102-090410/).- The folder name is the same as the log file name.
- YAML supports only the following.
key: value,- itemlists, comments and quoted strings- Wrap values that start with
*or{in quotes.
- The manifest, the config, the output and log folders, and symbolic links are excluded from the input.
- If some files fail, the rest are still processed and the exit code is 1.
Example: logs/example.log
- Every batch run creates
logs/YYYYMMDD-HHMMSS.log(plain text, permission0600).- A single-file run creates one only when
--log-diris given.
- A single-file run creates one only when
- If a file name contains a real name, it is also recorded in the log.
- The log warns the agent not to share or print it without a user request.
- Original text, pseudonyms and Keys are not recorded in the run log.
- It lists only the number of removed lines and replacements per file, and the match count per manifest row number.
python scripts/pseudonymize.py -c config.yaml --sync
- After editing the sources a little and running again, only the changed files are re-converted.
- Even with many files, unchanged files are not read; only
statis checked. - Use it only when
output_diris fixed inconfig.yaml.- If
output_diris empty (including a value with only a comment, such asoutput_dir: #default), the run ends with an error before any conversion. - No log or state file is created in that case, so nothing appears in
logs/. - Set
output_dirto an existing folder, or run without--sync.
- If
- Even with many files, unchanged files are not read; only
- The sync state is saved in
<log_dir>/state/<hash>/<hash>.sync.json.- Default permission
0600 - Records per-file size, modification time and content hash, and the manifest hash
- Created separately for each input/output folder pair
- Default permission
- How each input file is judged
- Same size and modification time:
unchanged - Only the modification time changed and the content is the same:
unchanged - Content changed:
ok (updated) - Result file is gone:
ok (restored)
- Same size and modification time:
- If the manifest file changed, all files are converted again.
- Changing only the
notecolumn does not count.
- Changing only the
- With the
-foption, sync is forced even when a pseudonymized file was edited and no longer matches.- Without this option, a
failed: output was edited ...error occurs.
- Without this option, a
- With the
--overwriteoption, both edited and unedited files are overwritten. - Results of files that no longer exist in the input are not deleted by default. They are listed as
keptin[orphan]of the log. - With the
--pruneoption, only results that are exactly as made by the last sync are deleted, and emptied subfolders are cleaned up.- Results edited by hand are deleted only with
-f. - So run
--syncalone first, check the[orphan]list, and then use it.
- Results edited by hand are deleted only with
- The sync state file contains file paths and content hashes, so it is as private as the logs.
- If you delete it, the next run becomes a first sync.
python scripts/pseudonymize.py -c config.yaml --restore [--from <pseudonymized folder>]
- The agent reads only pseudonymized files, writes a report and saves it in the pseudonymized folder (
output_dir). The new Markdown file is reverse-mapped using the manifest and saved at the same relative path in the original folder (input_dir).- The agent does not need to see the original text.
- If
output_diris empty (a new folder every run), specify the folder with--from output/<time>. - It cannot be combined with
--sync,--prune,--overwriteor-f. - How each file is judged
- If it is in the restore data and the pseudonymized file is unchanged:
unchanged.- For
--syncfiles that are not in the restore data, the same judgment uses the result hash in the state file.
- For
- If it is in the restore data but the pseudonymized file was edited:
skipped (edited existing file ...).- Restoring edits to existing files is not supported yet (the original is left untouched).
- A new file that is not in the restore data is restored.
- If a file already exists at the original path, it is not overwritten and the result is
failed.
- If a file already exists at the original path, it is not overwritten and the result is
- If it is in the restore data and the pseudonymized file is unchanged:
- Restore turns the Value of the manifest table back into the Key.
- The aliases of a row are restored to the first Key.
- When pseudonyms overlap, the longer one takes priority.
- Text removed with
remove_textorremove_lineis not in the new file, so it is not restored.
- Round-trip check: the restored text is pseudonymized again.
- If it equals the pseudonymized file, it is registered in the restore data and the sync state (
unchangedon the next--sync). - If it differs, it is marked
ok (roundtrip differs)with a[warning].
- If it equals the pseudonymized file, it is registered in the restore data and the sync state (
- The log is
<log_dir>/YYYYMMDD-HHMMSS.restore.log- Example (fake data): logs/example.restore.log
- Permission
0600 - Original text and pseudonyms are not recorded.
- No manifest log is created.
logs/
20261102-090410.log
20261102-090410.manifest.log
state/ (permission 0700)
<hash>/ (permission 0700, one per input/output folder pair)
<hash>.sync.json (permission 0600)
<hash>.restore.json (permission 0600)
<hash>is a 10-character hash made from the input and output folder paths.- Folder paths and names are not used.
- While a run is in progress,
<hash>.lockbriefly appears in the same folder. - They contain paths and parts of the original text, so the agent and other people must not read or share them.
- Used by
--sync. - Records the manifest hash and, per file, the size, modification time, input hash and result hash.
- On every (batch) pseudonymization, per-file records are saved in this single file.
- Contents
- Top level: manifest hash (
rules_sha), script hash (script) - Per file: input and result hash, number of original lines, the number and full text of removed lines, replacement positions (line, column, original, pseudonym)
- It is a partial copy of the original text, so it is as private as the manifest log.
- Top level: manifest hash (
- With
--sync, only the records of re-converted files are updated. - When
--prunedeletes a result, the record of that file is deleted too. - Synced files without a record are filled in automatically when the hashes match.
- If the manifest changed, the old records are discarded.
- If it cannot be read, a conversion rebuilds it and notes it in the log
[warning], while--restorestops with an error. - Pseudonym collision warnings
- If a Value in the manifest is part of another Value, or a result contains pseudonym text that no rule produced, it is reported to stderr and in the log
[warning]. - It means that restoring may become ambiguous.
- If a Value in the manifest is part of another Value, or a result contains pseudonym text that no rule produced, it is reported to stderr and in the log
- The two logs (run log, manifest log) are opened before conversion starts and appended to after each file.
- Even if the run is interrupted, what was recorded up to that point remains.
- The
result:line in[result]at the top of the run log is the run status.- While running, or after a forced kill:
running - Finished normally:
N ok, M failed - Ended by an exception or Ctrl+C:
aborted (N ok, M failed so far) - The field has a fixed width (padded with spaces) and is overwritten when the run ends.
- While running, or after a forced kill:
- Layout: warning, header (
files:= number of files to process),[result],[file](one line per file),[manifest](match count per row number)[manifest]is written at the end.- The manifest log keeps a table format, so judge completion from the run log.
- If a log cannot be created (for example a rerun in the same second), nothing is converted and the run ends with an error.
Example (fake data): logs/example.manifest.log
- Every run also creates
YYYYMMDD-HHMMSS.manifest.logwith the same time as the run log.- The location is
manifest_log_dirin the config. - If it is empty or omitted, it is saved in
log_dirtogether.
- The location is
- It records the original text (Key) and pseudonym (Value) as they are for each converted file.
- It is as private as the manifest.
- Like the run log, its permission is
0600(owner only); this does not apply on Windows. .gitignoreexcludes*.manifest.login any folder.
- As the warning at the top says, the agent and other people must not read or share it.
- The format is a single table:
output row action count key value(tab-separated)outputis the absolute path of the converted result file.- There is one line per matched (result file, rule); files with no match have no line.
rowis the line number in the manifest file and equals the row number in the[manifest]section of the run log.
For the detailed table, see 사고 대응 (incident handling) in SKILL.md.
- Basic principle: do not overwrite or delete, and when in doubt, stop or record a failure.
- Mixed-up paths
- Overlapping
input_dir/output_dir/log_dirare rejected. - A
--fromwithout pseudonymization records is refused for restore.
- Overlapping
- Changes during a run
- A run keeps using the config and manifest it read at the start.
--restoreafter the manifest changed is refused; pseudonymize again with--sync.- Check which code ran by the
script:hash in the log.
- Deletion accidents
- If results or logs disappear during a run, this is detected at the end and reported as a failure (exit code 1).
- If the log folder was deleted, rebuild the records with
--sync --overwrite.
- Concurrent runs
- Only one run per folder pair is allowed, using
state/<hash>/<hash>.lock. - The lock of a dead process is taken over automatically.
- Only one run per folder pair is allowed, using
- Interrupted writes
- Result files are written completely to a temporary file and then renamed, so no truncated file remains.
python -m unittest discover -s tests -v
- Fake data is used in temporary folders, so real documents and manifests are never touched.
- Covers glob, the YAML parser, path conversion, log formats,
--overwrite/-f/--sync/--prune, restore and incident handling.
python scripts/pseudonymize.py --version
- manifest.md table:
Key | Value | note | action - If the
actioncolumn or cell is missing, it is replace.noteis used only as a description.
actiontypes- replace: replace Key with Value (Value required)
- remove_text: delete only the Key string (Value left empty)
- remove_line: delete the whole line containing the Key (Value left empty)
- A row starts with
|.KeyandValueare the first two columns;note/actionare optional columns.
- The Key cell can hold several aliases separated by commas or
<br>, and all are handled as the same Value.- Write a literal comma as
,. - Empty aliases caused by repeated or trailing separators are ignored.
- Spaces inside an alias are kept.
- Write a literal comma as
- Rows with all cells empty are skipped.
- If a
Keyis duplicated, only a warning is shown and the rule of the last row read is used. - If different rows use the same
Value, the run is aborted.- Gather the originals that should share a pseudonym as aliases in one row.
- A
Keywith no alias, an invalidactionor a wrong number of columns is an error. - Error and warning messages show only row numbers.
- Line removal is done first, then the remaining text is replaced or deleted in one pass.
- At the same position, the longer string wins, and replaced text is not replaced again.
- Matching is exact and case-sensitive, with no word-boundary or particle adjustment.
- UTF-8 input, UTF-8 (no BOM) output.
- LF/CRLF is preserved.
- Originals, the manifest and existing outputs are not overwritten by default.
--overwrite: overwrites existing result files only.- Originals, the manifest and the config are always refused.
- With
--sync, existing results unknown to the state file are overwritten too.
-f(--force): includes--overwriteand also goes past the safeguards.- With
--sync, it overwrites results edited by hand. - With
--prune, it also deletes results edited by hand.
- With
- Files are written to a temporary file and then replaced, so an existing result is not broken by a mid-way failure.
- Files behind hard links or symbolic links are not changed.
- Logs are never overwritten, whatever the options.
--prunedeletes only regular files insideoutput_dir.- Paths outside it, symbolic links and folders are not deleted even with
-f.
- Paths outside it, symbolic links and folders are not deleted even with
- The parent folder of the output must exist.
- If
-ois omitted,<source name>.pseudonymized.mdis created. - On Windows you can also use
py, and on WSLpython3. - Personal information that is not in the manifest, and image or external link targets, are not detected automatically.
- It applies to the whole text, including tables, code and links, so check the document structure after conversion.
- Keep the manifest and any result you share separately.
- Old JSON rules are still read for compatibility, but cannot be mixed with a table in one file.