[Bug]+io: settings files written under system default encoding can fail on non-UTF-8 codepages

[reddraconi]

Checklist

• I am using an up-to-date version.
• I have read the documentation.
• I have searched existing issues.

TagStudio Version

9.5.6

Operating System & Version

Fedora 43 x86_64

Description

There's a few places where we don't specify UTF-8 as the format for files that TagStudio owns. For Linux and Mac it's probably not a problem since they should default to UTF-8 but on Windows systems that can have non-UTF-8 codepages, this will cause problems.

Examples:

• A user on Japanese Windows (cp932) saves a library path containing 写真 to settings.toml. The file's written in cp932. If that user opens TagStudio on system with a different Windows codepage, or on Linux, read_settings() blows up with UnicodeDecodeError.
• TOML and JSON specs both mandate UTF-8, so anything we write that isn't UTF-8 is nonconforming and won't
  round-trip through other tools. (Probably pretty minor, in the grand scheme of things)
• The compiled ignore file (.compiled_ignore) passed to ripgrep is opened with default encoding, so a user with non-ASCII glob patterns can poison the ripgrep input.
• ResourceManager.read_text() reads bundled resource files (English translations, etc.) under the system encoding.

PR inbound.

Expected Behavior

TagStudio files (configurations like settings.toml, .ts_ignore, etc.) should use a standard codepage to ensure files moved between systems are readable by those systems if one or the other(s) are using a non-UTF-8 codepage.

If a TagStudio-owned file is written with non-UTF-8 formatting, it should be rewritten in UTF-8 on next save.

Steps to Reproduce

  # Build a settings.toml whose ASCII structure is intact but whose string *value*
  # contains raw cp932 bytes (invalid as UTF-8).
  import pathlib
  legacy = "写真".encode("cp932")  # b"\x8e\xca\x90^"
  payload = b'language = "ja"\ndate_format = "' + legacy + b'"\n'
  pathlib.Path("/tmp/settings.toml").write_bytes(payload)

Use that settings file in TagStudio on a non-UTF-8 system with a non-cp932 codepage and watch it fall over.

Logs

No response

[TrigamDev]

You say your operating system is a Linux distro, but this is a bug that affects Windows.
Have you actually been able to reproduce this bug, in an actual Windows environment, with an unmodified copy of TagStudio without making a purposefully poisoned settings file?
Can TagStudio create a broken settings file, cause if no, this isn't an issue, most programs are going to have issues when you purposefully replace program files with broken ones they aren't meant to read.

[reddraconi]

I don't think things are "broken", it's a common file encoding issue. In fact, Python's planning to make UTF-8 the default encoding in Python 3.15 to fix these specific kinds of issues with PEP 686.

Like I mentioned earlier, if we don't specify an encoding when open is called and we're on a non-UTF-8 system, files written in other codepages cause issues. TagStudio would end up writing similar content (mixed ASCII and system-encoded strings) like in the minimal settings.toml I scripted.

For the settings file specifically, if I rsync my TagStudio install and configs to another system with a different codepage, the second system would have a bad time (simulated below).

This also occurs with .ts_ignore and .compiled_ignore files, but specifically in libraries. If two users with different codepages try to open a shared library (like on a network share) they'd end up with a different failure.

I used the script in the body of the ticket to make the non-UTF-8-encoded file and ran the following bash to ensure it fell over by forcing Python to fall back to ASCII during open() calls instead of the default UTF-8. This is the same behavior that would happen in the rsync scenario above.

PYTHONUTF8=0 LC_ALL=POSIX /home/reddraconi/git/redd_ts/.venv/bin/tagstudio --settings-file /tmp/settings.toml

My local folder, redd_ts, is at the tip of TagStudioDev/TagStudio (c4c7496).

This is the stacktrace from that command.

2026-04-25 11:44:59 [info     ] [FFmpeg] Using FFprobe location: ffprobe (Found)
2026-04-25 11:44:59 [info     ] [FFmpeg] Using FFmpeg location: ffmpeg (Found)
2026-04-25 11:45:00 [info     ] [ResourceManager] Resources Registered: count=37
Traceback (most recent call last):
  File "/home/reddraconi/git/redd_ts/.venv/bin/tagstudio", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/reddraconi/git/redd_ts/src/tagstudio/main.py", line 67, in main
    driver = QtDriver(args)
             ^^^^^^^^^^^^^^
  File "/home/reddraconi/git/redd_ts/src/tagstudio/qt/ts_qt.py", line 231, in __init__
    self.settings = GlobalSettings.read_settings(self.global_settings_path)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/reddraconi/git/redd_ts/src/tagstudio/qt/global_settings.py", line 86, in read_settings
    filecontents = file.read()
                   ^^^^^^^^^^^
  File "/home/reddraconi/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8e in position 31: ordinal not in range(128)

Sorry for the wall of text. I wanted to be thorough without sounding like a jackass.
I'm working on fixing my Windows VM to do actual testing instead of simulation.

[reddraconi]

Understood on the current settings.toml creation in the codebase. If I jack it up myself because I modified it outside of the app or do weird things like share installs between systems that don't share codepages, that's probably my own dumb fault.

The failure mode I found was not nearly as dire as the raw python sim made me think it was going to be in the current code base. So, that's nice.

I wasn't able to get the UI to drop bad content into the settings.toml with the current settings UI code. If user-editable settings get added later, this might still be a problem if users do weird things between systems.

I think I may have found a staleness issue when updating the ignore list (see steps 20-22) though. Not sure at this point.

Note: The subfolder I added to the library on the Japanese Windows VM is named "しお". That's is "shio" in romanji and what I wrote in the steps below because I got tired of swapping my inputs back and forth, in case that's confusing.

Background:

I stood up a Windows 11 25H2 VM this afternoon/evening and swapped it into Japanese. I installed git, ripgrep, VSCode, and Python 3.12.10 using winget; downloaded the TagStudio Windows release from GitHub; and cloned the latest TagStudio git repo to this fresh VM.

I took screenshots as I tested. They're in the attached zip in case you want/need them. I don't have live recording set up on my Fedora box and I didn't think anyone would want to watch hours of flailing anyway.

BLUF:

I found a codepage mismatch when .ts_ignore is used by refresh.py to create the .compiled_ignore file that's handed and handed to ripgrep on a system that doesn't use a UTF-8 codepage.

The temp file gets created using the system default encoding and then sent to ripgrep as UTF-8, making non-UTF-8 matches fail.

So, when I added the library subfolder to the ignore list and refreshed my library, my gifs were still present.

Steps To Reproduce on Windows 11 25H2 in English, converted to Japanese:

0. Made a new W11 25H2 VM in English.
1. Downloaded the 9.5.6 Alpha release from github and extracted to Desktop\TagStudio (image 01)
2. Ran intl.cpl and swapped from English to Japanese (images 02 through 04)
3. Installed the Japanese language support packages and ran chcp to show the current codepage (image 05)
4. Launched TagStudio from the github releases and pulled up the About window. I went to Settings and swapped the Language to Japanese, in case that mattered. (images 06 and 07)
5. Made a new folder in Pictures named "shio" using the Japanese katakana keyboard (image 08). I stole two gifs from the Windows installer ISO and dropped them in for some content.
6. Opened "Pictures" as a library folder. (image 08)
7. The images came in as unlinked for some reason. (image 09)
8. I ran a refresh, then search and relink. They showed up as expected (image 10)
9. Added the path "/shio" to the ignore list via the UI (image 11)
10. I ran another refresh, then searched and relinked. Got the same results (image 12). I expected the gifs to disappear...
11. I made sure ripgrep was installed (image 13)
12. I closed TagStudio to make sure I didn't do something silly. Re-opened it and now I have the original unlinked results plus the linked results, and if I understand the repgrep work correctly none of them should be present. (image 14)
13. Moved to the source in VScode so I could run it from PowerShell with Python to easily see the logs. Ran it with python -m tagstudio.main (image 15 and 16, plus log file)
14. rg threw errors that it encountered non-UTF-8 characters in the .compiled_ignore file so it never ignored the Japanese characters. That's the second- and third-to-last logs lines in the log file.
15. I added 'encoding=True' to the temp file creation and re-ran the new compiled version. (images 17 and 18)
16. rg no longer complains about the malformed characters, but the images still appeared even after I refreshed and relinked (images 19 and 20)
17. I added a copy job to grab the compiled output (images 21) and checked it in the hex editor (image 22), it looks the Japanese is UTF-8 encoded now instead of cp932.
18. I manually ran ripgrep using the ignore file and it doesn't return the image paths. (image 23) but they still appear in the library, but not as broken links any more. (image 24)
19. I tossed the .TagStudio folder in Pictures and re-ran steps 6-10 again to make sure everything was clean.
20. I tried to run the Fix Ignored Entries command, but clicking Refresh didn't do anything. The logs did report it was going to do that, though. (image 25)
21. I updated the ignore list to drop the leading slash (image 26) and re-ran the Fix Ignored Entries command.
22. I got the same results again. The gifs were still present in the library. So I closed TagStudio and tried again. This time I was able to scan and remove the entries that should have been skipped.

snapshots.zip
tagstudio_log_20260424.txt

Let me know if you'd like further testing or if I missed something.