Encoding and Line Ending Normalization Plan

This proposal is a draft, was generated by AI, and may not have been reviewed for accuracy.

Purpose

This document proposes a safe, staged plan for normalizing text encoding and line endings in the repository.

The main problems being addressed are:

• mojibake in some docs/templates
• mixed line endings
• patch/edit instability caused by byte-level mismatches
• Windows terminal/editor codepage issues

Goals

• Standardize source-like text files on `UTF-8` without BOM
• Reduce future mojibake risk
• Make diffs and patching more reliable
• Avoid unnecessary churn in vendor or generated files

Non-Goals

• Reformatting unrelated code
• Bulk-editing vendor bundles by hand
• Changing runtime message encodings used for FTN payloads

Current Symptoms

Observed examples include:

• `â?"` instead of an em dash (?)
• `â?"` instead of an en dash (?)
• `â??` / `â?` instead of curly quotes (" ")
• `â??` instead of a right single quote (')

These are classic mojibake symptoms. The UTF-8 byte sequences for these characters (e.g. E2 80 94 for em dash) were decoded as CP1252, producing three separate Latin characters, which were then saved back literally into the file. The file now contains those wrong bytes ? it is not an encoding display issue, it is corrupted content.

Likely Causes

1. AI tooling writing through a CP1252 shell on Windows

The most likely source of new mojibake in this repository is an AI coding agent (e.g. Codex) running in a Windows terminal with a CP1252 or Windows-1252 codepage. When the agent generates text containing em dashes or curly quotes and writes it to disk through a codepage-unaware pipe, the UTF-8 bytes land corrupted.

Mitigation for this specific cause: ensure the shell used by the agent has chcp 65001 active, or that its file write path explicitly specifies UTF-8. Until that is addressed, files written by the agent should be visually scanned for â? sequences before committing.

2. UTF-8 opened as CP1252 or Latin-1 by an editor

A less common cause in this repo, but still possible. If an editor opens a UTF-8 file under the wrong encoding and saves it, the same damage occurs.

3. Windows shell/editor encoding mismatches

On Windows, PowerShell, console host, git, editors, and browser/devtools can all display or emit text differently if encoding assumptions do not match.

4. Mixed line endings

CRLF/LF mismatches do not directly cause mojibake, but they increase patch fragility and make file churn harder to reason about.

5. Manual copy/paste through lossy paths

Text copied between terminals, web tools, AI tools, or editors can arrive with already-damaged punctuation.

Why This Causes Trouble for Codex

Codex in this environment is especially sensitive to:

• exact-context patching
• shell output rendering
• byte-level line matching

When a file visually shows one thing but actually contains different bytes, patch tools can fail even when the intended edit is correct.

This tends to show up as:

• patch context not found
• repeated failed edits on apparently identical lines
• accidental duplication when trying to work around bad context

This is not primarily a reasoning problem. It is a file-bytes and tooling-consistency problem.

Normalization Policy

Recommended text policy

For source-like files:

• encoding: `UTF-8` without BOM
• line endings: normalized by git

Applies to:

• `*.php`
• `*.twig`
• `*.md`
• `*.txt`
• `*.js`
• `*.css`
• `*.json`
• `*.sql`
• `*.yml`
• `*.yaml`

Exceptions

Do not bulk-normalize by hand:

• `vendor/`
• bundled/minified third-party assets
• binary payloads
• files intentionally using legacy encodings for protocol/test fixtures

Recommended .gitattributes

Add a repository .gitattributes file like:

* text=auto

*.php  text eol=lf
*.twig text eol=lf
*.md   text eol=lf
*.txt  text eol=lf
*.js   text eol=lf
*.css  text eol=lf
*.json text eol=lf
*.sql  text eol=lf
*.yml  text eol=lf
*.yaml text eol=lf

vendor/ -text
public_html/vendor/ -text
*.png  binary
*.jpg  binary
*.jpeg binary
*.gif  binary
*.ico  binary
*.woff binary
*.woff2 binary
*.ttf  binary
*.zip  binary
*.qwk  binary
*.rep  binary
*.pkt  binary

Warning: Adding .gitattributes with text=auto or explicit eol=lf rules will trigger a mass line-ending renormalization the first time tracked files are touched or checked out. This will produce a large diff of CRLF?LF changes across many files. This must be committed in isolation ? nothing else staged ? and the diff reviewed carefully before pushing. Do not mix this commit with feature work.

Proposed Cleanup Phases

Phase 1: Define policy only

1. Add `.gitattributes`
2. Agree on `UTF-8 without BOM`
3. Document exceptions

Important: commit .gitattributes alone. Expect a large line-ending normalization diff on the next checkout or git add. Review it before pushing.

This phase should happen before bulk content cleanup.

Phase 2: Fix known human-maintained files

The currently known affected files are:

• `templates/qwk.twig`
• `docs/proposals/Translations.md`

The fix is a direct search-and-replace ? not a re-encoding operation. The mojibake bytes are already literally present in the file. There is no need to "reopen with explicit encoding controls." Simply replace the corrupted sequences with the correct Unicode characters:

| Wrong bytes (literal) | Correct character | |---|---| | â?" | ? (em dash, U+2014) | | â?" | ? (en dash, U+2013) | | â?? | " (left double quote, U+201C) | | â? | " (right double quote, U+201D) | | â?? | ' (right single quote, U+2019) |

This can be done with a single find-and-replace pass in any editor, or via sed. Review the diff to confirm only punctuation changed.

Phase 3: Audit all source-like files

Search for common mojibake markers:

• `â`
• `Ã`
• `Â`

Then classify each result:

• real project file to fix
• acceptable legacy fixture
• vendor/minified file to leave alone

Phase 4: Normalize line endings

After .gitattributes is in place:

1. refresh checkout or renormalize tracked files
2. review the resulting diff carefully
3. avoid mixing this with feature work

This should be done in a dedicated PR/commit.

Safe Working Rules During Cleanup

• Do not combine normalization with feature changes
• Do not touch vendor/minified files unless replacing from upstream
• Fix a small batch of files at a time
• Review every diff for accidental content corruption
• Prefer ASCII punctuation in code/comments/templates where stylistically acceptable
• After any AI-generated file is written, scan it for `â?` before committing

Suggested Validation

After each cleanup batch:

1. run existing lint/check scripts
2. inspect diffs for punctuation-only changes
3. verify templates/docs render normally
4. verify no binary files were accidentally treated as text

Tooling Recommendations

Editors

Use an editor that shows and controls:

• current file encoding
• line endings
• BOM status

Examples:

• VS Code
• Notepad++
• PhpStorm

Git

Recommended user-level settings to review on Windows:

git config --global core.autocrlf false
git config --global core.eol lf
git config --global i18n.commitEncoding utf-8
git config --global i18n.logOutputEncoding utf-8

Note: These are global settings that affect all git repositories on the machine, not just this project. Review them against your normal workflow before applying. The .gitattributes file takes precedence over core.autocrlf for files it explicitly covers, but the global settings still affect unspecified file types.

PowerShell / terminal

If terminal output itself is garbling UTF-8, review:

• terminal font
• code page (`chcp 65001` for UTF-8)
• PowerShell output encoding
• editor integrated terminal behavior

Risks

1. Large noisy diffs from .gitattributes

Adding .gitattributes will trigger a mass CRLF?LF renormalization across many files on the first checkout or git add. This is expected behavior, not corruption, but it makes the diff very large and hard to review if mixed with other changes.

Mitigation:

• commit `.gitattributes` alone
• do it in a dedicated PR/commit with nothing else staged

2. Accidental content corruption

If a file is written under the wrong encoding again (e.g. by the AI agent), damage can recur or worsen.

Mitigation:

• scan AI-generated files for `â?` sequences before committing
• address the root cause in the agent's shell environment (see Likely Causes above)

3. Vendor churn

Changing vendored/minified files by normalization alone is usually not worth it.

Mitigation:

• treat vendor paths as exceptions

Recommended First Steps

1. Add `.gitattributes` in a standalone commit; expect and review the line-ending normalization diff
2. Fix the two known-bad files (`templates/qwk.twig`, `docs/proposals/Translations.md`) using search-and-replace in a separate commit
3. Leave vendor bundles alone unless replaced from upstream
4. Address the agent shell codepage issue to prevent recurrence

Recommendation

The best approach is a small, explicit normalization pass, not a repo-wide mass rewrite.

The actual scope of content damage is small ? two human-maintained files. The fixes are straightforward search-and-replace operations. The .gitattributes addition is the more consequential change and needs its own commit.

The higher-priority fix is the root cause: the AI agent's shell environment writing files through a CP1252 codepage. Without that, mojibake will continue to be introduced faster than it is cleaned up.