UTF-8 vs Shift-JIS CSV
UTF-8 is the interoperable Unicode choice for new CSV files; Shift_JIS remains a compatibility requirement for some Japanese Windows and business systems.
Open CSV Encoding FixerUse the encoding explicitly required by the receiving system. Prefer UTF-8 for new integrations and mixed-language data. When a partner requires Shift_JIS, test a representative round trip because its repertoire is narrower and labels such as Shift_JIS, Windows-31J, and CP932 can hide implementation differences.
The practical difference
UTF-8 represents Unicode and can carry Japanese, Latin scripts, emoji, and most modern text in one file. It is ASCII-compatible and is recommended by the W3C's CSV guidance. Shift_JIS is a legacy multibyte Japanese encoding designed around a smaller character repertoire and long-standing Japanese software conventions.
The file extension does not identify either encoding. A .csv file is just bytes plus a CSV dialect. Ask the receiving system for an exact encoding name and an example file; “Japanese CSV” is not specific enough.
Why Shift_JIS labels can be confusing
On the web platform, the WHATWG Encoding Standard maps labels including shift_jis, sjis, ms932, and windows-31j to its Shift_JIS decoder. Other runtimes may distinguish standards-based Shift_JIS from Microsoft CP932 extensions. That difference can affect vendor characters and punctuation.
A detector can say that a byte sequence is plausible Shift_JIS, but it cannot prove the export's business meaning. Verify names that use uncommon kanji, full-width punctuation, half-width katakana, the yen/backslash position, and the wave-dash or minus characters relevant to your data.
Plan for round-trip loss
Converting from Shift_JIS to UTF-8 is normally safe once the source bytes are decoded correctly. Converting from UTF-8 to Shift_JIS can fail or substitute characters when the input contains code points outside the target repertoire, such as emoji or scripts not represented by the chosen mapping.
Run a round-trip test on a copy: source → target encoding → Unicode, then compare the resulting text and row structure. Do not rely on a visually similar glyph; different code points can look alike but fail exact matching in downstream systems.
Choose an exchange policy
For a new pipeline, document UTF-8, the delimiter, line endings, header presence, and whether a BOM is expected. For a legacy receiver, isolate the Shift_JIS conversion at the boundary and retain a UTF-8 master so unsupported characters can be reported rather than silently lost.
Tools For CSV converts supported legacy source encodings to UTF-8. It does not transliterate Japanese, normalize half-width and full-width forms, or promise that a later Shift_JIS export can represent every Unicode character.
Broken and corrected examples
A UTF-8 file decoded as Shift_JIS
譁�蟄励′譁�縺代＠縺�文字が文字化けしているGarbled Japanese does not identify one encoding by itself; return to the original bytes and test the documented source encoding.
A character outside a legacy repertoire
顧客メモ: launch 🚀Keep the master file in UTF-8; validate or reject at the Shift_JIS boundarySilent substitution would destroy information and may make two distinct values compare as equal.
How to fix it safely
- Read the receiver's specification and confirm whether it means UTF-8, Shift_JIS, or a Windows-31J/CP932-compatible implementation.
- Preview the untouched source with representative Japanese and non-Japanese values.
- Convert to UTF-8 for the maintained copy and verify row counts, quotes, and delimiters.
- If Shift_JIS output is mandatory, test unsupported characters and a round trip before production exchange.
Common questions and edge cases
Is Shift_JIS the same as CP932?
They are closely related in common software, but implementations and extension repertoires can differ. The web Encoding Standard aliases several labels to one decoder; another platform may use a different mapping.
Should a UTF-8 CSV include a BOM?
UTF-8 does not require one. A BOM can be a useful compatibility hint for software that otherwise guesses, so follow the receiving application's documented behavior.
Will conversion normalize full-width and half-width text?
No. Encoding conversion maps representable characters; width normalization is a separate, meaning-changing text operation.