Skip to content

Python cannot run in the ja_JP.sjis locale used windows-31j encoding. #102388

@moriyama

Description

@moriyama

Bug report

Linux using glibc cannot run Python when ja_JP.sjis locale is set as follows.

$ sudo dnf install glibc-locale-source # RHEL or RHEL compatible Linux distribution
$ sudo localedef -f WINDOWS-31J -i ja_JP ja_JP.sjis
$ export LANG=ja_JP.SJIS
$ python3
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = 'lib64'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib64/python39.zip',
    '/usr/lib64/python3.9',
    '/usr/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: WINDOWS-31J

Current thread 0x00007f49d76ee740 (most recent call first):
<no Python frame>

The charset name "Windows-31J" is registered in the IANA Charset Registry[1].
Windows-31J is supported by perl[2], php[3], ruby[4], java[5], etc.
Python's cp932 is equivalent to Windows-31J, so I propose to add windows_31j to aliases for cp932.

[1] https://www.iana.org/assignments/charset-reg/windows-31J
[2] https://perldoc.perl.org/Encode::JP
[3] https://www.php.net/manual/en/mbstring.encodings.php
[4] https://docs.ruby-lang.org/ja/latest/class/Encoding.html
[5] https://docs.oracle.com/en/java/javase/19/intl/supported-encodings.html

Your environment

  • CPython versions tested on: 3.9.13, 3.12.0a5+
  • Operating system and architecture: MIRACLE LINUX 8.6 x86_64 (RHEL 8.6 compatible)

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No fields configured for issues without a type.

    Projects

    Status
    Done
    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions