locale.getlocale() problems on OSX

I need to get the system locale to do a number of things, ultimately I want to translate my app using gettext. I am going to distribute it on both Linux and OSX, but I ran into problems on OSX Snow Leopard:

$ python
Python 2.5.2 (r252:60911, Jan  4 2009, 17:40:26) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, '')
'sv_SE.UTF-8'
>>> locale.getlocale()
('sv_SE', 'UTF8')

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> i开发者_JS百科mport locale
>>> locale.setlocale(locale.LC_ALL, '')
'C'
>>> locale.getlocale()
(None, None)

Both systems are using Swedish languages. On Linux, the environment variable LANG is already set to "sv_SE.UTF-8". If I pass that variable to python on OSX (LANG="sv_SE.UTF-8" python instead), locale is detected nicely. But shouldn't locale.getlocale()be able to fetch whatever language the operating system has? I don't want to force users to set LANG, LC_ALL or any environment variable at all.

Output of locale command:

$ locale
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

---

Odd on OSX (Smow Leopard 10.6.1) I get

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.  
>>> import locale
>>> locale.getlocale()
(None, None)
>>> locale.setlocale(locale.LC_ALL, '')
'en_GB.UTF-8'
>>> locale.getlocale()
('en_GB', 'UTF8')

Edit:

I just found this on the apple python mailing list

Basically it depends on what is set in your environment at run time (one of LANG, LANGUAGE, LC_ALL) I had LANG=en_GB.UTF-8 in my shell environment

---

Looks like you can change locale by changing environment variable LC_ALL.

$ export LC_ALL=C
$ python
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getlocale()
(None, None)
>>> locale.setlocale(locale.LC_ALL, "")
'C'
>>> locale.getlocale()
(None, None)    

$ export LC_ALL=en_GB.UTF-8
$ python
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getlocale()
(None, None)
>>> locale.setlocale(locale.LC_ALL, "")
'en_GB.UTF-8'
>>> locale.getlocale()
('en_GB', 'UTF8')

---

Addmittedly a horrible hack, but I inserted this:

import platform

# ...

# XXX horrendous OS X invalid locale hack
if platform.system() == 'Darwin':
    import locale
    if locale.getlocale()[0] is None:
        locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

at an early point in a program of mine. After that I could run my program using unmodified shell environment on all OS'es relevant to me (my program figures out the language to be used later in it's processing anyway).

---

From here: Try adding or editing the ~/.profile or ~/.bash_profile file for it to correctly export your locale settings upon initiating a new session.

export LC_ALL=en_US.UTF-8  
export LANG=en_US.UTF-8

---

Old question, but this may help others: this is a Python bug that as of March 2016 is still unresolved in either Python 2 or 3: https://bugs.python.org/issue18378 .

The summary is that Python assumes GNU-like locales and balks on (POSIXly correct) divergences like those in BSD environments (as is OS X). And the UTF8 locale exists in BSD, not in Linux, hence the problem.

As for solutions or debugging: the local environment variables can be set by Terminal.app (see Preferences - Profiles - Advanced - International; similarly so for iTerm or whatever). So one can find the locale environment variables set when in a terminal window, but find the variables NOT set when running a packaged application.

For some cases (like Sphinx in python 2.7 and 3.5 dying in OS X because of "ValueError: unknown locale: UTF-8"), disabling the preference checkbox to set locale environment variables is the solution.

But that can cause problems in other programs: if the locale vars are not set, bash 4.3 (from MacPorts) will complain at every prompt with "warning: setlocale: LC_CTYPE: cannot change locale (): No such file or directory" ...

So, given that the bug is in Python, the workaround should be probably done in the python program (as in @Jacob Oscarson's answer) or python invocation (by setting the locale vars to some adequate value).

---

my setup

$ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 11.4 (20F71)
      Kernel Version: Darwin 20.5.0

When the current language in System Preferences > Language & Reginon is English, then

Python 3.9.5 (v3.9.5:0a7dcbdb13, May  3 2021, 13:17:02) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
(None, 'UTF-8')
>>> 

and the low-level shell command output

$ locale
LANG=""
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

When the current language is Chinese, Simplified, then

$ py3
Python 3.9.5 (v3.9.5:0a7dcbdb13, May  3 2021, 13:17:02) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
('zh_CN', 'UTF-8')
>>> 

and the low-level shell command output

$ locale
LANG="zh_CN.UTF-8"
LC_COLLATE="zh_CN.UTF-8"
LC_CTYPE="zh_CN.UTF-8"
LC_MESSAGES="zh_CN.UTF-8"
LC_MONETARY="zh_CN.UTF-8"
LC_NUMERIC="zh_CN.UTF-8"
LC_TIME="zh_CN.UTF-8"
LC_ALL=

Note that whenever we switch the system language in System Preferences, we must restart Terminal to see the differences.