开发者

What encoding looks exactly like ASCII but has NULL bytes before each byte?

开发者 https://www.devze.com 2023-01-17 11:01 出处:网络
I have a string that looks and behaves as follows (Python code provided). WTF?! What encoding is it in?

I have a string that looks and behaves as follows (Python code provided). WTF?! What encoding is it in?

s = u'\x00Q\x00u\x00i\x00c\x00k'
>>> print s
Quick
>>>
>>> s == 'Quick'
False
>>>
>>> import re
>>> re.search('Quick', s)
>>>
>>> import chardet
>>> chardet.detect(s)
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:69: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if aBuf[:3] == '\xEF\xBB\xBF':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:72: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\xFF\xFE\x00\x00':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:75: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as be开发者_如何学运维ing unequal
  elif aBuf[:4] == '\x00\x00\xFE\xFF':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:78: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\xFE\xFF\x00\x00':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:81: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\x00\x00\xFF\xFE':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:84: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:2] == '\xFF\xFE':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:87: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:2] == '\xFE\xFF':
{'confidence': 1.0, 'encoding': 'ascii'}
>>>
>>> chardet.detect(s)
{'confidence': 1.0, 'encoding': 'ascii'}
>>> 


UTF-16 big endian


You have UTF-16BE without a BOM. As documented, chardet doesn't grok UTF-nnxE without a BOM.

>>> s = '\x00Q\x00u\x00i\x00c\x00k' #### Note: dropping the spurious `u` prefix
>>> s.decode('utf_16be')
u'Quick'
>>>

chardet is also not smart enough to raise a DontBeSilly exception if you feed it unicode :-)

0

精彩评论

暂无评论...
验证码 换一张
取 消