开发者

How make support cyrillic letters in JavaCC?

开发者 https://www.devze.com 2023-02-27 18:10 出处:网络
Can I make cyrillic tokens for *.jjt file? For exampe tokens are given in jjt: TOKEN : /* LITERALS */ { < TEST: \"тест\" >

Can I make cyrillic tokens for *.jjt file?

For exampe tokens are given in jjt:

TOKEN : /* LITERALS */
{
    < TEST: "тест" >
|   < DEVELOP: "разработка" >
}

but the tokens in jj file look terrible:

TOKEN : /* LITERALS */
{
    < TEST: "\u0421\u201a\u0420\u00b5\u0421\u0403\u0421\u201a" >
|   < DEVELOP: "\u0421\u0402\u0420\u00b0\u0420\u00b7\u0421\u0402\u0420\u00b0\u0420\u00b1\u0420\u0455\u0421\u201a\u0420\u0454\u0420\u00b0" >
}

Use the following options:

options {
  JDK_VERSION = "1.开发者_如何学C6";
  UNICODE_INPUT = true;
  JAVA_UNICODE_ESCAPE = false;
  TRACK_TOKENS=true;
  STATIC=false;
}

If I do Unicode with the help of her class:

class GetUnicode {
  public static void main(String[] args) {
      if (args.length < 1) return;
      for (String input: args) {
        for (int index = 0; index < input.length(); ++index) {
            final char c = input.charAt(index); 
            final String s = String.format ("\\u%04x", (int)c);
            System.out.print(s);
        }
        System.out.println();
      }
  }
}

After the start of the program:

$ java GetUnicode тест разработка                                                                                                                                                                                       
\u0442\u0435\u0441\u0442
\u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0430

As a result, I can see that the unicode strings do not match. For example:

"\u0442\u0435\u0441\u0442" != "\u0421\u201a\u0420\u00b5\u0421\u0403\u0421\u201a"

Do you have ideas why this happens?

PS: This bug occurs only under Windows OS.


The .jj file has been generated with the cyrillic characters escaped (Unicode). This is a good thing since they can then not be misread by using the wrong encoding. You should not worry about it; the token manager will work just fine.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号