Why do Perl string operations on Unicode characters add garbage to the string?

Perl:

$string =~ s/[áàâã]/a/gi; #This line always prepends an "a"
$string =~ s/[éèêë]/e/gi;
$string =~ s/[úùûü]/u/gi;

This regular expression should convert "été" into "ete". Instead, it is converting it to "aetae". In other words, it prepends an "a" to every matched element. Even "à" is converted to "aa".

If I change the first line to this

$string =~ s/(á|à|â|ã)/a/gi;

it works, but... Now it prepends an e to every m开发者_如何学编程atched element (like "eetee").

Even though I found a suitable solution, why does it behave that way?

Edit 1:

I added "use utf8;", but it did not change the behavior (although it broke my output in JavaScript/AJAX).

Edit2:

The Stream originates from an Ajax Request, performed by jQuery. The site it originates from is set to UTF-8.

I am using Perl v5.10 (perl -v returns "This is perl, v5.10.0 built for i586-linux-thread-multi").

---

The problem is very likely down to not having

use utf8;

(or its equivalent for whatever coding system you are using) in your program. The weird replacements you have there look like problems with bytewise rather than characterwise regular expression replacement.

#!/usr/local/bin/perl
use warnings;
use strict;
use utf8;
binmode STDOUT, "utf8";
my $string = "été";

$string =~ s/[áàâã]/a/gi; #This line always prepends an "a"
$string =~ s/[éèêë]/e/gi;
$string =~ s/[úùûü]/u/gi;

print "$string\n";

prints

ete

If you are reading input from a file or from standard input, make sure you have the stream set to utf8 or whatever is appropriate for the encoding. For STDIN use

binmode STDOUT, "utf8";

If you are reading from a file, use

open my $file, "<:utf8", "file_name"

to get the encoding right. If it is not in UTF-8, use encoding(name) instead of utf8.

---

But did you really want to use regexes at all? Perhaps something like Text::Unidecode would be better

$ perl -Mutf8 -MText::Unidecode -E 'say unidecode("été")'
ete

---

I suspect that what is happening is that the [áàâã] part of your regular expression is not actually matching characters, but matching bytes. The UTF-8 encoding of those characters would look literally like this in the regular expression:

[\xC3\xA1\xC3\xA0\xC3\xA2\xC3\xA3]

And so when the regular expression is fed, for example , 'é' (\xC3\xA9), it looks at it a byte at a time, matches the \xC3, and replaces it with an 'a'. It does this for all of the \xC3 bytes it can find. So, 'été' is turned into 'a\xA9ta\xA9'.

Then the second regular expression, which looks like this:

[\xc3\xA9\xC3\xA8\xC3\xAA\xC3\xAB]

comes along, and it matches the \xA9 portion, and replaces it with an 'e'. So now, 'a\xA9ta\xA9' is turned into 'aetae'.

When you replace the [áàâã] with (á|à|â|ã), then that matches complete characters correctly on the first pass, but then your second regular expression has the original problem, and \xC3 characters are replaced with 'e' instead.

If this is still happening, even with use utf8;, then there may be a bug (or at least a limitation) in the Perl regular expression engine.

---

This is probably due to the fact that you're using UTF8 strings, and it's parsing them as if they're not, or similar.

Instead of using something like [áàâã] you should probbaly use something like [\xE1-\xE5]

and probably use use utf8; in your code too

---

This could also be a problem with Unicode Normalisation, as certain systems (I'm looking at you, OS X) represent extended Latin1 glyphs as a specific normalised representation that can break regular expressions when you refer to a character specifically instead of using a unicode or hex representation.

---

Something tells me it's because it doesn't know how to behave with characters with accent. By looking at your regular expression, everything seems fine. You might want to add:

use utf8;

---

I'd say you shouldn't really use regular expressions here. The easiest way to achieve this (although this might be undesirable) would be to convert your input string into US ASCII. The appropriate conversion tables should know that e is the closest equivalent to é.

Another option would be to use Unicode and normalize your string into NFD. This will break up all accented letters into base letter + diacritic. Then you can just go through your string and remove all combining diacritical characters.