How can I strip out odd copy-pasted characters like: ’

I have a php web app/tool that people end up copy-pasting data into开发者_如何学编程. The data eventually turns into XML, for which certain characters produce really odd character once they are saved. I am not sure if "’" looked like that before it was copy-pasted. It might have just been interpreted that way. It might have just been a long "-". In any case, all these characters are really odd. Is there a way to strip them out easily?

---

That is because PHP uses 8-bit encoding but your data is mostly likely written in UTF-8. You will find Joel's article on Encoding very enlightening.

And for the short answer try just encoding it in UTF-8

<?php

  $text = $entity['Entity']['title'];
  echo 'Original : ', $text."<br />";

  $enc = mb_detect_encoding($text, "UTF-8,ISO-8859-1");
  echo 'Detected Encoding '.$enc."<br />";

  echo 'Fixed Result: '.iconv($enc, "UTF-8", $text)."<br />";

?>

---

It would probably be easier in your case to whitelist rather than blacklist; i.e., make a list of acceptable characters and strip the rest. You can do this easily using preg_replace:

$str = preg_replace($str, "/[A-Za-z0-9'-._\(\)/");
                                               |
                                               V
                                              add more chars here

---

When you see a character pair starting with an accented "A" or "a", it generally means you're seeing a character whose actual encoding is iso-8859-1 displayed by software that thinks it's displaying utf-8.

If you're going to allow people to modify text in an XML document using tools that aren't XML-aware, the likelihood is that you will end up with characters encoded in iso-8859-1. That should be no problem provided the XML declaration at the start of the file is present and says that the encoding is iso-8859-1. But if there's no XML declaration, or if the encoding in the declaration is utf-8, you're going to end up with corrupt data.

You've asked about how to repair the data, but when you experience data corruption the focus should always be on prevention rather than repair.