If I implement UTF-16 file handler, can it accurately process all other encodings

I am writing a small scale HTML crawler in java. I want to have a single file handler that can open all the HTML file one by one and process them. But, there is no way to know what the HTML file is encoded in before actually opening that particular file. So, I am willing to know if I can have something like this :

new BufferedReader(
    new InputStreamReader(
        new FileInputStream(file), UTF16));

and the handler will be able to read all possible encodings (开发者_Python百科in an accurate way) the files may have (my idea is UTF16 is backward compatible with all other encodings). I will have to deal with following encodings.

charset=iso-8859-1
charset=utf-8
charset=iso-8859-1
charset=iso-8859-15'
charset="UTF-8"
charset=windows-1252
charset=utf-16

Thanks. Any suggestion would be highly appreciated.

---

No, UTF-16 is certainly not compatible with all other encodings (in that you can't use a UTF-16 decoder to decode any old text). Try using it for UTF-8, ISO-Latin-1 or any number of other encodings and it will fail.

Assuming this HTML has been fetched from a web server, you should remember the content type given in the response. Alternatively you could heuristically guess the encoding, of course.

---

No UTF16 can only understand files encoded in UTF16. Your best bet lies in determining the encoding before you process the file. Use the GuessEncoding library to detect the encoding and then construct the reader in the encoding detected.

I'd use this in combination with Jon Skeet's suggestion

---

Wow. Just wow.

The only way to do this is to read the first few hundred bytes in a safe encoding such as Windows-1252 and look for the NULLS that indicate UTF-16/32 and the META charset tag.

Failing in that, look at the headers for a charset.

If no header found, assume UTF-8 (standard) unless it parses out, then assume Windows-1252 (common error is sending Windows-1252 with no charset header).