Python string processing, Unicode & Beautiful Soup

I've been searching for solutions to a bug that I have but haven't found/understood one that will work. Essentially, if I use the string functions (translate, strip, etc.) I get Unicode errors (ascii' codec can't encode character 'x' in position y: ordinal not in range(128) 开发者_Go百科. But when I try Beautiful Soup to process text, I don't get Unicode errors but the degree of difficulty (should I say unfamiliarity) is pretty high for me. Here's excerpts of the code I have:

...

import urllib2,sys
import re
import os
import urllib
import string
import time
from BeautifulSoup import BeautifulSoup,NavigableString, SoupStrainer
from string import maketrans
import codecs

trantab=string.maketrans(",",";") 
...

                html5 = urllib2.urlopen(address5).read()
                time.sleep(1.5)

                soup5 = BeautifulSoup(html5)

                for company in iter(soup5.findAll(height="20px")):
                    stream = ""
                    count_detail = 1
                    for tag in iter(company.findAll('td')):
                        if count_detail > 1:
                            stream = stream + string.translate(str(tag.text),trantab)
                            if count_detail < 4 :
                                stream=stream+","
                        count_detail = count_detail + 1
                    print str(storenum)+","+branch_name_address+","+ stream

....

This script runs for a while and then bombs at stream = stream + string.translate(str(tag.text),trantab)

Basically, I'm just trying to replace commas with semicolon in the fields that I am processing.

Also, was trying to remove embedded whitespace/blanks using string.strip but I get similar errors.

How would I do the same thing using Beautiful soup (as far as replacing commas with semicolon and removing whitespace)?

Or is there code to solve those pesky Unicode errors if I just stick to string functions?

---

You are mixing str objects with unicode objects, which leads the Python interpreter to coerce one into the other. String/Unicode coercion requires an encoding, which is assumed to be ascii by default. When this assumption doesn't hold, you get this kind of errors.

The general solution is not to mix str with unicode: use unicode everywhere possible and make any conversion explicit with string.encode('utf8', 'strict') and unicode_string.decode('utf8', 'strict') (UTF-8 is an example).

In this case, replace

stream = stream + string.translate(str(tag.text),trantab)

with

stream = stream + tag.text.replace(u',', u';')