Unicode Regular Expressions in Python

November 12, 2010

I was struggling with some unicode issues in Python. I wanted my app to match Eikanger-Bjørsvik Musikklag, which contained extended characters.

I was able to achieve this by doing the following:


Prefixing my regular expression with ur made it into a unicode raw string:

lRegEx = ur'\s*([\w\(\)&\'\-\. ]+)'

and then the regular expression was compiled as unicode:

lMatches = re.compile(lRegEx, re.U).match(line)  
# This used to be lMatches = re.match(lRegEx, line)

This meant that the \w would match unicode characters too. We extract the group values like normal:

lBandName = lMatches.group(1).strip()

Form Errors

In this case I was returning a form error which contained the unicode string, using the following code:

lBandName = lFormErrors[len(_BAND_PREFIX):-15]

This string wasn't unicode, and was failing later in the code when it was used as a %s replacement in a string, with:

'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)

To fix this, the above line of code was changed to:

lBandName = unicode(lFormErrors[len(_BAND_PREFIX):-15], 'UTF-8', errors='strict')


Tags: python regex unicode