Tag Archives: unicode

#python uses #ascii instead of #unicode by default #gotcha

by default python seems to run in ascii which means if you try and process a string with a utf-8 unicode character in it you will get the error

UnicodeEncodeError: 'ascii' codec can't encode character 
u'\xXX' in position YYYYY: ordinal not in range(128)

so to force your script to use utf-8 unicode instead you need to

see http://nedbatchelder.com/text/unipain.html and

import sys
reload(sys)
sys.setdefaultencoding('utf8')
print sys.getdefaultencoding()

php unicode utf-16 bom processing with iconv and mb_convert

basically php iconv doesnt handle FEFF BOMs while mb_convert does !

        $fh = fopen($s, "r");
        
        // watch out for notepad unicode FEFF BOM
        
        $utf16 = fgets($fh, 1024);

        $utf8 = mb_convert_encoding($utf16,'UTF-8','UTF-16');
        var_export($utf8);
        echo PHP_EOL;

        $utf8 = mb_convert_encoding($utf16,'UTF-8','UTF-16LE');
        var_export($utf8);
        echo PHP_EOL;

        $utf8 = iconv('UTF-16LE','UTF-8',mb_substr($utf16,1,null,'UTF-16LE'));
        var_export($utf8);
        echo PHP_EOL;

php and notepad unicode files

notepad unicode csvfiles have a 2 byte BOM (FEFF) and are UTF-16/UCS-2 little endian encoded

$ od -t x1z hot.csv | head
0000000 ff fe 48 00 6f 00 74 00 65 00 6c 00 49 00 44 00  >..H.o.t

$ od -t x2z hot.csv | head
0000000 feff 0048 006f 0074 0065 006c 0049 0044  >..H.o.t