Tag Archives: bom

php unicode utf-16 bom processing with iconv and mb_convert

basically php iconv doesnt handle FEFF BOMs while mb_convert does !

        $fh = fopen($s, "r");
        
        // watch out for notepad unicode FEFF BOM
        
        $utf16 = fgets($fh, 1024);

        $utf8 = mb_convert_encoding($utf16,'UTF-8','UTF-16');
        var_export($utf8);
        echo PHP_EOL;

        $utf8 = mb_convert_encoding($utf16,'UTF-8','UTF-16LE');
        var_export($utf8);
        echo PHP_EOL;

        $utf8 = iconv('UTF-16LE','UTF-8',mb_substr($utf16,1,null,'UTF-16LE'));
        var_export($utf8);
        echo PHP_EOL;

php and notepad unicode files

notepad unicode csvfiles have a 2 byte BOM (FEFF) and are UTF-16/UCS-2 little endian encoded

$ od -t x1z hot.csv | head
0000000 ff fe 48 00 6f 00 74 00 65 00 6c 00 49 00 44 00  >..H.o.t

$ od -t x2z hot.csv | head
0000000 feff 0048 006f 0074 0065 006c 0049 0044  >..H.o.t