Wedge

Public area => The Pub => Off-topic => Topic started by: snoopy-virtual on July 14th, 2011, 12:32 AM

Title: Changing from ISO-8859-15 to UTF-8
Post by: snoopy-virtual on July 14th, 2011, 12:32 AM

I have a small problem here.

I started ages ago doing all my pages in ISO-8859-1 and then later in ISO-8859-15

After a few years I installed some pieces of software in those pages that were working in UTF-8 because I was not aware at the time of the problems of different encodings.

With the years everything in all my pages have been going worst and worst with mixed encodings all over the place, and it's even worst because there are lots of different programs with bridges connecting all of them.

So I already decided some time ago to sort that out deciding to leave just one encoding and changing every thing with a different one, but was not decided which one I should leave until I learned you were doing Wedge with only UTF-8. Since then I have already been changing all the pages and the databases I have in ISO-8859-15 to UTF-8.

Doing this, when I am ready to move all of them to Wedge it will be easier for me.

The problem I have is everything is so mixed up and have relations all over the place that sometimes I have a situation like this one:

Imaging I have a database that used to be in ISO-8859-15 and I am getting data from that database and displaying it in a lot of different pages.

I have already changed all the data inside the DB to UTF-8, but I have only changed some of the pages to UTF-8. Some of the pages are still in ISO-8859-15, so I need to convert the data to ISO-8859-15 before displaying it in those page.

What I have done is something like:

Code: [Select]

if ($context['character_set'] != 'UTF-8')
$data_text = iconv("UTF-8", "ISO-8859-15", $data_text);


This is working for me, but today I was trying to help a guy that has exactly my same problem, I tried to do exactly that same trick and he got an error saying the function iconv() was not defined.

So I went to the manual and found out that function is not installed by default in all the PHP installations, but it's in an optional package and obviously it's not installed in this guy's server.

Now, the problem is that's the only function I knew to do the conversions from one encoding to another and that's the one I have been always using.

I have been checking the manual and cannot find another one to do the same job.

Any ideas?

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: Arantor on July 14th, 2011, 12:34 AM

Well, if you can't get the host to install iconv, you're kind of stuffed on the PHP side; there isn't anything AFAIK that is the replacement of iconv, and a PHP routine would be scary-slow by comparison.

However, all is not lost - if you're brave of heart. There is a way you can do it in MySQL if you need to - http://dev.mysql.com/doc/refman/5.0/en/charset-convert.html - by converting it that way but I'd be inclined to do bulk transpositions of the tables instead.

Specifically, create new tables, same structure as the old, but with pure UTF-8 collation etc, then INSERT INTO new_table SELECT FROM old_table.field, wherein you can plug in the CONVERT call into that select.

Then you just rename the two tables, effectively switching the newly minted UTF-8 only table for what was ISO-something.

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: snoopy-virtual on July 14th, 2011, 12:52 AM

The problem is not mine because I have iconv in my server. So I will need to tell this friend he will need to do it the hard way.

Luckily for him is not so bad, because he has a lot less pages than me and with fewer bridges between them.  :eheh:

Posted: July 14th, 2011, 12:49 AM

Anyway I suppose I will be the one doing it for him at the end of the day, so in that sense it IS my problem  :lol:

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: Arantor on July 14th, 2011, 12:53 AM

Well, once you make the conversion to UTF-8, it's done and there won't be any more complicated bridges or hurdles to cross.

UTF-8: it's like the future but here and now. ;)

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: chilly on July 16th, 2011, 05:04 PM

what about.... mb_convert_encoding found some interesting stuff looking through the php manual. had to find a good way to handle those conversions as I had to deal with multiple databases and wanted to be sure that my file would be ok. (texts are in the database for translation purpose. so the user can add new stuff on his own. but I'm still using lang files because I don't want to query the database on every page request. so everytime a user makes changes to lang strings it gets saved in the database and than written to lang file)

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: snoopy-virtual on July 16th, 2011, 05:53 PM

That function is part of mbstring extension and it's not installed by default in PHP, so it will be OK in some servers but not in others. I was looking for something more general.

Anyway I will try to see if my friend has this one installed.

The good thing is (as Arantor was saying) once you have everything in UTF-8, all those problems are history.

Title: Re: Changing from ISO-8859-15 to UTF-8
Post by: Arantor on July 16th, 2011, 09:13 PM

mb_* is great if it's available, but it's often not available. That said, I understand that in PHP 5.4 it is supposed to be enabled by default, finally.

That said, mb_* is not particularly fast.