Language editing inside Wedge

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Language editing inside Wedge
« on September 18th, 2012, 12:32 AM »
You all know my feelings on editing files, and currently the language editor does just that - even for plugins (eek), and right now it's broken. So it needs an overhaul.

So here's what I was thinking. How about, we store the changed strings in the database and load them when necessary? That sounds awful, but let me temper it with this which only occurred to me today.

We know what files contain what strings, and we know what file is being edited at any one time. So if we keep a record of what strings have been modified in the DB versus the original files, we can add a check to loadLanguage to say 'look, we've asked for <language file x>, has there been any changes to that file?' then load any changes to it after the fact, as it were.

All we then need is a DB table containing language, file, optional plugin ID, and the new string and we're golden. Plus if a given file hasn't been updated, no harm, no foul (and no extra query)

This way we get to allow language strings to be edited (cleanly), without any file edits (which is also a security improvement). I'd also move the registration agreement into the language files too and have them all supported in one clean place.

Thoughts?
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #1, on September 18th, 2012, 03:16 AM »
I'm fairly sure this is approximately how MyBB does things. Seems to me to be a solution that works fine.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #2, on September 18th, 2012, 03:27 AM »
Well, I know other systems dump everything into the DB, but I'd rather avoid that if possible.

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #3, on September 18th, 2012, 03:46 AM »
It's really only the translations and themes/skins. But I also think themes/skins are easier to maintain as flat files, so I guess you're right.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #4, on September 18th, 2012, 03:48 AM »
See, there's a problem right there - there is absolutely no need to store all the standard stuff in the DB - people do not modify everything.

I was gobsmacked when I looked through the installer for XenForo and noted hundreds of KBs of stuff dumped into the DB, IMO unnecessarily. I understand why, but that doesn't mean I have to like it.

Doing it this way does give the compromise of ease of access, the security upside and whatnot.

* Arantor wonders what others (Nao in particular) thinks about it.

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #5, on September 18th, 2012, 03:57 AM »
A question, how would you handle updates to these where a conflict has arisen from someone editing the original? Or would you not "allow" people to edit the original, instead letting people copy everything, making updates to the original and asking people to merge?

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #6, on September 18th, 2012, 04:09 AM »
That's the thing: the software itself - assuming I did this - wouldn't ever touch the original files. Meaning that their changes are always a delta to the original - but if they edit the master file itself, that's entirely their look out.

* Arantor is confused by the question, seems to be asking something a bit irrelevant to the issues at hand...

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #7, on September 18th, 2012, 12:41 PM »
It's really relevant to any topic discussing translations, in my opinion.
Quote
Meaning that their changes are always a delta to the original
So there will always be a copy to edit, or will you rely on users to do so themselves?

Bugo

  • Posts: 3
Re: Language editing inside Wedge
« Reply #8, on September 18th, 2012, 01:52 PM »
Quote from Arantor on September 18th, 2012, 12:32 AM
So here's what I was thinking. How about, we store the changed strings in the database and load them when necessary? That sounds awful, but let me temper it with this which only occurred to me today.
The idea is very interesting. Is caching implied too?
Quote from Norodo on September 18th, 2012, 03:16 AM
I'm fairly sure this is approximately how MyBB does things. Seems to me to be a solution that works fine.
There is a pretty easy interface to change the language strings in MyBB.
(Though I used to use only Notepad++ :))

 mybb_lang_select.png - 16.74 kB, 1228x363, viewed 227 times.

 mybb_lang_edit.png - 43.83 kB, 1381x626, viewed 219 times.


Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #9, on September 18th, 2012, 03:48 PM »
Quote from Arantor on September 18th, 2012, 03:48 AM
* Arantor wonders what others (Nao in particular) thinks about it.
I have absolutely no idea... Would this make the code slower overall?
Also, how do we deal with PHP code in languages files, you know, the array declarations, string concatenation and other things...?

Oh, and while I'm at it -- I'm also working on integrating language strings directly into JavaScript files.
How do I do that...? Oh, so simple.

editor-123456.js.gz become editor-french-123456.js.gz (not renamed if it's the default language), and we put the strings directly into the different cached versions... Voilà.
It's something I should have done long ago... But it's made easier by JS having its own cache folder now.
I'm not ready for committing it, though. And I'm in the middle of trying to fix sidebars so they don't use tables anymore -- including IE crap.

Oh, and while I'm at it (bis repetita) -- Pete, do you remember the outcome of our conversation on membergroups for contact lists...? What did we decide in the end? Did we decide anything at all, actually? install.sql has an entry for {db}contact_lists, and my local (at least) install.sql also defined privacy tables that set a privacy flag between 'member ID', 'membergroup ID' and 'contact list ID'... I *thought* we'd settled on membergroups, but I'm not so sure anymore now... And perhaps it'd be best to really separate forum membergroups from contact lists, as long as all privacy-related code supports contact lists as well... I don't know.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #10, on September 18th, 2012, 06:17 PM »
Quote
It's really relevant to any topic discussing translations, in my opinion.
Not really, it isn't.

The master language files are always left alone, whatever the language. Then the edits are made locally to a given language. Doing a translation means in the first case manually copying the files and editing them - but only until we get around to making a proper language editor here for translators (which will auto rebuild language packs)

In any case this setup is not designed for translators' benefit, it is designed for people who want to customise their installation without having to edit files directly (with all the attendant problems that go with it)

You guys already knew that SMF had a built-in language editing facility that also wasn't designed for translation, right?
Quote
The idea is very interesting. Is caching implied too?
Maybe, but fortunately unlike SMF, it shouldn't require the usual realms of having to keep clearing the cache because it should be able to be automated.
Quote
There is a pretty easy interface to change the language strings in MyBB.
Great, more people that don't know SMF has a language editor... it looks almost the same. Just that it edits raw files.
Quote
I have absolutely no idea... Would this make the code slower overall?
Also, how do we deal with PHP code in languages files, you know, the array declarations, string concatenation and other things...?
There shouldn't be much if any string concatenation, I'm fairly sure I phased most of that crap out.

As far as the array declarations, it's still not a problem, the language editor just has to understand how to read the array, and then be able to overwrite the array later. All that really means is to store the serialised array and unserialise it when loading from the DB.

The speed issue is the big one. Practically it means intercepting loadLanguage and loadPluginLanguage calls, perform them then pull from the database. We can also cache the resulting language file, though.
Quote
Oh, and while I'm at it -- I'm also working on integrating language strings directly into JavaScript files.
How do I do that...? Oh, so simple.
This would complicate the above significantly, though. The only real problem I see with doing that is you end up consuming a lot of space in cache files by having editor-english.js and editor-french.js - and if you have a site with lots of languages, you potentially end up with dozens of files at once.
Quote
What did we decide in the end?
I really can't remember :(

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #11, on September 18th, 2012, 06:25 PM »
Quote from Arantor on September 18th, 2012, 06:17 PM
The master language files are always left alone, whatever the language. Then the edits are made locally to a given language. Doing a translation means in the first case manually copying the files and editing them - but only until we get around to making a proper language editor here for translators (which will auto rebuild language packs)
This might be how it is for Wedge, but not neccessarily how it is for every program or how it should be or needs to be. Hence the question. I'm merely trying to probe my way into what would be the most user friendly way of doing things.

To me it sounds like a fairly decent way of doing things, I suppose. Perhaps one should put the master language(s) in the database, "only" able to be printed out to a file that can be edited. This way you'd avoid people accidentally editing the master file(s).

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #12, on September 18th, 2012, 06:50 PM »
You know me by now, I do not give that much credence in how other things do it. I care about how I think it should be done in Wedge. You're conflating two separate use cases under one umbrella, which as I already alluded to is not how it should be. Let's clear that up right now.

Use case 1: Translators
In the *short term*, translators have to copy files and edit them directly. Putting the bulk of stuff in the database makes it harder for them to translate, unless we build a complete translation interface directly into Wedge, which is a very large waste of time by making something that ends up with every user when only a fraction of a percent of users will use it.

In the *long term* we end up with something like sm.org has, a translation facility. It's a facility that tracks all the languages, all the changes and allows for building translation packages, much like SMF's language packs. This doesn't require any file editing, it's all a web based interface.

User friendliness is not really a concern in these cases because of the very small number of people that will have to deal with these things.


User case 2: Users wanting to customise their setup
This is the only scenario I give a damn about in the core package itself, because this would be useful to potentially any site owner who wants to customise their site. There is already a foundation of one of these in SMF and Wedge (see Admin > Languages, you can select a language then edit its files there)

I want to be able to give people the ability to edit the language files directly from the ACP, without them having to go into code or manually edit files, or to have to open the permissions up. Doing so as suggested would allow people to edit things from the ACP, without any of these other issues - and it could be made to be more friendly than what is presented available right now. I mean, it would be huge if people even knew it was there...
Quote
Perhaps one should put the master language(s) in the database, "only" able to be printed out to a file that can be edited. This way you'd avoid people accidentally editing the master file(s).
That's even less user friendly, in fact, and I'm well aware that other systems do this and it is one reason I've avoided them.

Norodo

  • Oh you Baidu, so randumb. (60 sites being indexed at once? Jeez)
  • Posts: 469
Re: Language editing inside Wedge
« Reply #13, on September 18th, 2012, 07:08 PM »
Quote from Arantor on September 18th, 2012, 06:50 PM
That's even less user friendly, in fact, and I'm well aware that other systems do this and it is one reason I've avoided them.
Interesting. I've never seen this implemented, it was an off the top of my head kind of idea. But thinking about how you divide this into translators and people who just want to modify a little bit, I agree it suits neither.
Quote
You're conflating two separate use cases under one umbrella
That might just be it.
Quote
There is already a foundation of one of these in SMF and Wedge (see Admin > Languages, you can select a language then edit its files there)
My personal opinion of said feature is that it is totally useless. I already installed the software using SFTP and file edits, and I'm going to have to open SFTP again to edit the file permissions in order to even be able to edit them in the browser. Total waste of time in my opinion, when you could just open the file in SFTP and get a much better editor.
Quote
Doing so as suggested would allow people to edit things from the ACP, without any of these other issues - and it could be made to be more friendly than what is presented available right now. I mean, it would be huge if people even knew it was there.
Good. Then I think we are on the same page here. I think I agree with your idea from OP. Translators will have to copy files, that is not an issue, and people modifying will only affect that specific key value that they have changed, and this would be read from the database. That sounds to me like an ideal solution and I would be happy if it made its way into final.

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #14, on September 23rd, 2012, 01:22 PM »
Quote from Arantor on September 18th, 2012, 06:17 PM
There shouldn't be much if any string concatenation, I'm fairly sure I phased most of that crap out.
There are still at least 9x2 cases of string concatenation across 7x2 language files, I'm afraid ;)
Quote
As far as the array declarations, it's still not a problem, the language editor just has to understand how to read the array, and then be able to overwrite the array later. All that really means is to store the serialised array and unserialise it when loading from the DB.
Hmm... Yeah, I didn't even consider that one lol...
Quote
The speed issue is the big one. Practically it means intercepting loadLanguage and loadPluginLanguage calls, perform them then pull from the database. We can also cache the resulting language file, though.
Yeah, it should be a given. A bit like Wess which takes, transforms and re-caches a text file ;)
Quote
This would complicate the above significantly, though. The only real problem I see with doing that is you end up consuming a lot of space in cache files by having editor-english.js and editor-french.js - and if you have a site with lots of languages, you potentially end up with dozens of files at once.
If you only have one language installed (or enabled?) on your system, it's not a problem I guess, because you'll only get script.js files... (I don't see why we should add a language prefix to something that's already the default language.)
Other than that, it's just on average twice the current amount of cached JS files, which is very, very small, especially when compared to the number of CSS files in the css cache... Just go have a look ;)

No, the real problem with language caching here, is two-fold. Here's how I've been writing the stuff: script files can contain $txt['something'] strings, and we add a "@language index" keyword at the start of the file (or anywhere...), to tell Wedge where it can find this string, thus it loads the language file beforehand. You can use multiple @language entries, or merge all entries into a "@language index, Admin, Media" call or whatever you want. Flexibility is my thing.

- The first problem is easily solved, but pretty ugly if you'll ask me... People using the default language: no problems. People using another language: add_js_file (or whatever it is) should test for script-french.js.gz, if it's there, return that, otherwise test for script.js.gz, if it's there, return that, otherwise cache the file, under the name script-french.js.gz if it has language strings in it, or script.js.gz otherwise. The problem is that French users have to go through two file_exists tests. Okay, it's not that big a deal...

- The second problem comes when you update the language files. Even if these aren't strings we use in the script, we still need to regenerate the cache, 'just in case'... But how do you determine that the language files are modified? If the script file is already cached, does that mean you have to open it the uncached version, then look for the @language command, list all language files in it, then check every single language file for date changes...? It screams "needs wedge_settings caching!" to me... But I don't even know how to deal with that one. e.g. everytime we do a loadLanguage(), do we need to see if the filedate has changed, and if yes, store it into a DB setting like 'lang_date_Admin_french' or something...? Does it make sense at all?

Other than that, this feature is something I'm really starting to like. It was an easy implementation (i.e. just a few regexps), it works very well (with the added benefit of being able to cache JavaScriptEscape calls!!), and it does save a lot of space in the HTML... Which is something I'm always going to push for!
I just need to coordinate this change with you, considering you're thinking it's going to cause trouble for plugins.
Quote
Quote
What did we decide in the end?
I really can't remember :(
I'm currently inclined to go for contact lists, because that way we could keep the current code for membergroups (find_in_set and other crap), and use cross-table queries for contact lists. Best of both worlds I guess... (Even if it makes things more complicated for media albums, because that's what they use... Membergroups and per-member settings... Adding contact list settings to these, hmm... I don't know. But OTOH, media albums are one of the most critical areas when it comes to privacy settings.)