Wedge

Public area => The Pub => Topic started by: Arantor on September 18th, 2012, 12:32 AM

Title: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 12:32 AM
You all know my feelings on editing files, and currently the language editor does just that - even for plugins (eek), and right now it's broken. So it needs an overhaul.

So here's what I was thinking. How about, we store the changed strings in the database and load them when necessary? That sounds awful, but let me temper it with this which only occurred to me today.

We know what files contain what strings, and we know what file is being edited at any one time. So if we keep a record of what strings have been modified in the DB versus the original files, we can add a check to loadLanguage to say 'look, we've asked for <language file x>, has there been any changes to that file?' then load any changes to it after the fact, as it were.

All we then need is a DB table containing language, file, optional plugin ID, and the new string and we're golden. Plus if a given file hasn't been updated, no harm, no foul (and no extra query)

This way we get to allow language strings to be edited (cleanly), without any file edits (which is also a security improvement). I'd also move the registration agreement into the language files too and have them all supported in one clean place.

Thoughts?
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 03:16 AM
I'm fairly sure this is approximately how MyBB does things. Seems to me to be a solution that works fine.
Title: Re: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 03:27 AM
Well, I know other systems dump everything into the DB, but I'd rather avoid that if possible.
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 03:46 AM
It's really only the translations and themes/skins. But I also think themes/skins are easier to maintain as flat files, so I guess you're right.
Title: Re: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 03:48 AM
See, there's a problem right there - there is absolutely no need to store all the standard stuff in the DB - people do not modify everything.

I was gobsmacked when I looked through the installer for XenForo and noted hundreds of KBs of stuff dumped into the DB, IMO unnecessarily. I understand why, but that doesn't mean I have to like it.

Doing it this way does give the compromise of ease of access, the security upside and whatnot.

/mewonders what others (Nao in particular) thinks about it.
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 03:57 AM
A question, how would you handle updates to these where a conflict has arisen from someone editing the original? Or would you not "allow" people to edit the original, instead letting people copy everything, making updates to the original and asking people to merge?
Title: Re: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 04:09 AM
That's the thing: the software itself - assuming I did this - wouldn't ever touch the original files. Meaning that their changes are always a delta to the original - but if they edit the master file itself, that's entirely their look out.

/meis confused by the question, seems to be asking something a bit irrelevant to the issues at hand...
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 12:41 PM
It's really relevant to any topic discussing translations, in my opinion.
Quote
Meaning that their changes are always a delta to the original
So there will always be a copy to edit, or will you rely on users to do so themselves?
Title: Re: Language editing inside Wedge
Post by: Bugo on September 18th, 2012, 01:52 PM
Quote from Arantor on September 18th, 2012, 12:32 AM
So here's what I was thinking. How about, we store the changed strings in the database and load them when necessary? That sounds awful, but let me temper it with this which only occurred to me today.
The idea is very interesting. Is caching implied too?
Quote from Norodo on September 18th, 2012, 03:16 AM
I'm fairly sure this is approximately how MyBB does things. Seems to me to be a solution that works fine.
There is a pretty easy interface to change the language strings in MyBB.
(Though I used to use only Notepad++ :))
Title: Re: Language editing inside Wedge
Post by: Nao on September 18th, 2012, 03:48 PM
Quote from Arantor on September 18th, 2012, 03:48 AM
/mewonders what others (Nao in particular) thinks about it.
I have absolutely no idea... Would this make the code slower overall?
Also, how do we deal with PHP code in languages files, you know, the array declarations, string concatenation and other things...?

Oh, and while I'm at it -- I'm also working on integrating language strings directly into JavaScript files.
How do I do that...? Oh, so simple.

editor-123456.js.gz become editor-french-123456.js.gz (not renamed if it's the default language), and we put the strings directly into the different cached versions... VoilĂ .
It's something I should have done long ago... But it's made easier by JS having its own cache folder now.
I'm not ready for committing it, though. And I'm in the middle of trying to fix sidebars so they don't use tables anymore -- including IE crap.

Oh, and while I'm at it (bis repetita) -- Pete, do you remember the outcome of our conversation on membergroups for contact lists...? What did we decide in the end? Did we decide anything at all, actually? install.sql has an entry for {db}contact_lists, and my local (at least) install.sql also defined privacy tables that set a privacy flag between 'member ID', 'membergroup ID' and 'contact list ID'... I *thought* we'd settled on membergroups, but I'm not so sure anymore now... And perhaps it'd be best to really separate forum membergroups from contact lists, as long as all privacy-related code supports contact lists as well... I don't know.
Title: Re: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 06:17 PM
Quote
It's really relevant to any topic discussing translations, in my opinion.
Not really, it isn't.

The master language files are always left alone, whatever the language. Then the edits are made locally to a given language. Doing a translation means in the first case manually copying the files and editing them - but only until we get around to making a proper language editor here for translators (which will auto rebuild language packs)

In any case this setup is not designed for translators' benefit, it is designed for people who want to customise their installation without having to edit files directly (with all the attendant problems that go with it)

You guys already knew that SMF had a built-in language editing facility that also wasn't designed for translation, right?
Quote
The idea is very interesting. Is caching implied too?
Maybe, but fortunately unlike SMF, it shouldn't require the usual realms of having to keep clearing the cache because it should be able to be automated.
Quote
There is a pretty easy interface to change the language strings in MyBB.
Great, more people that don't know SMF has a language editor... it looks almost the same. Just that it edits raw files.
Quote
I have absolutely no idea... Would this make the code slower overall?
Also, how do we deal with PHP code in languages files, you know, the array declarations, string concatenation and other things...?
There shouldn't be much if any string concatenation, I'm fairly sure I phased most of that crap out.

As far as the array declarations, it's still not a problem, the language editor just has to understand how to read the array, and then be able to overwrite the array later. All that really means is to store the serialised array and unserialise it when loading from the DB.

The speed issue is the big one. Practically it means intercepting loadLanguage and loadPluginLanguage calls, perform them then pull from the database. We can also cache the resulting language file, though.
Quote
Oh, and while I'm at it -- I'm also working on integrating language strings directly into JavaScript files.
How do I do that...? Oh, so simple.
This would complicate the above significantly, though. The only real problem I see with doing that is you end up consuming a lot of space in cache files by having editor-english.js and editor-french.js - and if you have a site with lots of languages, you potentially end up with dozens of files at once.
Quote
What did we decide in the end?
I really can't remember :(
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 06:25 PM
Quote from Arantor on September 18th, 2012, 06:17 PM
The master language files are always left alone, whatever the language. Then the edits are made locally to a given language. Doing a translation means in the first case manually copying the files and editing them - but only until we get around to making a proper language editor here for translators (which will auto rebuild language packs)
This might be how it is for Wedge, but not neccessarily how it is for every program or how it should be or needs to be. Hence the question. I'm merely trying to probe my way into what would be the most user friendly way of doing things.

To me it sounds like a fairly decent way of doing things, I suppose. Perhaps one should put the master language(s) in the database, "only" able to be printed out to a file that can be edited. This way you'd avoid people accidentally editing the master file(s).
Title: Re: Language editing inside Wedge
Post by: Arantor on September 18th, 2012, 06:50 PM
You know me by now, I do not give that much credence in how other things do it. I care about how I think it should be done in Wedge. You're conflating two separate use cases under one umbrella, which as I already alluded to is not how it should be. Let's clear that up right now.

Use case 1: Translators
In the *short term*, translators have to copy files and edit them directly. Putting the bulk of stuff in the database makes it harder for them to translate, unless we build a complete translation interface directly into Wedge, which is a very large waste of time by making something that ends up with every user when only a fraction of a percent of users will use it.

In the *long term* we end up with something like sm.org has, a translation facility. It's a facility that tracks all the languages, all the changes and allows for building translation packages, much like SMF's language packs. This doesn't require any file editing, it's all a web based interface.

User friendliness is not really a concern in these cases because of the very small number of people that will have to deal with these things.


User case 2: Users wanting to customise their setup
This is the only scenario I give a damn about in the core package itself, because this would be useful to potentially any site owner who wants to customise their site. There is already a foundation of one of these in SMF and Wedge (see Admin > Languages, you can select a language then edit its files there)

I want to be able to give people the ability to edit the language files directly from the ACP, without them having to go into code or manually edit files, or to have to open the permissions up. Doing so as suggested would allow people to edit things from the ACP, without any of these other issues - and it could be made to be more friendly than what is presented available right now. I mean, it would be huge if people even knew it was there...
Quote
Perhaps one should put the master language(s) in the database, "only" able to be printed out to a file that can be edited. This way you'd avoid people accidentally editing the master file(s).
That's even less user friendly, in fact, and I'm well aware that other systems do this and it is one reason I've avoided them.
Title: Re: Language editing inside Wedge
Post by: Norodo on September 18th, 2012, 07:08 PM
Quote from Arantor on September 18th, 2012, 06:50 PM
That's even less user friendly, in fact, and I'm well aware that other systems do this and it is one reason I've avoided them.
Interesting. I've never seen this implemented, it was an off the top of my head kind of idea. But thinking about how you divide this into translators and people who just want to modify a little bit, I agree it suits neither.
Quote
You're conflating two separate use cases under one umbrella
That might just be it.
Quote
There is already a foundation of one of these in SMF and Wedge (see Admin > Languages, you can select a language then edit its files there)
My personal opinion of said feature is that it is totally useless. I already installed the software using SFTP and file edits, and I'm going to have to open SFTP again to edit the file permissions in order to even be able to edit them in the browser. Total waste of time in my opinion, when you could just open the file in SFTP and get a much better editor.
Quote
Doing so as suggested would allow people to edit things from the ACP, without any of these other issues - and it could be made to be more friendly than what is presented available right now. I mean, it would be huge if people even knew it was there.
Good. Then I think we are on the same page here. I think I agree with your idea from OP. Translators will have to copy files, that is not an issue, and people modifying will only affect that specific key value that they have changed, and this would be read from the database. That sounds to me like an ideal solution and I would be happy if it made its way into final.
Title: Re: Language editing inside Wedge
Post by: Nao on September 23rd, 2012, 01:22 PM
Quote from Arantor on September 18th, 2012, 06:17 PM
There shouldn't be much if any string concatenation, I'm fairly sure I phased most of that crap out.
There are still at least 9x2 cases of string concatenation across 7x2 language files, I'm afraid ;)
Quote
As far as the array declarations, it's still not a problem, the language editor just has to understand how to read the array, and then be able to overwrite the array later. All that really means is to store the serialised array and unserialise it when loading from the DB.
Hmm... Yeah, I didn't even consider that one lol...
Quote
The speed issue is the big one. Practically it means intercepting loadLanguage and loadPluginLanguage calls, perform them then pull from the database. We can also cache the resulting language file, though.
Yeah, it should be a given. A bit like Wess which takes, transforms and re-caches a text file ;)
Quote
This would complicate the above significantly, though. The only real problem I see with doing that is you end up consuming a lot of space in cache files by having editor-english.js and editor-french.js - and if you have a site with lots of languages, you potentially end up with dozens of files at once.
If you only have one language installed (or enabled?) on your system, it's not a problem I guess, because you'll only get script.js files... (I don't see why we should add a language prefix to something that's already the default language.)
Other than that, it's just on average twice the current amount of cached JS files, which is very, very small, especially when compared to the number of CSS files in the css cache... Just go have a look ;)

No, the real problem with language caching here, is two-fold. Here's how I've been writing the stuff: script files can contain $txt['something'] strings, and we add a "@language index" keyword at the start of the file (or anywhere...), to tell Wedge where it can find this string, thus it loads the language file beforehand. You can use multiple @language entries, or merge all entries into a "@language index, Admin, Media" call or whatever you want. Flexibility is my thing.

- The first problem is easily solved, but pretty ugly if you'll ask me... People using the default language: no problems. People using another language: add_js_file (or whatever it is) should test for script-french.js.gz, if it's there, return that, otherwise test for script.js.gz, if it's there, return that, otherwise cache the file, under the name script-french.js.gz if it has language strings in it, or script.js.gz otherwise. The problem is that French users have to go through two file_exists tests. Okay, it's not that big a deal...

- The second problem comes when you update the language files. Even if these aren't strings we use in the script, we still need to regenerate the cache, 'just in case'... But how do you determine that the language files are modified? If the script file is already cached, does that mean you have to open it the uncached version, then look for the @language command, list all language files in it, then check every single language file for date changes...? It screams "needs wedge_settings caching!" to me... But I don't even know how to deal with that one. e.g. everytime we do a loadLanguage(), do we need to see if the filedate has changed, and if yes, store it into a DB setting like 'lang_date_Admin_french' or something...? Does it make sense at all?

Other than that, this feature is something I'm really starting to like. It was an easy implementation (i.e. just a few regexps), it works very well (with the added benefit of being able to cache JavaScriptEscape calls!!), and it does save a lot of space in the HTML... Which is something I'm always going to push for!
I just need to coordinate this change with you, considering you're thinking it's going to cause trouble for plugins.
Quote
Quote
What did we decide in the end?
I really can't remember :(
I'm currently inclined to go for contact lists, because that way we could keep the current code for membergroups (find_in_set and other crap), and use cross-table queries for contact lists. Best of both worlds I guess... (Even if it makes things more complicated for media albums, because that's what they use... Membergroups and per-member settings... Adding contact list settings to these, hmm... I don't know. But OTOH, media albums are one of the most critical areas when it comes to privacy settings.)
Title: Re: Language editing inside Wedge
Post by: Arantor on September 23rd, 2012, 04:30 PM
Quote
There are still at least 9x2 cases of string concatenation across 7x2 language files, I'm afraid
Then I'll fix that first. I am aware there are cases of concatenation in the language files for the purpose of encoding line-endings into certain strings. This is standard practice for the editor and it is already actually aware of this. But I'll check for any other cases and fix it.
Quote
Other than that, it's just on average twice the current amount of cached JS files, which is very, very small, especially when compared to the number of CSS files in the css cache... Just go have a look ;)
I count 9 JS files at present: admin, captcha, editor, mediadmin, register, script, script-sha1, suggest, topic. That's not necessarily definitive, but what I've encountered so far in general use.
Quote
Does it make sense at all?
Oh, it makes sense, but I'm not seeing that as a problem, actually.

add_js_file would load the file, you'd be calling loadLanguage or loadPluginLanguage anyway from there. That means, basically, you'll still fall into the process I'm talking about. There's no reason why the same process couldn't also contain the JS language file, though if I understand you correctly, you're almost talking about having multiple language JS files too...

Either way, I see no reason why an editor couldn't cope with being able to edit either and load from the DB, caching the results. It only has to worry about caching if people are daft enough to edit files directly, but then all bets are off anyway.
Title: Re: Language editing inside Wedge
Post by: Nao on September 23rd, 2012, 11:27 PM
(Re: latest rev -- should it be notewarn for new PMs..? IIRC I set the semantics to use notenice because a PM isn't usually a 'warning' but something neutral or maybe even 'good news'... Or maybe I was just a bit too thorough with these three notice states..?)
Quote from Arantor on September 23rd, 2012, 04:30 PM
I count 9 JS files at present: admin, captcha, editor, mediadmin, register, script, script-sha1, suggest, topic. That's not necessarily definitive, but what I've encountered so far in general use.
If you wander into the media area, you'll also get media and zoom files.
Quote
add_js_file would load the file, you'd be calling loadLanguage or loadPluginLanguage anyway from there.
Hmm it's called in wedge_cache_js, only at the time of caching, otherwise it'd defeat the purpose of it all... add_js_file doesn't call wedge_cache_js, unless the file isn't found.
Quote
That means, basically, you'll still fall into the process I'm talking about. There's no reason why the same process couldn't also contain the JS language file, though if I understand you correctly, you're almost talking about having multiple language JS files too...
(Process?)
script-123.js -> script.js with English strings (or whatever the default language is)
script-french-123.js -> script.js with French strings (or whatever the alternative language is)
Quote
Either way, I see no reason why an editor couldn't cope with being able to edit either and load from the DB, caching the results. It only has to worry about caching if people are daft enough to edit files directly, but then all bets are off anyway.
I'm not sure we're on the same line here..?

Anyway, to make things short:
We need to determine whether a language file has been modified. The easiest/quickest is to do it at loadLanguage time, because it's rather likely that PHP internally loads file data when including it (i.e. it knows the file modification date and doesn't waste time retrieving it again). If a language file is found to be updated, then we can do a simple call to clean_cache('js') to empty the JS folder. This means all JS files will have to be regenerated, even for a language file that never gets used in them -- but it's acceptable since (1) it's unlikely an admin is going to update their language files a lot...?, (2) these files only get generated when needed, thus they won't be regenerated all at the same time, and the extra CPU time is split over several minutes or hours.

Now, there are some potential problems once again...
- Imagine that script.js uses @language ManageTopics, and the admin never actually visits the admin area... It means the script file never gets updated, even if the corresponding language file is updated. It's unlikely, but still...?
- I still can't think of a 'correct' way to cache file dates. I'm thinking of something like this... Please give me your opinion?
A $settings['lang_dates'] array that contains keys like 'Admin_french' (as indicated in my previous post), and a value representing the modified date. The array is only filled with entries that are found in the JS scripts (@language entries), in the language requested. At compile time, the current language files and user language are used to store this data in the array, then submit it via updateSettings(). When it comes to doing loadLanguage(), the language and filename are determined, and if (isset($settings['lang_dates'][$filename . '_' . $lang]), then compare the actual filedate with that value... This would ensure that only the files actually cached in JS scripts will be tested for modifications. (Of course, if a plugin/feature is removed that used to include a certain language file, that entry will be wasting space in the database for nothing... But I don't have a solution for this, either.)
Title: Re: Language editing inside Wedge
Post by: Arantor on September 23rd, 2012, 11:46 PM
Quote
(Re: latest rev -- should it be notewarn for new PMs..
You're not too thorough with the three states. The reason I don't use notenice in the menu is simply because the green doesn't sit well in the menu when it's the active menu item. To replicate, make sure you have an unread PM that won't be marked read when you're on action=pm, then go to action=pm, so there's at least a (1) still present, and you'll see that green just doesn't sit nicely because of a lack of contrast. But it's more prominent on the menu.
Quote
(Process?)
script-123.js -> script.js with English strings (or whatever the default language is)
script-french-123.js -> script.js with French strings (or whatever the alternative language is)
Then you started talking about @language also referring to multiple language files... which seems unnecessary to me.
Quote
We need to determine whether a language file has been modified.
No, we don't. That's the underlying point of what I've been saying. If you have the files as-is and the database containing deltas, you do not need to worry about file dates, because you just cache it whenever the database changes. You don't need to keep rechecking - you tell people to use the language editor, which deals with all that stuff.

Hence all my comments about people that are stupid enough to edit the files.
Quote
these files only get generated when needed, thus they won't be regenerated all at the same time, and the extra CPU time is split over several minutes or hours.
And if you know what files are changed you can safely just regenerate only the file that's changed, just when it's changed, job done.
Quote
- Imagine that script.js uses @language ManageTopics, and the admin never actually visits the admin area... It means the script file never gets updated, even if the corresponding language file is updated. It's unlikely, but still...?
I'm thoroughly confused. Are you saying that you want to push the entirety of a language file into a JS construct so they can be used client side?

A language set is approximately half an MB. Even allowing for savings for language constructs being more compact, you're still talking hundreds of KB that almost entirely aren't necessary!

To be brutally honest, the number of strings actually needed and referred to solely in JS is actually surprisingly small, assuming it can actually be pushed to JS (because not all of them can, for example, some of the strings used in the moderation filters area are added inline but the bulk of them won't be known until a DB query has been run)

Better would simply be to find all the strings that are actually needed from JS and make a single file out of those which can be included as standard along with script.js. Then you get to simplify a lot of the architecture.
Quote
- I still can't think of a 'correct' way to cache file dates. I'm thinking of something like this... Please give me your opinion?
It's completely unnecessary under everything I've been saying...
Title: Re: Language editing inside Wedge
Post by: Nao on September 24th, 2012, 12:18 AM
Quote from Arantor on September 23rd, 2012, 11:46 PM
You're not too thorough with the three states. The reason I don't use notenice in the menu is simply because the green doesn't sit well in the menu when it's the active menu item. To replicate, make sure you have an unread PM that won't be marked read when you're on action=pm, then go to action=pm, so there's at least a (1) still present, and you'll see that green just doesn't sit nicely because of a lack of contrast. But it's more prominent on the menu.
Okay, I'll trust you on this, I try not to 'unread' my PMs because it tends not to work... (I use convo mode, and if I unread the latest convo, it just redirects me back to that convo and thus cancels the unread state... Well duh! We need to do something about this... Redirecting to the homepage if we're on the PM first page?)
Quote
Then you started talking about @language also referring to multiple language files... which seems unnecessary to me.
@language has a list of language file radixes, not a list of languages...?
Quote
No, we don't. That's the underlying point of what I've been saying. If you have the files as-is and the database containing deltas, you do not need to worry about file dates, because you just cache it whenever the database changes.
Oh... Right! Then that would be an excellent solution, yes!
Is it already written? I suppose not? What do we do in the meantime? (Because the code is already written, I'd simply postponed the language caching test code...)
Quote
And if you know what files are changed you can safely just regenerate only the file that's changed, just when it's changed, job done.
We'll still need to go through all language files and search for @language then... (or search directly for $txt which'll take more CPU power.)
Quote
Quote
- Imagine that script.js uses @language ManageTopics, and the admin never actually visits the admin area... It means the script file never gets updated, even if the corresponding language file is updated. It's unlikely, but still...?
I'm thoroughly confused. Are you saying that you want to push the entirety of a language file into a JS construct so they can be used client side?
No, no...
Okay, here's the beginning of the script.js file:

Code: [Select]
@language index;

var
oThought,
weEditors = [],
_formSubmitted = false,

we_loading = $txt['ajax_in_progress'],
we_cancel = $txt['form_cancel'],
we_delete = $txt['delete'],
we_submit = $txt['form_submit'],
we_ok = $txt['ok'],

See what I mean..?
I'm simply going to go through all $txt insertions in HTML files, and move them to the scripts themselves. It'll be mostly helpful in editor.js, because it's on many pages... Heck, maybe we could even cache stuff like the button list etc :P
Quote
Better would simply be to find all the strings that are actually needed from JS and make a single file out of those which can be included as standard along with script.js. Then you get to simplify a lot of the architecture.
It's a solution, but I'm not fond of it...
Title: Re: Language editing inside Wedge
Post by: Arantor on September 24th, 2012, 12:35 AM
Quote
Oh... Right! Then that would be an excellent solution, yes!
Is it already written? I suppose not? What do we do in the meantime? (Because the code is already written, I'd simply postponed the language caching test code...)
No, this was more of a 'this is what I think we could do' and wanting comments before I wrote it.
Quote
See what I mean..?
Oh..... so you load the index file, pull the $txt entries out and inject that into the result file that then gets cached. Now it makes sense to me.

And now I understand the concern about language files and caching. There's one entire class of cases that's screwed up in passing - plugins. While we could check the core files for updates, and force those ones to be updated from the ACP, we can't do the same with plugins. There are no requirements about where plugins must keep their files, so unless we were to go through every folder of every active plugin, looking for .js files, just to look for @language directives, we can't do that.

The alternative is to simply nuke the cache when doing such an update and immediately rebuilding the most common (script, editor, topic) if affected and let the others rebuild on demand.
Quote
Okay, I'll trust you on this, I try not to 'unread' my PMs because it tends not to work... (I use convo mode, and if I unread the latest convo, it just redirects me back to that convo and thus cancels the unread state... Well duh! We need to do something about this... Redirecting to the homepage if we're on the PM first page?)
I'm a stuck in the mud who uses one at a time mode, so it's very clear. I can post a screenshot if that'd help?
Title: Re: Language editing inside Wedge
Post by: Nao on September 24th, 2012, 09:31 AM
Quote from Arantor on September 24th, 2012, 12:35 AM
No, this was more of a 'this is what I think we could do' and wanting comments before I wrote it.
Well, I suppose it's up to you... I'll just leave my code as is -- without any caching involved. For now, admins will have to empty their JS cache folder to update any language strings. I'd rather integrate cache flushing once your side of the code is done (or abandoned).
Quote
Oh..... so you load the index file, pull the $txt entries out and inject that into the result file that then gets cached. Now it makes sense to me.
:)
Quote
And now I understand the concern about language files and caching. There's one entire class of cases that's screwed up in passing - plugins. While we could check the core files for updates, and force those ones to be updated from the ACP, we can't do the same with plugins. There are no requirements about where plugins must keep their files, so unless we were to go through every folder of every active plugin, looking for .js files, just to look for @language directives, we can't do that.
I don't see it that way...?
If we see a plugin language file as updated, we empty the JS cache folder... (clean_cache('js'), once again.)
Quote
The alternative is to simply nuke the cache when doing such an update and immediately rebuilding the most common (script, editor, topic) if affected and let the others rebuild on demand.
They're always rebuilt on demand -- whether it be script.js or admin.js or whatever...
That's the beauty of it. It always does a file_exists before including the file. And it's fast enough that we don't have to worry about it. (CSS caching might be more of a problem, with the number of files, even though I've deleted a lot recently. It's still worth considering caching the filedates for a couple of minutes each time.)

Oh, slightly unrelated: I've decided to stop including common.css in all files. The only reason is that I forgot how Wess still supposedly supports the { } syntax, but not in mixed situations, so there's no point in adding a tabbed file to a bracketed file.

The alternative would be to do the bracket conversion on every file each at a time, rather than everything together, but back then I'd calculated it would be noticeably slower, or maybe not but I decided against it, I don't remember if it was a technical issue though...
Would you like me to look into it and maybe add supporting for mixing different CSS file types..? I don't think it's that necessary.
Quote
I'm a stuck in the mud who uses one at a time mode, so it's very clear. I can post a screenshot if that'd help?
No, it's okay it's okay...
Title: Re: Language editing inside Wedge
Post by: Arantor on September 24th, 2012, 12:38 PM
Quote
I don't see it that way...?
If we see a plugin language file as updated, we empty the JS cache folder... (clean_cache('js'), once again.)
There's actually two plugin related cases.

1) We update a plugin's own language files. We don't know if any given plugin has any JS files that are modified by those language files. But it would be conceivable that if we have a specific plugin being targeted, that we could brute-force search it for .js files and check those for @language directives. That said, it would theoretically be possible for a plugin to load another plugin's language files and use another plugin's JS, so that's not a reliable method.

2) We update a core language file which is used by multiple plugins of indeterminate use. Like above, short of checking every plugin, we can't really do a lot. Which means, we're either looking into caching the last update by language file as you suggested, or we clear the JS cache.

The problem with clearing the JS cache is that of course it means the next page load is going to hurt for someone. Would it not be better to purge the cache but *immediately* rebuild the files you know you're going to need for definite? The idea is that you'll hopefully move the performance hurt of regenerating cache files away from end users to the admin who is in the ACP and therefore should be expecting the occasional delay.

But we can definitely worry about that after the fact. I've got some stuff to do today so won't be able to look at this until tonight if I'm lucky.
Quote
Oh, slightly unrelated: I've decided to stop including common.css in all files.
Makes sense.
Quote
I don't think it's that necessary.
You're right, it's probably not that necessary.
Title: Re: Language editing inside Wedge
Post by: Nao on September 25th, 2012, 12:22 PM
Quote from Arantor on September 24th, 2012, 12:38 PM
2) We update a core language file which is used by multiple plugins of indeterminate use. Like above, short of checking every plugin, we can't really do a lot. Which means, we're either looking into caching the last update by language file as you suggested, or we clear the JS cache.
Clearing the JS cache really isn't much of a big deal... Packer is an efficient piece of software, it's only slow when you try using it in real time obviously. The only 'big' chunk of code in Wedge is the jQuery library, and even that is never going to be packed manually (that's why I added the <jquery_cache> tags around the local copy of it. So that it's ignored by Packer later.)
Quote
The problem with clearing the JS cache is that of course it means the next page load is going to hurt for someone.
One person is going to have a one-second delay in their page load... That's pretty much all. And that person is likely to be the admin, testing their own changes.
Quote
Would it not be better to purge the cache but *immediately* rebuild the files you know you're going to need for definite?
In any case you're going to rebuild the files while loading a page, so I don't see the point in rebuilding a selection of pages when the cache is purged, or when the next user requests the cached files...

Well, now I think the best way to deal with the cache purge for now would be to cache an array of key/values where key is the requested script file, and values is the list of language files needed. We can easily maintain that list when rebuilding the JS cache: if the file list has changed, then we update the database values. This saves us: (1) the need to go through the original JS files to retrieve a list of language files to test (this way, if we want to test whether a file needs to be rebuilt, we can directly test the JS filedate and then the language filedates and take the most recent of all.) (2) the need to search for 'script-french-1234.js.gz' if we know that a particular file doesn't have language entries, in which case we can directly look for 'script-1234.js.gz'.

BTW, my code only supports loadLanguage for now... You'll have to build loadPluginLanguage into it if you want. :)
Title: Re: Language editing inside Wedge
Post by: Arantor on September 25th, 2012, 02:24 PM
Well, there's one very large problem to contend with: what happens if a theme (not a skin) decides to have its own language files?

There are all sorts of funky things tangled in the bowels of the code, such as the fact that themes can also have other themes as a dependent (so you can create one theme with a bunch of new code, and then create several full themes that depend on *that* theme)

I can fully see cases where people might push out their own actual theme, with their own templates and their own strings. For example, most of DzinerStudio's themes have a different board index layout to the standard, which also requires additional strings. Which can also be edited and will require caching.

Though I'm tempted to remove base_theme support, I don't believe anyone except Bloc ever used it and even then I'm not entirely sure about that. But I don't see that we can remove actual support for alternate themes in general.

But that makes caching potentially complicated, especially if people can choose between different themes as such, because even the JS caches might be different - if a theme declares its own index.english.php (which it is perfectly entitled to do) or replace any strings from index within its own ThemeStrings language file, that will need to be factored in, which means potentially you need to include the theme id in the cached filename.
Title: Re: Language editing inside Wedge
Post by: Nao on September 25th, 2012, 03:55 PM
Quote from Arantor on September 25th, 2012, 02:24 PM
Well, there's one very large problem to contend with: what happens if a theme (not a skin) decides to have its own language files?
Well, it goes through loadLanguage all the same, doesn't it..? And loadLanguage is the perfect place to monitor changes... Heck, I just checked -- it even does a file_exists on any language file you're going to load, meaning that the file stats are cached from that point on, meaning you can get the filemtime for free, meaning doing a check there is a no-brainer.
Of course it means that if you have a special JS file like JS.english.php that you only load for JS caching, it'll never be seen as out of date. Thus we could have some additional cache check somewhere else... (e.g. in add_js_file)
Quote
Though I'm tempted to remove base_theme support, I don't believe anyone except Bloc ever used it and even then I'm not entirely sure about that. But I don't see that we can remove actual support for alternate themes in general.
I don't know what base_theme is, but what I know is that I've never delved into alternate themes, had a few back in the day but now it's default only... I wouldn't know where it slows Wedge down, really. (Although there's quite a lot of code for alternate theme language files in loadLanguage...)
Quote
that will need to be factored in, which means potentially you need to include the theme id in the cached filename.
Hmm, yeah, I guess so...
Wess/Subs-Cache aren't tested for alternate themes anyway. I mean, it's pretty much as if theme support was broken/removed for me. For instance, if you have a 'Wine' folder in your alternate theme, the CSS cache will much certainly save its files in the same folders as the default theme's...

Anyway.
Title: Re: Language editing inside Wedge
Post by: Arantor on September 25th, 2012, 06:09 PM
Quote
Well, it goes through loadLanguage all the same, doesn't it..?
It does, but it's complicated. Basically, loadLanguage goes through a list of places to check and *stops* when it hits the first.
Quote
I don't know what base_theme is, but what I know is that I've never delved into alternate themes
Normally, in SMF, a theme says that it starts from its own folder and defers to the default theme in the event of templates and language files it doesn't have. base_theme is where a theme says it is based on another theme, so it attempts to load its own templates/language files, then defer to base_theme for templates and language files, *then* defer to the default theme.

base_theme strikes me as something not really needed any more, and I have no objections to removing it.

However, the question of multiple physical themes is a difficult one, do we actually need to leave this in (and check it works)? It depends, really, on how flexible CSS can be and whether or not additional layout markup would be needed - if new markup is required, you're in either plugin or theme territory.
Title: Re: Language editing inside Wedge
Post by: Nao on September 25th, 2012, 10:28 PM
So if it were up to you. When it comes to js language caching, what would you store in the db and where in the code would you check on updates being needed or not?
Title: Re: Language editing inside Wedge
Post by: Arantor on September 25th, 2012, 10:57 PM
It sort of IS up to me :P I sort of need to go through all of it mentally, so let's recap the whole thing end to end.

Ideally, I don't want anything in the DB that doesn't need to be there. The DB at a minimum needs to be able to contain what is different from the physical files, which means the DB needs to store enough to be able to work that out - which means, I guess: theme id, language file root, language of string, changed string. (Perhaps an additional column to indicate it is an array and should be unserialized on reading)

Then it's queried as necessary from loadLanguage.

From a cache perspective, we will likely need to recache language files, and to that end, I'm thinking we'll end up playing cache using the theme id, plus the language root, plus language - essentially caching something like '1-index-english' as the resultant content of that language file.

To me it seems that JS caching only needs to call on that combination (theme + language root + language) to find the file and then you have far fewer combinations. Clearing the cache when editing strings means the language cache, then the JS cache will be rebuilt. There should be no circumstance where files are edited manually and if users do do something stupid like that, it's going to be their own fault that it doesn't work properly. You don't have to worry about users that don't follow instructions - because they won't actively break anything by editing strings, it just won't propagate their changes.

That's for the worst case scenario of having multiple themes where the theme will also have to provide language strings itself, and we have to be able to make a distinction between two themes for the purposes of language files. It means you get a set of JS files per language per theme.

I'm just not seeing the need to store any cache values, unless there's something I'm missing...
Title: Re: Language editing inside Wedge
Post by: Nao on September 28th, 2012, 11:52 PM
Just to keep you posted...

- Fixed a surprising bug where strpos($del, $latest_date) would return false even if $latest_date was in $del... Because it was an integer. I thought PHP did some typecasting before a strpos, but apparently not...! So it means that this caching feature was pretty much broken since the beginning...
- The code doesn't (yet???) support (or care for) other themes. I figure you'll want to add support yourself :)
- Finished implementing per-language caching, i.e. generating 'script-french' and 'script' files depending on your language.
- Added support for plugin language files.
- Added a $settings['js_lang'] array. It is serialized in a similar fashion to $settings['registered_hooks'], which I'm not super-happy about, but it's still fast enough I guess. 'js_lang' is unserialized to an array of serialized arrays, so basically it looks like this in the end: array('script-' => array('index, Arantor:ThemeSelector:SkinSelector')) (<== SkinSelector being the language file inside the ThemeSelector plugin.)
- 'js_lang' is supposed to help in two ways: (1) if the script ID is missing from the array, it means there are no languages to be included, so we can directly include 'script-1234.js.gz' instead of adding the language string. (2) Supposedly, when we do loadLanguage, we can first test for file changes (I'm not recording this yet...), then if changed, go through js_lang and remove all JS files that use our language file.

As indicated in an earlier post, the main issue with js_lang is that it makes it impossible to test for changes to a file like 'JS.french.php' if the file is never loaded otherwise. I'd recommend maintaining a database of changed files somewhere, and then checking for changes before doing the wedge_cache_js test/call.
So... What do we do?
Title: Re: Language editing inside Wedge
Post by: Arantor on September 29th, 2012, 12:53 AM
Quote
- The code doesn't (yet???) support (or care for) other themes. I figure you'll want to add support yourself :)
Well, this is sort of why I brought the issue up. Multiple theme support is tricky because by definition you need to be able to override what's in the default theme. loadLanguage does this just fine, loadPluginLanguage doesn't care (by design).
Quote
Supposedly, when we do loadLanguage, we can first test for file changes
If at any point we are testing for file changes of the language files, the whole exercise becomes a waste of time. The whole point is to NEVER change the actual language files, which is why I'm so adamant that if the files do get changed, we should NOT lift a finger to help it along, under ANY circumstances.

Unless you're testing for changes of the resultant JS in some fashion but even then it's all down to things being in the DB or not, nothing more.
Title: Re: Language editing inside Wedge
Post by: Nao on September 29th, 2012, 11:34 AM
When I say 'test for file changes', I obviously say 'call filemdate on the file and compare it with $latest_date'... Should I re-explain everything again..?? :-/
Posted: September 29th, 2012, 07:59 AM

Also -- my bad: script files take the current theme into account... The theme's folder name is included in the file ID.
Posted: September 29th, 2012, 11:22 AM

The only 'way' I can see to quickly test whether we should check language file dates is...

if (filemtime($theme['theme_dir'] . '/languages/.') > $settings['latest_lang'])

('latest_lang' being an hypothetical variable where we store the most recent language file date.)

This could be done in Load.php at some point. If the language folder is updated, then we can either: (1) clean_cache('js') entirely (wanna say, *who minds*), (2) go through all files and test for their dates etc... (booooring!), (3) clean_cache('js') but only on files that use $txt strings in them. (A bit boring, too, especially since I'm going to be using $txt everywhere at that point...)

Two problems remain:
1/ should we store a latest_lang variable for each and every single language folder available to us..?
2/ how do we keep track of language files for plugins..?

Welcoming ideas, as always... :) Especially since this is probably the last remaining step before I can commit my work. (Which is already 3 times bigger than the original working code from a few days ago :P)
Title: Re: Language editing inside Wedge
Post by: PantsManUK on September 29th, 2012, 11:56 AM
Quote from Arantor on September 29th, 2012, 12:53 AM
If at any point we are testing for file changes of the language files, the whole exercise becomes a waste of time. The whole point is to NEVER change the actual language files, which is why I'm so adamant that if the files do get changed, we should NOT lift a finger to help it along, under ANY circumstances.
Interested observer here... Are you saying that for the *ENTIRE* lifetime of the product (however long that's going to be) you're *NEVER* going to add new strings to the base installation, because that's what your quoted text seems to suggest...?

My feeling (putting on a coders hat) is that you need to at least be prepared for the possibility that your base language files may need to be changed (during an upgrade), and have mechanisms in place to allow for that (remake the cache, whatever), even if you don't publicise that fact.
Title: Re: Language editing inside Wedge
Post by: Nao on September 29th, 2012, 12:06 PM
Quote from PantsManUK on September 29th, 2012, 11:56 AM
Interested observer here... Are you saying that for the *ENTIRE* lifetime of the product (however long that's going to be) you're *NEVER* going to add new strings to the base installation, because that's what your quoted text seems to suggest...?
No... I think what Pete means, is that since Wedge is going to be provided on the basis that "new updates are simply by dropping the updated files", the goal is for users not to have to modify their files manually.
So, you should understand this as: we're going to update the language files -- and we're going to be the only ones who can do it.

I'm guessing if you want to add your own language strings, manually and all, you can always use Custom.language.php... (Which I'm not even sure is a feature in Wedge?!)

Regarding the cache, so... I've finished implementing the filemdate check, and unfortunately it's not as solid as I thought. It should work properly on a Linux server, but when it comes to my Wamp install, my last modification date for the languages folder dates back to my last SVN update on it, rather than my last file change inside the folder... So it means it's going to work on a regular install, but not on my local install. Bugger! Now how do I deal with this... Is it a Windows bug? The php.net page on filemtime doesn't mention it...
Title: Re: Language editing inside Wedge
Post by: PantsManUK on September 29th, 2012, 12:38 PM
Quote from Nao on September 29th, 2012, 12:06 PM
No... I think what Pete means, is that since Wedge is going to be provided on the basis that "new updates are simply by dropping the updated files", the goal is for users not to have to modify their files manually.
Oh, I'm all for '"Users" DO NOT EVER need to modify files in Wedge directly' (and I applaud both your attempts to ensure this) *but* I'm concerned you're going to code yourselves (or more importantly, plugin devs[1]) into a corner that you're not going to get out of in a hurry.
 1. because nothing will kill off the project quicker than no supporting devs...
Title: Re: Language editing inside Wedge
Post by: Nao on September 29th, 2012, 01:09 PM
Before we start worrying about no supporting devs to author plugins, we should worry about the fact that no other dev has offered their help for the core dev at this point... :p

Not that it bothers me, though. But some days, I'd certainly like for emanuele to join us... He reminds me of my enthusiasm when we started work on Wedge ;) Plus he does some good and nicely formatted code. (Following SMF standards is important to me.) But right now he's SMF 2.1's main contributor so I'm not even going to bother asking...

So, I'm thinking of committing my language code 'as is'... (1) it's an alpha, (2) if you really need to update your script files and it doesn't work (e.g. you're working locally or your server's filesystem disabled last modified dates on folders) then you can always just empty the js cache folder manually, (3) gosh, it's an alpha! If Pete implements language deltas, then we'll be fine by the next beta!
Title: Re: Language editing inside Wedge
Post by: Arantor on September 29th, 2012, 03:01 PM
Quote
When I say 'test for file changes', I obviously say 'call filemdate on the file and compare it with $latest_date'... Should I re-explain everything again..?? :-/
This brings me back to the ACP. If you update it in the ACP, it clears the cache, job is done, I continue to fail to understand why checking the dates is relevant.

Install a new language pack, nothing changes - it just becomes a new set of cached files. Change the default language, clear the cache. Update the installation, clear the cache (good practice in any situation)

I still do not understand why there is a need to check every single page load, because there is NOT a need to do this, nor is there a need to keep track of which files are changed. As long as the language editor flushes the cache, the job is done automatically.

This also means you don't have to worry about language files for plugins because if you clean out the cache while you're doing it, they're automatically taken care of. We can implement a facility to clear the relevant bits of the cache by checking files, if necessary, but that's *still* preferable to tracking and over complicating things.

I do not see this being a problem for plugin authors, which is why I've been writing plugins to make sure of this.
Title: Re: Language editing inside Wedge
Post by: Nao on September 29th, 2012, 03:57 PM
As I said before. This file date business is temp stuff. I'm planning to get rid of it as soon as you implement language file caching.
But until this is done, I have to do it... And checking for the file date of two folders (one when default theme is used) once per page load is quite fast really... The only issue being with windows and still.
Title: Re: Language editing inside Wedge
Post by: Arantor on September 29th, 2012, 07:29 PM
*nods* Now I understand.

Getting back to the above, it would be drastically simpler to implement if base_theme were removed. It isn't a facility that is heavily used so I see no reason not to do that (and it would make all language loading slightly faster), are there any objections to removing it?
Title: Re: Language editing inside Wedge
Post by: Nao on September 29th, 2012, 10:51 PM
Would it really make things simpler...? From what I can see, it would shorter one line by a few dozen bytes, that's all. Also, loadLanguage is only (a bit) slower when base_theme_dir is enabled, if you're not using the default theme. Because it comes after it in terms of priority...

So, basically, I think these would be the wrong reasons for removing support.
The good reasons would be: (1) I don't know a thing about this feature and don't even know if it's documented somewhere, (2) if I don't know about it and it's probably not documented, considering that whenever I need something implemented I do it myself, I guess I myself see no reason to support something that I don't either need or know about...

Really, do as you please!
Title: Re: Language editing inside Wedge
Post by: Arantor on September 29th, 2012, 11:25 PM
Hell yes.

Let me explain the problem. As it stands right now, there are 3 separate places where a theme could load its language files from: current theme, base_theme (if current theme depends on another theme), default theme.

Trouble is, when a theme declares itself to be dependent on a base theme, it only stores the folder for it, not the actual id. Well, I need the actual id to be able to make this work. I could go through and hack in support so that any given theme also tracks the id of the theme on which it is based.
Quote
So, basically, I think these would be the wrong reasons for removing support.
Well... it's not a good reason but it's not a bad one either, as such.

1) It is fully undocumented.
2) I don't know anyone that used it for certain. It's possible Bloc used it but if he did, he's the only one that ever did. Everyone else just included their own templates and whatnot.
3) I'm not even sure it actually works properly in SMF, let alone Wedge.
Title: Re: Language editing inside Wedge
Post by: Arantor on February 18th, 2013, 06:05 AM
I have a working version of this stuff. At least, I have a cut down working version of this stuff.

I have it able to load from cache, rebuild cache and query the DB in the process, so all the mainline language files are built appropriately. I have merely commented the old code out for now.

There's not exactly caveats to this, but what I have got thus far works by recording the theme id (not skins, since skins don't have their own language files), plus language and whatnot. It shouldn't matter whether a theme is pulling from different places or not, all that matters is that for a given theme (whether it has its own language files or not), a given thing is going to get loaded, and whatever the result of that, that's what's going to be extended by the DB check and then saved.

This does mean that, currently, it *is* stored per theme. Whether that will change, I don't know. I can always assign 0 to force it for all themes and juggle the code. I don't know yet, I just implemented the base functionality and I'm not entirely sure where this is going yet.

It's not battle tested, but it seems to work (including respecting the DB content). It also does not yet do plugins, but they're going to be special anyway haha.

I don't know also how this will affect JS caching, or whether editing a file means just flushing all the JS cached files of the same language, that would be the easiest, if not the cleanest.
Posted: February 18th, 2013, 03:49 AM

Also... how come we never had agreement_french.txt anywhere in SVN that I can find? SMF supported multiple languages for the agreement for ages, not that most people really understood that it worked :/

Anyway, making a note that I'm moving it to a proper language file now, and given its name I'm giving it its own language file. But unlike the original, I'm going to be running the preparser on it. It gets shoved through parse_bbc anyway but I want to store in the language files as a directly parseable string for bbc purposes. (And I'll be excluding it from the main language editor for fairly obvious reasons; there's a dedicated UI for it, and as such I want to reuse that rather than anything else, plus there I can set up the bbc editing for it since so many people do not realise that it actually gets bbc parsed.)
Posted: February 18th, 2013, 04:18 AM

Also, the table's PK should be PRIMARY KEY (id_theme, id_lang, lang_file, lang_var, lang_key) not what's currently in SVN. I'll commit when I submit everything else related to changing the registration form code.
Title: Re: Language editing inside Wedge
Post by: Nao on February 18th, 2013, 10:33 PM
Okay... I don't really understand what you've been doing with language caching exactly... Not sure what it means for arrays like number_context things, etc...

What I know is that I'm getting tons of errors (hundreds per page load, which isn't handy to fix anything!) after installing the new table. And tons of language strings (not all?!) are missing on screen...
Can you tell me what to do to make it behave..?

Did you do any benchmarking? What are the performance gains exactly...?

Last time I checked, caching language files only saved a few milliseconds per page load...
Posted: February 18th, 2013, 10:19 PM

Okay, fixed by emptying my /cache folder manually... (Phew.)
Posted: February 18th, 2013, 10:24 PM
Quote from Arantor on February 18th, 2013, 06:05 AM
This does mean that, currently, it *is* stored per theme. Whether that will change, I don't know. I can always assign 0 to force it for all themes and juggle the code. I don't know yet, I just implemented the base functionality and I'm not entirely sure where this is going yet.
It all seems very complicated to me... Still wondering if it was worth all that work..?
Quote
I don't know also how this will affect JS caching, or whether editing a file means just flushing all the JS cached files of the same language, that would be the easiest, if not the cleanest.
Even JS/CSS caching is far from being a full implementation. I mean, I don't plan to "leave it this way" forever. I'd like to cache the list of files (skins, skeletons and JS) into the database somewhere, and only check whether there are any changes from time to time... But it's always the same thing: what happens when the admin uploads a new file? Do they have to manually go to the admin and empty some cache? That sucks...
Quote
Also... how come we never had agreement_french.txt anywhere in SVN that I can find? SMF supported multiple languages for the agreement for ages, not that most people really understood that it worked :/
It never was in there because I never was translated it. It was translated, though... But I don't know its quality. Attaching the file. I'll read through it another day... There was a discussion about it 2 years ago about updating its translation so it was probably bad to begin with (like easily 10% of the SMF French translation), but I'd already stepped down from my French translation manager position, so didn't care about it at all.
Quote
Also, the table's PK should be PRIMARY KEY (id_theme, id_lang, lang_file, lang_var, lang_key) not what's currently in SVN. I'll commit when I submit everything else related to changing the registration form code.
Speaking of primary keys... I remember you added an entry recently to another table, custom fields I think, and noticed you didn't add a key for it although you did some extra ORDER BY queries on it... Was it voluntary, or something you forgot?
Title: Re: Language editing inside Wedge
Post by: Arantor on February 18th, 2013, 10:43 PM
Quote
Okay... I don't really understand what you've been doing with language caching exactly... Not sure what it means for arrays like number_context things, etc...
Everything still seems to work with respect to number_context anyway ;)

OK, here's the deal. We have a mostly functional language editor in the admin panel. But it relies on directly editing language files which is a bad idea, not just from a security standpoint but it's not entirely survivable through upgrades and so on.

For months I've wanted to replace it with it being held in the DB, so that users can change it with impunity, without having to worry about the language files being changed in permissions. It's friendlier to the user too. (Or at least it will be once I finish the rest of it.)
Quote
Did you do any benchmarking? What are the performance gains exactly...?
I rarely, if ever, do anything primarily for performance. Performance is not my greatest concern, if I'm honest. This one is primarily a usability/security feature. The caching changes are to preserve existing performance without compromising the benefits of having the language changes table, since the hit of having another query every call to loadLanguage would be huge. Or even another query per page (to scoop up everything) wouldn't be much better.

The caching is to prevent DB queries, not to deal with perceived slowdowns of loading files. From a straight performance standpoint, once a cache file is actually built it should be near enough the same performance as loading language files was before. There is some slight overhead for a given file, but that's offset by the fact that it will invariably be only checking for one file and rebuilding the cache if it does not exist.

Already there is an advantage of this - the new registration agreement area, which no longer requires changing agreement.txt (or agreement_french.txt as was, which also had side issues potentially related to incorrect encodings) is now fully editable from the admin panel without having to navigate FTP permissions.
Quote
It all seems very complicated to me... Still wondering if it was worth all that work..?
Oh yes. This is a means to an end, not an end in itself. Consider the updated registration agreement page. While that was technically possible before (the editor aspect), the file permissions part wasn't.

It also simplifies language pack distribution too by only having one folder to put everything in.
Quote
But it's always the same thing: what happens when the admin uploads a new file? Do they have to manually go to the admin and empty some cache? That sucks...
No, it doesn't.

When, exactly, is the admin going to upload new language files? I will say it again: if they're going to manually mess with the files, that's their lookout. There will be no need for them to mess with the files directly as there will be a better interface available from the ACP which will deal with the caching aspect itself.
Quote
It never was in there because I never was translated it. It was translated, though... But I don't know its quality. Attaching the file. I'll read through it another day... There was a discussion about it 2 years ago about updating its translation so it was probably bad to begin with (like easily 10% of the SMF French translation), but I'd already stepped down from my French translation manager position, so didn't care about it at all.
Well, if it's suitable from your perspective, it just needs to preparsecode'd (or do as I did, post it into a post, then copy the content back out the DB after) and put into Agreement.french.php.
Quote
Speaking of primary keys... I remember you added an entry recently to another table, custom fields I think, and noticed you didn't add a key for it although you did some extra ORDER BY queries on it... Was it voluntary, or something you forgot?
I didn't add a key on it (and yes, it was the custom fields table) because I couldn't validate that adding an index would make it any faster. The field is simply for ordering the entries of custom fields. A certain amount of that is cached elsewhere inside the core anyway meaning that adding an index is almost counter productive.
Title: Re: Language editing inside Wedge
Post by: MultiformeIngegno on February 22nd, 2013, 01:04 PM
Seems great! :) But what happens when there's an upgrade (with new language strings or some existing updated) and the admin modified some lang strings? Will the system replace the edits of the admin with the new updated strings or leave the edits and just update the other strings not touched by the admin?
Title: Re: Language editing inside Wedge
Post by: Arantor on February 22nd, 2013, 03:07 PM
Modified language strings would actually be preserved entirely under the current setup. Modified strings are stored entirely outside of the languages area.

We could nudge the admin, I guess. It's something to think about in the upgrader.
Title: Re: Language editing inside Wedge
Post by: Arantor on February 22nd, 2013, 09:06 PM
Actually, there is one way it could be used to optimise performance, but it's going to take me a little while to figure out how exactly to do it without more breakages ;)

Right now we load English as a fallback. What occurs to me is that we could load English, load the other language, then the DB query, then cache the result so we wouldn't have duplicate loads for non English languages.

It would also mean that the English UK pack would simply be the changes from the default, and we could expand upon that by having other language packs be able to declare a fallback as well, e.g. Portuguese (Brazil) vs Portuguese (Portugal) if that makes sense.

Is complicated, but should make it possible.
Title: Re: Language editing inside Wedge
Post by: Arantor on February 24th, 2013, 03:09 AM
Meanwhile, here's a picture of a fish.

Or not. ;)

WIP of the new language editing area. Replaces that godawful dropdown we all know and despise that some of us know and have come to hate. For those of you who didn't know about it, fire up SMF, Admin > Languages > select a language > play with the dropdown in the corner.
Title: Re: Language editing inside Wedge
Post by: Arantor on February 24th, 2013, 09:57 PM
OK, another update. Probably the last for the evening because I'm at the stage where I'm not entirely sure what to do with it, so throwing it open a bit.

First up, the more finalised list of templates. Aside from the fact everything is now linked to where it should be, the things of note are that I moved Modlog from main to admin panel and gave it a better description, plus I created a block for 'configurable language crap that isn't in this area', namely the email templates and the agreements. I have no qualms about them being outside the language editor, they are where they are for a very good and specific reason, namely that they have a bunch of other specific behaviours that are attached to them that have no place being here.[1]

Then we have the actual editing page itself. You may remember SMF's was a two-pane affair. I didn't like it, still don't like it, and I've started on a half-assed replacement but I'm really not that fond of THAT either. It's just marginally better to me than what we had before. Second screenshot shows the main index, part way down - mostly so it shows the one actual entry I have that does have both a master value and a replaced value (namely 'admin', which is the main $txt['admin'] menu item)

I'm open to any suggestions of what to do with it because I got nothing right now!

 1. Email templates has all the stuff related to variable injections, agreements do not currently but will have the whole 'force reagreement' facility. Either way it's a ton of stuff that is unique to those areas and would get in the way of the rest of the stuff.
Title: Re: Language editing inside Wedge
Post by: Nao on March 1st, 2013, 12:55 PM
'kay, a few comments on the new area...

- Looks nice :)

- License agreement editor shows 3 line breaks for French even though I only added two br tags. The difference with the English file is, I actually inserted a linebreak before adding the two br tags, so I guess Wedge is converting linebreaks to br's... Is it as intended..? Might as well remove the br's and do two line-breaks per paragraph in the language file.

- Finding the wysiwyg editor in that area is amusing... ;) Cursor over the scrollbar is the editor cursor though, not the regular cursor.. (??) And I just realized it's the same everywhere.. Argh?! Doesn't that bother you..??

- 'Master values' pages could use more styling... I know it's a WIP, but I'm thinking 'Master values' should be in italics.

- Also, err... I'm assuming eventually one will be able to edit these, right..? :^^;:

- Suggesting that the E-mail templates links point to the page for the current language being edited. (It's already in the URL -- lid=french/english.)

- Search engine doesn't work, it redirects to the admin homepage. Reminder: I have pretty URLs enabled... Maybe related. Even the variable separators have ampersands instead of semi-colons... (??!!)

- I forgot... Do you still plan to allow for caching English + French next, or something...? I don't remember.
Title: Re: Language editing inside Wedge
Post by: Arantor on March 1st, 2013, 02:21 PM
Quote
- License agreement editor shows 3 line breaks for French even though I only added two br tags. The difference with the English file is, I actually inserted a linebreak before adding the two br tags, so I guess Wedge is converting linebreaks to br's... Is it as intended..?
The underlying editing code (i.e. the likes of Class-Editor itself, plus stuff like un_preparsecode, un_htmlspecialchars and friends) expects never to deal with line breaks. No post has line breaks in it, because preparsecode strips them and converts to <br>s (and this needs to stay, too)
Quote
- Finding the wysiwyg editor in that area is amusing...
Well, there's no reason not to have it, if the WYSIWYG editor is reliable. I did not do any tests specifically with WYSIWYG, I merely added the correct code and in theory it should work *shrug*
Quote
Doesn't that bother you..??
It would if I ever, ever used the WYSIWYG editor... :whistle:
Quote
- 'Master values' pages could use more styling... I know it's a WIP, but I'm thinking 'Master values' should be in italics.
Well, there's master and current and both could easily be made italic. It's the sort of thing that I got fed up with it for, I didn't really know what to do with the UI going forward.
Quote
- Also, err... I'm assuming eventually one will be able to edit these, right..? :^^;:
Um, yes :lol:
Quote
- Suggesting that the E-mail templates links point to the page for the current language being edited. (It's already in the URL -- lid=french/english.)
What I meant to do was add another layer of linktree to point to the specific template.
Quote
- Search engine doesn't work, it redirects to the admin homepage. Reminder: I have pretty URLs enabled... Maybe related. Even the variable separators have ampersands instead of semi-colons... (??!!)
As per r1961's changelog: "Also note the search function doesn't work yet either." ;) The reason for the form conversion nonsense is because the <form> is literally just <form> at the moment, so not content with the fact there's no backend to search, there's a broken front-end too! But as per r1961, it was because I'd been staring at the code for days that I was just blinded by what I wanted to do with it, you know?
Quote
- I forgot... Do you still plan to allow for caching English + French next, or something...? I don't remember.
It's the plan. It's just a plan at this stage though.
Title: Re: Language editing inside Wedge
Post by: Nao on March 1st, 2013, 04:48 PM
Quote from Arantor on March 1st, 2013, 02:21 PM
The underlying editing code (i.e. the likes of Class-Editor itself, plus stuff like un_preparsecode, un_htmlspecialchars and friends) expects never to deal with line breaks. No post has line breaks in it, because preparsecode strips them and converts to <br>s (and this needs to stay, too)
But apparently, here, it converts them...
Quote
Well, there's no reason not to have it, if the WYSIWYG editor is reliable. I did not do any tests specifically with WYSIWYG, I merely added the correct code and in theory it should work *shrug*
When I say Wysiwyg editor, I simply mean post box, with all smileys etc. I don't use Wysiwyg either, as you well know, so I'd hardly enable it by default... ;)

So, the extra line is in 'normal' mode.
Quote
Well, there's master and current and both could easily be made italic. It's the sort of thing that I got fed up with it for, I didn't really know what to do with the UI going forward.
I dunno, maybe having the Master/Current captions in italics, the actual strings in regular, and if the current version is different from master, show it in bold...? (Like in Firefox's hidden setting page ;))
Quote
What I meant to do was add another layer of linktree to point to the specific template.
Is there a problem in simply updating the link...? :unsure:
Quote
As per r1961's changelog: "Also note the search function doesn't work yet either." ;)
All right then. I tend to overlook the changelog, it reminds me too much that I have to finish translations... :lol:
Quote
Quote
- I forgot... Do you still plan to allow for caching English + French next, or something...? I don't remember.
It's the plan. It's just a plan at this stage though.
It's a bit strange to be in a situation where missing strings are again generating errors... :^^;: (e.g. EmailTemplates.)
Title: Re: Language editing inside Wedge
Post by: Arantor on March 1st, 2013, 05:00 PM
Quote
But apparently, here, it converts them...
textarea takes linebreaks as linebreaks, and the unpreparser converts brs to linebreaks too. There should be no linebreaks in the file.
Quote
When I say Wysiwyg editor, I simply mean post box, with all smileys etc. I don't use Wysiwyg either, as you well know, so I'd hardly enable it by default...
Nah, doesn't bother me. Don't really notice. Mostly related to the exciting and fun custom draggable thing we inherited.
Quote
I dunno, maybe having the Master/Current captions in italics, the actual strings in regular, and if the current version is different from master, show it in bold...?
If it has a current version, it's different from master ;)
Quote
Is there a problem in simply updating the link...?
Yes, it's the wrong thing to do. Each link is supposed to be one level of hierarchy, not repurposing a link that is already present.
Quote
It's a bit strange to be in a situation where missing strings are again generating errors... :^^;: (e.g. EmailTemplates.)
English should still be loaded first though... in all cases, even email templates.
Title: Re: Language editing inside Wedge
Post by: Nao on March 17th, 2013, 05:06 PM
Quote from Arantor on March 1st, 2013, 05:00 PM
Quote
But apparently, here, it converts them...
textarea takes linebreaks as linebreaks, and the unpreparser converts brs to linebreaks too. There should be no linebreaks in the file.
I'm only saying what I'm seeing... :-/
The French language files have linebreaks everywhere. It's just "how they're done". These linebreaks are converted to br's at runtime. It's also much more readable (and easier to manipulate for translators...?), if you ask me, so I'm thinking I'll just use linebreaks for the agreement, rather than br tags, and I'd strongly encourage you do the same on yours.
Quote from Arantor on March 1st, 2013, 05:00 PM
Quote
It's a bit strange to be in a situation where missing strings are again generating errors... :^^;: (e.g. EmailTemplates.)
English should still be loaded first though... in all cases, even email templates.
I didn't follow the recent developments, if any.
Did you end up restoring the 'fallback to English' feature in your cached versions..? Or will I still have empty strings instead of English when a French one isn't translated?

Also, I noticed that a change you made last night with $language broke the default language behavior in Wess, more specifically in how Subs-Cache generates a language string modifier in the filename. My local install has French by default, but since your update, my files are named 'script-m-french-1234.js.gz' instead of just 'script-m-1234.js.gz'... Should I look into it? I haven't yet finished reviewing your $language changes. Lots of things to do on my side... :-/
Title: Re: Language editing inside Wedge
Post by: Arantor on March 17th, 2013, 05:40 PM
Quote
The French language files have linebreaks everywhere. It's just "how they're done". These linebreaks are converted to br's at runtime. It's also much more readable (and easier to manipulate for translators...?), if you ask me, so I'm thinking I'll just use linebreaks for the agreement, rather than br tags, and I'd strongly encourage you do the same on yours.
WHY are they converted to brs at runtime? They're only converted as far as I can see through the textarea component. Other than that, they should NOT be there.
Quote
Did you end up restoring the 'fallback to English' feature in your cached versions..? Or will I still have empty strings instead of English when a French one isn't translated?
Not yet. It's on my todo list.
Quote
Also, I noticed that a change you made last night with $language broke the default language behavior in Wess, more specifically in how Subs-Cache generates a language string modifier in the filename. My local install has French by default
That is because you need to set $settings['language'] to french, or otherwise set the default language in the admin panel. Right now it's going to be falling back to English because you haven't got it set in the settings table. But your profile still has it set to French anyway which forces it to show you French even if the default is not.
Title: Re: Language editing inside Wedge
Post by: Nao on March 19th, 2013, 07:28 PM
Quote from Arantor on March 17th, 2013, 05:40 PM
WHY are they converted to brs at runtime? They're only converted as far as I can see through the textarea component. Other than that, they should NOT be there.
The agreement..?
Register.php
loadLanguage('Agreement');
$agreement = parse_bbc($txt['agreement']);
Or something to that effect...
So, it's converting newlines to br's. Thus, you have to choose between a HTML string (English version), or a BBC one (French version)... Which one do we get..? I vote for BBC.
Quote
That is because you need to set $settings['language'] to french, or otherwise set the default language in the admin panel.
Hadn't done it, indeed. ;)
Title: Re: Language editing inside Wedge
Post by: Arantor on March 19th, 2013, 07:45 PM
Quote
So, it's converting newlines to br's.
Um... no, it's not.

The registration page for French has each paragraph separated by two line breaks. Which is what's in Agreement.french.php. parse_bbc has precisely squat to do with dealing with line breaks. The extra line break is introduced by the editor component when it un_preparses the brs back into line breaks so there are now 3 line breaks.
Quote
Thus, you have to choose between a HTML string (English version), or a BBC one (French version)... Which one do we get..? I vote for BBC.
And if you look back at what I said, I explained how I generated the string I did. IT IS BBC. I copied it from the old agreement.txt file into a post, saved it, and extracted the raw content from the DB. preparsecode does the nl2br conversion, it always has done.

If you want the bbc version, look to the English one because that's the 'correct' one. Look at the source for every post - you will not find newlines in general bbcode, because that's how it's always been done. Even down to how the news item does its thing - the newline is the separator between items, with the actual item having had nl2br conversion.

And parse_bbc adheres - it does no such conversion of newlines to bbc. I am not entirely sure why it is done this way around but it is how it has always been.