Language editing inside Wedge

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #30, on September 29th, 2012, 11:34 AM »
When I say 'test for file changes', I obviously say 'call filemdate on the file and compare it with $latest_date'... Should I re-explain everything again..?? :-/
Posted: September 29th, 2012, 07:59 AM

Also -- my bad: script files take the current theme into account... The theme's folder name is included in the file ID.
Posted: September 29th, 2012, 11:22 AM

The only 'way' I can see to quickly test whether we should check language file dates is...

if (filemtime($theme['theme_dir'] . '/languages/.') > $settings['latest_lang'])

('latest_lang' being an hypothetical variable where we store the most recent language file date.)

This could be done in Load.php at some point. If the language folder is updated, then we can either: (1) clean_cache('js') entirely (wanna say, *who minds*), (2) go through all files and test for their dates etc... (booooring!), (3) clean_cache('js') but only on files that use $txt strings in them. (A bit boring, too, especially since I'm going to be using $txt everywhere at that point...)

Two problems remain:
1/ should we store a latest_lang variable for each and every single language folder available to us..?
2/ how do we keep track of language files for plugins..?

Welcoming ideas, as always... :) Especially since this is probably the last remaining step before I can commit my work. (Which is already 3 times bigger than the original working code from a few days ago :P)

PantsManUK

  • [me=PantsManUK]would dearly love to dump SMF 1.X at this juncture...[/me]
  • Posts: 174
Re: Language editing inside Wedge
« Reply #31, on September 29th, 2012, 11:56 AM »
Quote from Arantor on September 29th, 2012, 12:53 AM
If at any point we are testing for file changes of the language files, the whole exercise becomes a waste of time. The whole point is to NEVER change the actual language files, which is why I'm so adamant that if the files do get changed, we should NOT lift a finger to help it along, under ANY circumstances.
Interested observer here... Are you saying that for the *ENTIRE* lifetime of the product (however long that's going to be) you're *NEVER* going to add new strings to the base installation, because that's what your quoted text seems to suggest...?

My feeling (putting on a coders hat) is that you need to at least be prepared for the possibility that your base language files may need to be changed (during an upgrade), and have mechanisms in place to allow for that (remake the cache, whatever), even if you don't publicise that fact.
« What is this thing you hoomans call "Facebook"? »

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #32, on September 29th, 2012, 12:06 PM »
Quote from PantsManUK on September 29th, 2012, 11:56 AM
Interested observer here... Are you saying that for the *ENTIRE* lifetime of the product (however long that's going to be) you're *NEVER* going to add new strings to the base installation, because that's what your quoted text seems to suggest...?
No... I think what Pete means, is that since Wedge is going to be provided on the basis that "new updates are simply by dropping the updated files", the goal is for users not to have to modify their files manually.
So, you should understand this as: we're going to update the language files -- and we're going to be the only ones who can do it.

I'm guessing if you want to add your own language strings, manually and all, you can always use Custom.language.php... (Which I'm not even sure is a feature in Wedge?!)

Regarding the cache, so... I've finished implementing the filemdate check, and unfortunately it's not as solid as I thought. It should work properly on a Linux server, but when it comes to my Wamp install, my last modification date for the languages folder dates back to my last SVN update on it, rather than my last file change inside the folder... So it means it's going to work on a regular install, but not on my local install. Bugger! Now how do I deal with this... Is it a Windows bug? The php.net page on filemtime doesn't mention it...

PantsManUK

  • [me=PantsManUK]would dearly love to dump SMF 1.X at this juncture...[/me]
  • Posts: 174
Re: Language editing inside Wedge
« Reply #33, on September 29th, 2012, 12:38 PM »
Quote from Nao on September 29th, 2012, 12:06 PM
No... I think what Pete means, is that since Wedge is going to be provided on the basis that "new updates are simply by dropping the updated files", the goal is for users not to have to modify their files manually.
Oh, I'm all for '"Users" DO NOT EVER need to modify files in Wedge directly' (and I applaud both your attempts to ensure this) *but* I'm concerned you're going to code yourselves (or more importantly, plugin devs[1]) into a corner that you're not going to get out of in a hurry.
 1. because nothing will kill off the project quicker than no supporting devs...

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #34, on September 29th, 2012, 01:09 PM »
Before we start worrying about no supporting devs to author plugins, we should worry about the fact that no other dev has offered their help for the core dev at this point... :p

Not that it bothers me, though. But some days, I'd certainly like for emanuele to join us... He reminds me of my enthusiasm when we started work on Wedge ;) Plus he does some good and nicely formatted code. (Following SMF standards is important to me.) But right now he's SMF 2.1's main contributor so I'm not even going to bother asking...

So, I'm thinking of committing my language code 'as is'... (1) it's an alpha, (2) if you really need to update your script files and it doesn't work (e.g. you're working locally or your server's filesystem disabled last modified dates on folders) then you can always just empty the js cache folder manually, (3) gosh, it's an alpha! If Pete implements language deltas, then we'll be fine by the next beta!

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #35, on September 29th, 2012, 03:01 PM »
Quote
When I say 'test for file changes', I obviously say 'call filemdate on the file and compare it with $latest_date'... Should I re-explain everything again..?? :-/
This brings me back to the ACP. If you update it in the ACP, it clears the cache, job is done, I continue to fail to understand why checking the dates is relevant.

Install a new language pack, nothing changes - it just becomes a new set of cached files. Change the default language, clear the cache. Update the installation, clear the cache (good practice in any situation)

I still do not understand why there is a need to check every single page load, because there is NOT a need to do this, nor is there a need to keep track of which files are changed. As long as the language editor flushes the cache, the job is done automatically.

This also means you don't have to worry about language files for plugins because if you clean out the cache while you're doing it, they're automatically taken care of. We can implement a facility to clear the relevant bits of the cache by checking files, if necessary, but that's *still* preferable to tracking and over complicating things.

I do not see this being a problem for plugin authors, which is why I've been writing plugins to make sure of this.
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #36, on September 29th, 2012, 03:57 PM »
As I said before. This file date business is temp stuff. I'm planning to get rid of it as soon as you implement language file caching.
But until this is done, I have to do it... And checking for the file date of two folders (one when default theme is used) once per page load is quite fast really... The only issue being with windows and still.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #37, on September 29th, 2012, 07:29 PM »
*nods* Now I understand.

Getting back to the above, it would be drastically simpler to implement if base_theme were removed. It isn't a facility that is heavily used so I see no reason not to do that (and it would make all language loading slightly faster), are there any objections to removing it?

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #38, on September 29th, 2012, 10:51 PM »
Would it really make things simpler...? From what I can see, it would shorter one line by a few dozen bytes, that's all. Also, loadLanguage is only (a bit) slower when base_theme_dir is enabled, if you're not using the default theme. Because it comes after it in terms of priority...

So, basically, I think these would be the wrong reasons for removing support.
The good reasons would be: (1) I don't know a thing about this feature and don't even know if it's documented somewhere, (2) if I don't know about it and it's probably not documented, considering that whenever I need something implemented I do it myself, I guess I myself see no reason to support something that I don't either need or know about...

Really, do as you please!

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #39, on September 29th, 2012, 11:25 PM »
Hell yes.

Let me explain the problem. As it stands right now, there are 3 separate places where a theme could load its language files from: current theme, base_theme (if current theme depends on another theme), default theme.

Trouble is, when a theme declares itself to be dependent on a base theme, it only stores the folder for it, not the actual id. Well, I need the actual id to be able to make this work. I could go through and hack in support so that any given theme also tracks the id of the theme on which it is based.
Quote
So, basically, I think these would be the wrong reasons for removing support.
Well... it's not a good reason but it's not a bad one either, as such.

1) It is fully undocumented.
2) I don't know anyone that used it for certain. It's possible Bloc used it but if he did, he's the only one that ever did. Everyone else just included their own templates and whatnot.
3) I'm not even sure it actually works properly in SMF, let alone Wedge.
Re: Language editing inside Wedge
« Reply #40, on February 18th, 2013, 06:05 AM »
I have a working version of this stuff. At least, I have a cut down working version of this stuff.

I have it able to load from cache, rebuild cache and query the DB in the process, so all the mainline language files are built appropriately. I have merely commented the old code out for now.

There's not exactly caveats to this, but what I have got thus far works by recording the theme id (not skins, since skins don't have their own language files), plus language and whatnot. It shouldn't matter whether a theme is pulling from different places or not, all that matters is that for a given theme (whether it has its own language files or not), a given thing is going to get loaded, and whatever the result of that, that's what's going to be extended by the DB check and then saved.

This does mean that, currently, it *is* stored per theme. Whether that will change, I don't know. I can always assign 0 to force it for all themes and juggle the code. I don't know yet, I just implemented the base functionality and I'm not entirely sure where this is going yet.

It's not battle tested, but it seems to work (including respecting the DB content). It also does not yet do plugins, but they're going to be special anyway haha.

I don't know also how this will affect JS caching, or whether editing a file means just flushing all the JS cached files of the same language, that would be the easiest, if not the cleanest.
Posted: February 18th, 2013, 03:49 AM

Also... how come we never had agreement_french.txt anywhere in SVN that I can find? SMF supported multiple languages for the agreement for ages, not that most people really understood that it worked :/

Anyway, making a note that I'm moving it to a proper language file now, and given its name I'm giving it its own language file. But unlike the original, I'm going to be running the preparser on it. It gets shoved through parse_bbc anyway but I want to store in the language files as a directly parseable string for bbc purposes. (And I'll be excluding it from the main language editor for fairly obvious reasons; there's a dedicated UI for it, and as such I want to reuse that rather than anything else, plus there I can set up the bbc editing for it since so many people do not realise that it actually gets bbc parsed.)
Posted: February 18th, 2013, 04:18 AM

Also, the table's PK should be PRIMARY KEY (id_theme, id_lang, lang_file, lang_var, lang_key) not what's currently in SVN. I'll commit when I submit everything else related to changing the registration form code.

Nao

  • Dadman with a boy
  • Posts: 16,079
Re: Language editing inside Wedge
« Reply #41, on February 18th, 2013, 10:33 PM »
Okay... I don't really understand what you've been doing with language caching exactly... Not sure what it means for arrays like number_context things, etc...

What I know is that I'm getting tons of errors (hundreds per page load, which isn't handy to fix anything!) after installing the new table. And tons of language strings (not all?!) are missing on screen...
Can you tell me what to do to make it behave..?

Did you do any benchmarking? What are the performance gains exactly...?

Last time I checked, caching language files only saved a few milliseconds per page load...
Posted: February 18th, 2013, 10:19 PM

Okay, fixed by emptying my /cache folder manually... (Phew.)
Posted: February 18th, 2013, 10:24 PM
Quote from Arantor on February 18th, 2013, 06:05 AM
This does mean that, currently, it *is* stored per theme. Whether that will change, I don't know. I can always assign 0 to force it for all themes and juggle the code. I don't know yet, I just implemented the base functionality and I'm not entirely sure where this is going yet.
It all seems very complicated to me... Still wondering if it was worth all that work..?
Quote
I don't know also how this will affect JS caching, or whether editing a file means just flushing all the JS cached files of the same language, that would be the easiest, if not the cleanest.
Even JS/CSS caching is far from being a full implementation. I mean, I don't plan to "leave it this way" forever. I'd like to cache the list of files (skins, skeletons and JS) into the database somewhere, and only check whether there are any changes from time to time... But it's always the same thing: what happens when the admin uploads a new file? Do they have to manually go to the admin and empty some cache? That sucks...
Quote
Also... how come we never had agreement_french.txt anywhere in SVN that I can find? SMF supported multiple languages for the agreement for ages, not that most people really understood that it worked :/
It never was in there because I never was translated it. It was translated, though... But I don't know its quality. Attaching the file. I'll read through it another day... There was a discussion about it 2 years ago about updating its translation so it was probably bad to begin with (like easily 10% of the SMF French translation), but I'd already stepped down from my French translation manager position, so didn't care about it at all.
Quote
Also, the table's PK should be PRIMARY KEY (id_theme, id_lang, lang_file, lang_var, lang_key) not what's currently in SVN. I'll commit when I submit everything else related to changing the registration form code.
Speaking of primary keys... I remember you added an entry recently to another table, custom fields I think, and noticed you didn't add a key for it although you did some extra ORDER BY queries on it... Was it voluntary, or something you forgot?

 agreement.french.txt - 4.06 kB, downloaded 68 times.


Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #42, on February 18th, 2013, 10:43 PM »
Quote
Okay... I don't really understand what you've been doing with language caching exactly... Not sure what it means for arrays like number_context things, etc...
Everything still seems to work with respect to number_context anyway ;)

OK, here's the deal. We have a mostly functional language editor in the admin panel. But it relies on directly editing language files which is a bad idea, not just from a security standpoint but it's not entirely survivable through upgrades and so on.

For months I've wanted to replace it with it being held in the DB, so that users can change it with impunity, without having to worry about the language files being changed in permissions. It's friendlier to the user too. (Or at least it will be once I finish the rest of it.)
Quote
Did you do any benchmarking? What are the performance gains exactly...?
I rarely, if ever, do anything primarily for performance. Performance is not my greatest concern, if I'm honest. This one is primarily a usability/security feature. The caching changes are to preserve existing performance without compromising the benefits of having the language changes table, since the hit of having another query every call to loadLanguage would be huge. Or even another query per page (to scoop up everything) wouldn't be much better.

The caching is to prevent DB queries, not to deal with perceived slowdowns of loading files. From a straight performance standpoint, once a cache file is actually built it should be near enough the same performance as loading language files was before. There is some slight overhead for a given file, but that's offset by the fact that it will invariably be only checking for one file and rebuilding the cache if it does not exist.

Already there is an advantage of this - the new registration agreement area, which no longer requires changing agreement.txt (or agreement_french.txt as was, which also had side issues potentially related to incorrect encodings) is now fully editable from the admin panel without having to navigate FTP permissions.
Quote
It all seems very complicated to me... Still wondering if it was worth all that work..?
Oh yes. This is a means to an end, not an end in itself. Consider the updated registration agreement page. While that was technically possible before (the editor aspect), the file permissions part wasn't.

It also simplifies language pack distribution too by only having one folder to put everything in.
Quote
But it's always the same thing: what happens when the admin uploads a new file? Do they have to manually go to the admin and empty some cache? That sucks...
No, it doesn't.

When, exactly, is the admin going to upload new language files? I will say it again: if they're going to manually mess with the files, that's their lookout. There will be no need for them to mess with the files directly as there will be a better interface available from the ACP which will deal with the caching aspect itself.
Quote
It never was in there because I never was translated it. It was translated, though... But I don't know its quality. Attaching the file. I'll read through it another day... There was a discussion about it 2 years ago about updating its translation so it was probably bad to begin with (like easily 10% of the SMF French translation), but I'd already stepped down from my French translation manager position, so didn't care about it at all.
Well, if it's suitable from your perspective, it just needs to preparsecode'd (or do as I did, post it into a post, then copy the content back out the DB after) and put into Agreement.french.php.
Quote
Speaking of primary keys... I remember you added an entry recently to another table, custom fields I think, and noticed you didn't add a key for it although you did some extra ORDER BY queries on it... Was it voluntary, or something you forgot?
I didn't add a key on it (and yes, it was the custom fields table) because I couldn't validate that adding an index would make it any faster. The field is simply for ordering the entries of custom fields. A certain amount of that is cached elsewhere inside the core anyway meaning that adding an index is almost counter productive.

MultiformeIngegno

  • Posts: 1,337
Re: Language editing inside Wedge
« Reply #43, on February 22nd, 2013, 01:04 PM »
Seems great! :) But what happens when there's an upgrade (with new language strings or some existing updated) and the admin modified some lang strings? Will the system replace the edits of the admin with the new updated strings or leave the edits and just update the other strings not touched by the admin?

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Language editing inside Wedge
« Reply #44, on February 22nd, 2013, 03:07 PM »
Modified language strings would actually be preserved entirely under the current setup. Modified strings are stored entirely outside of the languages area.

We could nudge the admin, I guess. It's something to think about in the upgrader.