Nao

  • Dadman with a boy
  • Posts: 16,079
Development, full speed ahead!
« on August 6th, 2012, 06:38 PM »Last edited on August 7th, 2012, 12:43 PM
Hey everyone! Long time no blog. Nope, none of us are dead... We just had 'things' happen in real life, which slowed everything down, sometimes to a halt. I made sure to have at least a couple of commits out every week, just so I never lose focus on what I'm planning to do, but it's true that I've had a tendency to focus on smaller details rather than the big picture.

The main point is that we've often gone through the same story: deciding to implement a minor feature that would be cool, and then realizing, weeks later, that we didn't realize that 'minor' doesn't mean 'quick to implement'... And it went on, and on. As a result, I myself became wary of starting the heavy-duty work on a number of 'heavy' features that I'd like to have in Wedge from day one[1], and I think if we keep it that way, we'll always have a good excuse not to release Wedge. Not that we're in a hurry to do it -- but we've got to keep it real, and part of it means we have to keep it real for users.

So, the reason why I'm so reluctant to go public is mainly the fact that it's hard to update the database once it's installed somewhere. If you don't tell people to use the upgrade script etc, then you're likely to get annoying support questions...
So here's what I'm suggesting: storing a $settings['db_version'] variable (or a variation if there's already an equivalent), and set it to the SVN revision number corresponding to the lastest database structure change. So, at runtime we can simply, as soon as we load $settings, check for the variable, and if it's lower than the current required database version, load an extra file that will automatically adjust the database structure. e.g. if the db_version is 1652, and user installs version 1815, Wedge will execute all database (and file structure) modifier functions located between those rev numbers -- db_update_1664, db_update_1727, etc... It should be easy to calculate, really, and we can safely call them, and then update db_version to the latest database version available. Then it wouldn't be (so) hard to change the database, whether for us or for users.

What do you think...?

PS: bugger, another issue I just discovered... The selection is off by a few bytes when I click the Bold button in the bbc box, perhaps related to the Opera hack that I wrote for editor.js in both SMF and Wedge... Maybe they FINALLY, after years, fixed that bug...? :edit: Yes, they did, but they didn't fix all of it, so I'll have to keep an eye open...
 1. Custom boards, board types, custom groups, general privacy system, general tag system, attachments as media items, and so on...

Re: Speeding up development
« Reply #1, on August 6th, 2012, 07:13 PM »
Quote
So here's what I'm suggesting: storing a $settings['db_version'] variable (or a variation if there's already an equivalent),
This is what smfVersion was supposed to be used for.

I like the idea of the schema changes, not sure how practical it would really be in practice - or to do safely on an otherwise running install. I foresee lots of locking type issues attached especially on bigger installs.

Re: Speeding up development
« Reply #2, on August 6th, 2012, 09:57 PM »
So that'd be we_version..?
I'd like to use a SVN revision number for simplicity (for us!), since it'll be internal use anyway... (And I don't plan to do away with our SVN rev numbers, if you catch my drift :P)

I was thinking that we could disable the forum temporarily while upgrading. The script would only be run if the admin is online (and by the admin.)
There's bound to be a few minutes where things will go awry for everyone, which is logical considering one is updating their forum files manually by FTP... At some point there'll be a v1.0 file running alongside a v2.0 file, etc. Anyway, it's no biggie.
Re: Speeding up development
« Reply #3, on August 6th, 2012, 09:58 PM »
Quote from Nao on August 6th, 2012, 06:38 PM
PS: crap, another bug I just discovered... The selection is off by a few bytes when I click the Bold button in the bbc box, perhaps related to the Opera hack that I devised for editor.js... Maybe they FINALLY, after years, fixed that bug...?
Problem doesn't happen in Opera 12.01... So, conclusion: this is an Opera 12.50 bug. Dunno if I should report it (they never, ever bothered to look into my bug reports), or if they're already aware of the problem...

Re: Speeding up development
« Reply #4, on August 6th, 2012, 10:22 PM »
Quote
So that'd be we_version..?
Yes, though I didn't remember if we'd ditched it or not.
Quote
I was thinking that we could disable the forum temporarily while upgrading. The script would only be run if the admin is online (and by the admin.)
So they download the updated files, then figure out what changes have to be made if any, then apply it all at once... hmm.

Re: Speeding up development
« Reply #5, on August 6th, 2012, 11:15 PM »
We didn't ditch it, I think... It was probably renamed to 'weVersion' or 'we_version' or even 'weVer'... Can't remember.

No, they don't have to figure out anything, they just need to reupload everything...? It's slower, but it's doable by anyone ahah...



My public report...
http://my.opera.com/desktopteam/blog/show.dml/51546052?startidx=250#comment94192592

It's definitely a bug on their side, but it also means that Opera will be buggy in SMF 2.0.x!

Re: Speeding up development
« Reply #6, on August 6th, 2012, 11:31 PM »
That's the point, though, it really doesn't work like that in practice. WordPress does something similar, and I've seen enough of those go wrong in the past.

Re: Speeding up development
« Reply #7, on August 7th, 2012, 04:52 AM »
I like the basic premise that Nao proposes. If Wedge calls home when the Admin CP is loaded, it could get all the update information at that time including an indication that the database schema needs to be updated. The Admin could be asked if he wants to update and if so, the entire site is placed in maintenance mode whilst the update script(s) execute. This could possibly be direct server to server so no need for the admin to download the update and then FTP it to his site.
Or is that too simplistic a view? :)


Re: Speeding up development
« Reply #9, on August 7th, 2012, 12:37 PM »
Oh, I never suggested going with what Aeva Media used to do, i.e. downloading its site list automatically and installing it transparently for the admin...
Well, I used to want to do that, but Pete reminded me that it's unlikely it's going to work for the forum code itself, as there are so many ways to screw that up ;)

I'm just saying that we should do away with our 'big' features which, to be honest, I'm too scared to try and implement because they might take so long to do, and instead go with a 'light' Wedge, which is already great enough as it is (even though it has nothing to do with our original vision). A bit of realism is required here... I keep holding off the release date because I want to have these features in and do something 'big', and because I couldn't see myself write some converter for database stuff, but at some point it struck me that whether it's now or later, we WILL have to write a converter for whatever we do like converting attachments to media items... That's just the way it is. So, we could simply include TE's converter inside the Wedge codebase and have it run upgrades transparently to the user whenever a manual update of files is done. And that means, getting rid of upgrade.php (which isn't updated or suitable to Wedge anyway), and instead have: (1) install.sql (which shall be updated every time the database is modified), (2) the internal upgrade script in the admin area, which will either (a) do changes incrementally with one upgrade function per new database change, or (b) let's be crazy (and unrealistic), analyze the install.sql and compare it to the current database, and apply any changes, although it doesn't help when it comes to doing conversion work not related to the database itself (which is why I'm saying it's unrealistic...)

On August 25, 2012, Wedge will already be two years old. I think it would be a good date to go alpha :) Probably private alpha, for now (private read access to the SVN to all our Friends). Using our custom Wedge license, which will remain in place at least until we go for a wide release. Then we'll go public alpha, then public beta, then RC.

After we go wide, we'll determine what our final license for Wedge 1.0 will be. (For those who didn't follow -- our current favorite is the Mozilla Public License v2.0 with Exhibit B, i.e. can't be sub-licensed to GPL, although we have no qualms with seeing redistributions of GPL code alongside our MPL code. Go ask the GPL guys, it's their problem :))

Re: Speeding up development
« Reply #10, on August 7th, 2012, 02:58 PM »
Quote from Nao on August 7th, 2012, 12:37 PM
On August 25, 2012, Wedge will already be two years old. I think it would be a good date to go alpha :) Probably private alpha, for now (private read access to the SVN to all our Friends).
Sounds interesting ::)

Re: Development, full speed ahead!
« Reply #11, on August 7th, 2012, 04:39 PM »
I'd like for the private alpha to be as short as possible, it's just there to ensure that the deployment process is working OK, so that we don't get too many support requests right now. I'm also considering asking TE not to release his converters to the public until Wedge hits final, so that people aren't too 'tempted' to convert their SMF forums to use Wedge before it's set as 'fully working'.

Re: Development, full speed ahead!
« Reply #12, on August 7th, 2012, 05:28 PM »
Sounds interesting. I know I would be tempted to go full bore too fast. I'm bad that way,



Re: Speeding up development
« Reply #15, on August 8th, 2012, 08:09 PM »
Quote from markham on August 7th, 2012, 04:52 AM
I like the basic premise that Nao proposes. If Wedge calls home when the Admin CP is loaded, it could get all the update information at that time including an indication that the database schema needs to be updated. The Admin could be asked if he wants to update and if so, the entire site is placed in maintenance mode whilst the update script(s) execute. This could possibly be direct server to server so no need for the admin to download the update and then FTP it to his site.
Or is that too simplistic a view? :)
Yes. Because there are a million things that could (and will) go wrong. WordPress does a *fairly* good job for automatically updating the base software, but it can be a nightmare on non-default installation. Forums are rarely run as default installations, most are somewhat, many are heavily modified through mods, plugins, custom themes and some even use custom code modifications. No matter how good you implement an auto-updater for a complex piece of software - it will fail, period. Not for everyone, but for some, at least. The time needed to implement such a feature can be spent elsewhere and unless you have plenty of manpower (which most open source project do not have), wasting time is a no-go.

We are not talking about end-user crap here. This is admin stuff and admins should not give a damn about one-click solutions and getting everything the easy way (and if they do, they should get a different job, because they'll never become good admins).