Wedge

Public area => Development blog => Topic started by: Nao on August 6th, 2012, 06:38 PM

Title: Development, full speed ahead!
Post by: Nao on August 6th, 2012, 06:38 PM
Hey everyone! Long time no blog. Nope, none of us are dead... We just had 'things' happen in real life, which slowed everything down, sometimes to a halt. I made sure to have at least a couple of commits out every week, just so I never lose focus on what I'm planning to do, but it's true that I've had a tendency to focus on smaller details rather than the big picture.

The main point is that we've often gone through the same story: deciding to implement a minor feature that would be cool, and then realizing, weeks later, that we didn't realize that 'minor' doesn't mean 'quick to implement'... And it went on, and on. As a result, I myself became wary of starting the heavy-duty work on a number of 'heavy' features that I'd like to have in Wedge from day one[1], and I think if we keep it that way, we'll always have a good excuse not to release Wedge. Not that we're in a hurry to do it -- but we've got to keep it real, and part of it means we have to keep it real for users.

So, the reason why I'm so reluctant to go public is mainly the fact that it's hard to update the database once it's installed somewhere. If you don't tell people to use the upgrade script etc, then you're likely to get annoying support questions...
So here's what I'm suggesting: storing a $settings['db_version'] variable (or a variation if there's already an equivalent), and set it to the SVN revision number corresponding to the lastest database structure change. So, at runtime we can simply, as soon as we load $settings, check for the variable, and if it's lower than the current required database version, load an extra file that will automatically adjust the database structure. e.g. if the db_version is 1652, and user installs version 1815, Wedge will execute all database (and file structure) modifier functions located between those rev numbers -- db_update_1664, db_update_1727, etc... It should be easy to calculate, really, and we can safely call them, and then update db_version to the latest database version available. Then it wouldn't be (so) hard to change the database, whether for us or for users.

What do you think...?

PS: bugger, another issue I just discovered... The selection is off by a few bytes when I click the Bold button in the bbc box, perhaps related to the Opera hack that I wrote for editor.js in both SMF and Wedge... Maybe they FINALLY, after years, fixed that bug...? :edit: Yes, they did, but they didn't fix all of it, so I'll have to keep an eye open...
 1. Custom boards, board types, custom groups, general privacy system, general tag system, attachments as media items, and so on...
Title: Re: Speeding up development
Post by: Arantor on August 6th, 2012, 07:13 PM
Quote
So here's what I'm suggesting: storing a $settings['db_version'] variable (or a variation if there's already an equivalent),
This is what smfVersion was supposed to be used for.

I like the idea of the schema changes, not sure how practical it would really be in practice - or to do safely on an otherwise running install. I foresee lots of locking type issues attached especially on bigger installs.
Title: Re: Speeding up development
Post by: Nao on August 6th, 2012, 09:57 PM
So that'd be we_version..?
I'd like to use a SVN revision number for simplicity (for us!), since it'll be internal use anyway... (And I don't plan to do away with our SVN rev numbers, if you catch my drift :P)

I was thinking that we could disable the forum temporarily while upgrading. The script would only be run if the admin is online (and by the admin.)
There's bound to be a few minutes where things will go awry for everyone, which is logical considering one is updating their forum files manually by FTP... At some point there'll be a v1.0 file running alongside a v2.0 file, etc. Anyway, it's no biggie.
Title: Re: Speeding up development
Post by: Nao on August 6th, 2012, 09:58 PM
Quote from Nao on August 6th, 2012, 06:38 PM
PS: crap, another bug I just discovered... The selection is off by a few bytes when I click the Bold button in the bbc box, perhaps related to the Opera hack that I devised for editor.js... Maybe they FINALLY, after years, fixed that bug...?
Problem doesn't happen in Opera 12.01... So, conclusion: this is an Opera 12.50 bug. Dunno if I should report it (they never, ever bothered to look into my bug reports), or if they're already aware of the problem...
Title: Re: Speeding up development
Post by: Arantor on August 6th, 2012, 10:22 PM
Quote
So that'd be we_version..?
Yes, though I didn't remember if we'd ditched it or not.
Quote
I was thinking that we could disable the forum temporarily while upgrading. The script would only be run if the admin is online (and by the admin.)
So they download the updated files, then figure out what changes have to be made if any, then apply it all at once... hmm.
Title: Re: Speeding up development
Post by: Nao on August 6th, 2012, 11:15 PM
We didn't ditch it, I think... It was probably renamed to 'weVersion' or 'we_version' or even 'weVer'... Can't remember.

No, they don't have to figure out anything, they just need to reupload everything...? It's slower, but it's doable by anyone ahah...



My public report...
http://my.opera.com/desktopteam/blog/show.dml/51546052?startidx=250#comment94192592

It's definitely a bug on their side, but it also means that Opera will be buggy in SMF 2.0.x!
Title: Re: Speeding up development
Post by: Arantor on August 6th, 2012, 11:31 PM
That's the point, though, it really doesn't work like that in practice. WordPress does something similar, and I've seen enough of those go wrong in the past.
Title: Re: Speeding up development
Post by: markham on August 7th, 2012, 04:52 AM
I like the basic premise that Nao proposes. If Wedge calls home when the Admin CP is loaded, it could get all the update information at that time including an indication that the database schema needs to be updated. The Admin could be asked if he wants to update and if so, the entire site is placed in maintenance mode whilst the update script(s) execute. This could possibly be direct server to server so no need for the admin to download the update and then FTP it to his site.
Or is that too simplistic a view? :)
Title: Re: Speeding up development
Post by: Powerbob on August 7th, 2012, 09:48 AM
Works for me :angel:
Title: Re: Speeding up development
Post by: Nao on August 7th, 2012, 12:37 PM
Oh, I never suggested going with what Aeva Media used to do, i.e. downloading its site list automatically and installing it transparently for the admin...
Well, I used to want to do that, but Pete reminded me that it's unlikely it's going to work for the forum code itself, as there are so many ways to screw that up ;)

I'm just saying that we should do away with our 'big' features which, to be honest, I'm too scared to try and implement because they might take so long to do, and instead go with a 'light' Wedge, which is already great enough as it is (even though it has nothing to do with our original vision). A bit of realism is required here... I keep holding off the release date because I want to have these features in and do something 'big', and because I couldn't see myself write some converter for database stuff, but at some point it struck me that whether it's now or later, we WILL have to write a converter for whatever we do like converting attachments to media items... That's just the way it is. So, we could simply include TE's converter inside the Wedge codebase and have it run upgrades transparently to the user whenever a manual update of files is done. And that means, getting rid of upgrade.php (which isn't updated or suitable to Wedge anyway), and instead have: (1) install.sql (which shall be updated every time the database is modified), (2) the internal upgrade script in the admin area, which will either (a) do changes incrementally with one upgrade function per new database change, or (b) let's be crazy (and unrealistic), analyze the install.sql and compare it to the current database, and apply any changes, although it doesn't help when it comes to doing conversion work not related to the database itself (which is why I'm saying it's unrealistic...)

On August 25, 2012, Wedge will already be two years old. I think it would be a good date to go alpha :) Probably private alpha, for now (private read access to the SVN to all our Friends). Using our custom Wedge license, which will remain in place at least until we go for a wide release. Then we'll go public alpha, then public beta, then RC.

After we go wide, we'll determine what our final license for Wedge 1.0 will be. (For those who didn't follow -- our current favorite is the Mozilla Public License v2.0 with Exhibit B, i.e. can't be sub-licensed to GPL, although we have no qualms with seeing redistributions of GPL code alongside our MPL code. Go ask the GPL guys, it's their problem :))
Title: Re: Speeding up development
Post by: feline on August 7th, 2012, 02:58 PM
Quote from Nao on August 7th, 2012, 12:37 PM
On August 25, 2012, Wedge will already be two years old. I think it would be a good date to go alpha :) Probably private alpha, for now (private read access to the SVN to all our Friends).
Sounds interesting ::)
Title: Re: Development, full speed ahead!
Post by: Nao on August 7th, 2012, 04:39 PM
I'd like for the private alpha to be as short as possible, it's just there to ensure that the deployment process is working OK, so that we don't get too many support requests right now. I'm also considering asking TE not to release his converters to the public until Wedge hits final, so that people aren't too 'tempted' to convert their SMF forums to use Wedge before it's set as 'fully working'.
Title: Re: Development, full speed ahead!
Post by: lazyt on August 7th, 2012, 05:28 PM
Sounds interesting. I know I would be tempted to go full bore too fast. I'm bad that way,
Title: Re: Development, full speed ahead!
Post by: MultiformeIngegno on August 7th, 2012, 07:53 PM
Good news!! :D
Title: Re: Development, full speed ahead!
Post by: Powerbob on August 8th, 2012, 10:24 AM
Looks like my weekend is planned out now :lol:
Title: Re: Speeding up development
Post by: Nightwish on August 8th, 2012, 08:09 PM
Quote from markham on August 7th, 2012, 04:52 AM
I like the basic premise that Nao proposes. If Wedge calls home when the Admin CP is loaded, it could get all the update information at that time including an indication that the database schema needs to be updated. The Admin could be asked if he wants to update and if so, the entire site is placed in maintenance mode whilst the update script(s) execute. This could possibly be direct server to server so no need for the admin to download the update and then FTP it to his site.
Or is that too simplistic a view? :)
Yes. Because there are a million things that could (and will) go wrong. WordPress does a *fairly* good job for automatically updating the base software, but it can be a nightmare on non-default installation. Forums are rarely run as default installations, most are somewhat, many are heavily modified through mods, plugins, custom themes and some even use custom code modifications. No matter how good you implement an auto-updater for a complex piece of software - it will fail, period. Not for everyone, but for some, at least. The time needed to implement such a feature can be spent elsewhere and unless you have plenty of manpower (which most open source project do not have), wasting time is a no-go.

We are not talking about end-user crap here. This is admin stuff and admins should not give a damn about one-click solutions and getting everything the easy way (and if they do, they should get a different job, because they'll never become good admins).
Title: Re: Development, full speed ahead!
Post by: Norodo on August 8th, 2012, 11:18 PM
Yes. This is a good idea.
Title: Re: Development, full speed ahead!
Post by: william777 on August 8th, 2012, 11:49 PM
Great news. Better than Olympic Gold. Thanks Nao.
Title: Re: Development, full speed ahead!
Post by: godboko71 on August 9th, 2012, 01:42 AM
What ever makes you guys happy :) I have a few different configs to install on when the 25th hits :) and if that day comes and goes thats cool too :)
Title: Re: Development, full speed ahead!
Post by: Powerbob on August 9th, 2012, 10:34 AM
TBH when updating my forums I always uninstall all mods to be safe, and after updating re-install them one by one!
Title: Re: Development, full speed ahead!
Post by: Arantor on August 9th, 2012, 12:47 PM
That should be far less of an issue with Wedge because of the way plugins work. It could be made almost transparent with little or no real effort, too.
Title: Re: Development, full speed ahead!
Post by: Powerbob on August 9th, 2012, 08:15 PM
cool :cool:
Title: Re: Development, full speed ahead!
Post by: jpb on August 13th, 2012, 10:35 AM
Great!
Title: Re: Development, full speed ahead!
Post by: Nao on August 13th, 2012, 10:44 AM
Quote from Arantor on August 9th, 2012, 12:47 PM
That should be far less of an issue with Wedge because of the way plugins work. It could be made almost transparent with little or no real effort, too.
Yeah.
Technically, I think we could even get away with a webinstall if we provide a single PHP file to upload somewhere, and that will retrieve the latest files from our server and install them. That way, we can be sure the entire Wedge folder has the same rights as the PHP user... (Of course, updating files manually could then cause issues.)

Well, anyway I don't even know if Wedge will be released anyway. I spent my entire weekend fixing bugs and ended up making them worse. For instance, Opera had the fantastic idea of releasing a new alpha where CSS gradient prefixes are removed -- and support for the prefixed version is removed! Thank you guys, now I have to include both versions... -_- Anyway, as Opera updates itself automatically, once v12.50 is out it won't be much of a problem. When it comes to Firefox, though, v16+ removes the prefixes, but if you want to support both, it won't work: showing the prefixed version then the unprefixed version will fail altogether in v<16. Thankfully, they still support the prefixed version so I'm sticking to that one only for now...
Stupid browsers and incompatibilities.
Now it also shows that simply providing a '-prefix' variable in WeCSS isn't enough to warrant compatibility. Well, it COULD be enough, but it'd mean I have to add browser-specific files to disable some of the CSS that would make everything fail. That's stupid. So I rewrote the system to simplify it for users... And, of course, it doesn't work well. (Still working on it.)

Not only that, but my confirm() rewrite is stalled, as well as my virtual keyword bug.

I'm pretty much screwed these days.
Title: Re: Development, full speed ahead!
Post by: billy2 on August 13th, 2012, 02:50 PM
It gets soul destroying, one step forward, one and a half steps back.
You have both come a long way.... you can do it
 :cool:
Title: Re: Development, full speed ahead!
Post by: Arantor on August 13th, 2012, 03:21 PM
Quote
That way, we can be sure the entire Wedge folder has the same rights as the PHP user...
There is absolutely no way in hell I am going to do that. It's a recipe for being hacked.
Title: Re: Development, full speed ahead!
Post by: Nao on August 13th, 2012, 03:58 PM
Quote from Arantor on August 13th, 2012, 03:21 PM
There is absolutely no way in hell I am going to do that. It's a recipe for being hacked.
Then we'll have to live with people who keep bugging us about upgrade paths...

Eh.

As if it's not already enough that I have to live with internal W3C wars that force me to rewrite WeCSS for days... -_-

God why did I start work on Wedge?! Why am I heading towards an OS license when we should really be selling it for a million dollars?!
Title: Re: Development, full speed ahead!
Post by: Arantor on August 13th, 2012, 04:20 PM
Quote
Then we'll have to live with people who keep bugging us about upgrade paths...
It's the same reason I've been so stubborn about the plugin manager and having write access - doing it all via FTP is the only safe path.
Quote
God why did I start work on Wedge?! Why am I heading towards an OS license when we should really be selling it for a million dollars?!
Because we wanted to make a difference. And because it wasn't ever really about the money.
Title: Re: Development, full speed ahead!
Post by: Nao on August 13th, 2012, 05:33 PM
Quote from Arantor on August 13th, 2012, 04:20 PM
Because we wanted to make a difference. And because it wasn't ever really about the money.
Yeah it never was... But money can be an incentive to continue. Money can help pay for treating the cancer I'm likely to get in a few years from now (if it's not already too late) due to all the stress Wedge has generated in my life.

Anyway... I've rewritten WeCSS in a lot of ways. Now it also caches the browser version, even if not IE. It's a bit sad I know, but it's the only way I can safely enable/disable prefixes at will. (As of now, if you run wedge.org with the latest Opera 12.50 snapshot, none of the gradients will show up.)
I'm not even going to complain... I added a small script to automatically purge the filecache every month, so that older browser versions don't clutter the filesystem after being unused for months.