This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
7141
Development blog / Package Manager, how we won't miss thee
« on June 22nd, 2011, 06:12 PM »
The question about the package manager, SMF mods in general, still haunts us. I recently proposed what I want to do about it to the private board, and now I'd like to expand on that and put it in general terms for everyone; my proposal was more technical than general in nature. Besides, I figure that you want to see not only where we are but a few thoughts and insight into where we're going. (That's what the blog's for, right? :lol:)
SMF's package manager is an interesting artefact, and certainly more than a nod to what came before. But we're bidding it farewell in Wedge, to be replaced with a very different beast, even if it looks similar on the surface.
The Add-on Manager. It sounds nicer than package manager, and it implies its purpose better; packages don't just have to be new functions even in SMF, they can be packs of avatars, language packs and so on, whereas an add-on is physically a new thing added on to the core.
In many ways it's more than just a change of name, it's also a change of mindset: the add-on manager does not allow add-ons to modify files AT ALL. I'm not just removing the functionality from the existing package manager, I'm actually implementing a whole new branch of code to replace it, every aspect is being gutted, torched and rebuilt. It means you're never trying to break old habits, you start out by learning better ones from the word go. It also means that SMF mods will have to be rewritten, but honestly, that's a blessing rather than a curse.
No permissions issues where you have to open up all your files to higher permissions than they needed. You'll even be able to add and remove packages simply by uploading folders and removing them, without risking it just breaking randomly - but you'll have to visit the admin panel to enable it, of course.
It's also much smarter: instead of this version emulate madness and tying it to versions, the process is much closer to feature detection - add-ons declare the facilities they need to be able to use, and if those needs are met, it can be activated. (Add-ons can even, if they so wish, declare that they provide certain facilities, and other add-ons can indicate they want to use them, so you can build add-ons that have implicit dependencies)
Also if a plugin has specific requirements such as obscure PHP extensions, these can be indicated in the package itself preventing installation without them.
Naturally, all this stuff will be available through whatever we come up with as an add-on repository, so it becomes much clearer as to what's needed and what works with what.
Just for an encore, the add-on management will be able to figure out what languages are supported in an add-on, so we can also display that on the add-on repo, as well as displaying it in the core.
Add-ons will live in their own folder, which I expect to call Addons (and removing the old Packages folder), one folder per add-on. That way, a plugin's functionality is self-contained rather than filling up the Sources and Themes/default/languages folders with stuff.
Now, all this sounds wonderful, too good to be true in fact... except it isn't. It's not only workable, but a slimmer, less refined version is even built in to SimpleDesk 2.0 and has been since I prototyped it a year ago. I've learnt some lessons since then, some refinements to that design, and some of the changes we've made in Wedge make it possible to really do it justice.
I won't get into the innards right now, other than that Modifications.*.php files will be disappearing, that the add-ons still use XML to relate their details, much like package-info.xml but nicer, and that it'll have more in common with WordPress' plugins area than SMF's package manager.
I'll release a few more details once I'm comfortable and can show you a bit more about how it works.
SMF's package manager is an interesting artefact, and certainly more than a nod to what came before. But we're bidding it farewell in Wedge, to be replaced with a very different beast, even if it looks similar on the surface.
The Add-on Manager. It sounds nicer than package manager, and it implies its purpose better; packages don't just have to be new functions even in SMF, they can be packs of avatars, language packs and so on, whereas an add-on is physically a new thing added on to the core.
In many ways it's more than just a change of name, it's also a change of mindset: the add-on manager does not allow add-ons to modify files AT ALL. I'm not just removing the functionality from the existing package manager, I'm actually implementing a whole new branch of code to replace it, every aspect is being gutted, torched and rebuilt. It means you're never trying to break old habits, you start out by learning better ones from the word go. It also means that SMF mods will have to be rewritten, but honestly, that's a blessing rather than a curse.
No permissions issues where you have to open up all your files to higher permissions than they needed. You'll even be able to add and remove packages simply by uploading folders and removing them, without risking it just breaking randomly - but you'll have to visit the admin panel to enable it, of course.
It's also much smarter: instead of this version emulate madness and tying it to versions, the process is much closer to feature detection - add-ons declare the facilities they need to be able to use, and if those needs are met, it can be activated. (Add-ons can even, if they so wish, declare that they provide certain facilities, and other add-ons can indicate they want to use them, so you can build add-ons that have implicit dependencies)
Also if a plugin has specific requirements such as obscure PHP extensions, these can be indicated in the package itself preventing installation without them.
Naturally, all this stuff will be available through whatever we come up with as an add-on repository, so it becomes much clearer as to what's needed and what works with what.
Just for an encore, the add-on management will be able to figure out what languages are supported in an add-on, so we can also display that on the add-on repo, as well as displaying it in the core.
Add-ons will live in their own folder, which I expect to call Addons (and removing the old Packages folder), one folder per add-on. That way, a plugin's functionality is self-contained rather than filling up the Sources and Themes/default/languages folders with stuff.
Now, all this sounds wonderful, too good to be true in fact... except it isn't. It's not only workable, but a slimmer, less refined version is even built in to SimpleDesk 2.0 and has been since I prototyped it a year ago. I've learnt some lessons since then, some refinements to that design, and some of the changes we've made in Wedge make it possible to really do it justice.
I won't get into the innards right now, other than that Modifications.*.php files will be disappearing, that the add-ons still use XML to relate their details, much like package-info.xml but nicer, and that it'll have more in common with WordPress' plugins area than SMF's package manager.
I'll release a few more details once I'm comfortable and can show you a bit more about how it works.
7142
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 05:48 PM »
That's the thing: the admin panel is still very much in flux. There will be a LOT more in it than there is in SMF's, simply because we've added more to it than we have taken out.
Right now it is basically just SMF's admin panel with half a dozen things gone and plenty more added. So, thus, my question stands: you've said that you see a problem with the look and layout options. What problem do you see? How would you resolve it?
I have my own ideas, some of which are fairly controversial on the subject of pruning the admin panel, so let's start on a smaller area...
Right now it is basically just SMF's admin panel with half a dozen things gone and plenty more added. So, thus, my question stands: you've said that you see a problem with the look and layout options. What problem do you see? How would you resolve it?
I have my own ideas, some of which are fairly controversial on the subject of pruning the admin panel, so let's start on a smaller area...
7143
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 04:21 PM »
Oh, that's a good lecture.Quote So, what is it you're asking for?
as how the users perceive the site
7144
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 03:56 PM »
I'm not sure I know exactly what you mean about 'making things appear/disappear'. But I certainly won't disagree that the admin panel is a clusterfuck of stuff that doesn't really make sense. 1.1 was better, 2.0 took everything in 1.1 and added more "stuff" to it and it's confused a lot of people, particularly the way permissions were done.
That said, you can't, realistically, have configuration options for every little thing: the admin panel already exceeds 100 1st and 2nd level menu items (which means you're basically talking ~90 pages of stuff, all told - first level items just display the first of an item's second level items)
There is a certain degree of value in 'less is more', in fact. Or at least, the perception of less stuff even if there's actually more stuff available to you.
That said, you can't, realistically, have configuration options for every little thing: the admin panel already exceeds 100 1st and 2nd level menu items (which means you're basically talking ~90 pages of stuff, all told - first level items just display the first of an item's second level items)
There is a certain degree of value in 'less is more', in fact. Or at least, the perception of less stuff even if there's actually more stuff available to you.
7145
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 03:43 PM »...is Wedge really trying to appeal to "noobs"?
Yet, I heard people talking about running it on shared hosts and so on. Regularly.
If it turns out well, the name gets about, people will want to use it, noob or not.
The thing is, by making things more logical and run more smoothly, the odds are fairly in favour that it will be more noob friendly than SMF by accident anyway, even though we're not encouraging its use by such.
7146
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 01:36 PM »
There's nothing that we expressly need 5.3 for in Wedge right now, but that's not to say we have to leave it that way. There's plenty of time to change yet :)
7147
Features: Forward thinking / Re: MySQL only!
« on June 22nd, 2011, 10:36 AM »
Except that we removed half of the abstraction layer because it added complexity and a series of kludgy workarounds for things.
7148
Development blog / Re: New directions
« on June 22nd, 2011, 10:31 AM »Hopefully we'll see a revolution of open source forks ;) that will commit to the community instead of an individual or group of individuals.
7149
Features: Forward thinking / Re: MySQL only!
« on June 22nd, 2011, 01:16 AM »
Yes, I saw. I'm not entirely sure what the benefits would be of using prepared statements vs what we're already doing (and how that's only going to change going forward anyway), and looking at the documentation it seems that we'd have to use the SQL syntax rather than the more elegant method (or converting everything to use MySQLi) which really seems to eradicate any benefits we could get from it.
Most hosts still provide ext/mysql and that's fine for now. Changing to async queries and the OOP style doesn't really achieve all that much so we don't really stand to gain much from changing to MySQLi; in any case the biggest DB overhead is one that MySQLi can't optimise for and that schema changes need to come first.
Most hosts still provide ext/mysql and that's fine for now. Changing to async queries and the OOP style doesn't really achieve all that much so we don't really stand to gain much from changing to MySQLi; in any case the biggest DB overhead is one that MySQLi can't optimise for and that schema changes need to come first.
7150
The Pub / Re: So this is where you folks have been hiding
« on June 22nd, 2011, 12:14 AM »Sooner or later SMF High Command will find out where we dug the escape tunnel. :huh:
7151
Other software / Re: Excuse me while I vomit.......
« on June 21st, 2011, 11:51 PM »
Sure there are, though such techniques aren't that prevalent any more, not when you can bulk buy batches for 1USD per 1000 solved. It's idiotically cheap now, such that it would actually be cheaper to do that than to mess about developing your own framework to do so.
7152
Off-topic / Re: Salary in web development
« on June 21st, 2011, 07:59 PM »
I doubt it, don't really know. Though if it's a really well known university it could make a difference, e.g. Harvard or Yale comes to mind.
7153
Off-topic / Re: Salary in web development
« on June 21st, 2011, 07:23 PM »
I have what's called an "A" Level in Computing, it's a UK qualification, roughly like a US high school diploma, nothing higher. A degree does give you a little more leverage, but experience and portfolio is key.
7154
Off-topic / Re: htmlspecialchars while inserting into DB
« on June 21st, 2011, 06:04 PM »This wasn't done for security reasons (although it's not necessarily a bad thing security wise.)
But yes, your other comment about not changing it on upgrading is a good one. In our case, we explicitly allowed for that sort of thing because we were looking at fixing otherwise broken links when coming from non SMF sources.
And lastly because you need to be able to edit your posts, and we knew there would be bugs if we tried to "reverse the translation" - but were worried about the additional storage requirements of storing two copies of every single post.
1. String parsing in PHP is slow.
I think whatever happens, we can't really afford to leave PHP, it's not like we can just conjure up a parser in C, and even if we could and did, I'd honestly not want the hassle of support for something like that.
2. Caching posts needs a bit of a better solution
parse_bbc does caching, but only under certain circumstances: if a post takes over a certain length of time to parse, and the parse_tags parameter wasn't given. parse_tags is only applicable when you invoke the parser with a subset of tags to accept, e.g. for signatures that have a different set of rules to normal.
In real practice, only truly huge posts ever hit this limit anyway.
What I've done is an "indexed cache", which means I mark entries in the cache that are popular, and over time, cache those popular items.
3. The parse_bbc() routine is far from perfect
4. Someone needs to profile parse_bbc()
Right. I think htmlspecialchars is relatively cheap, so I'd either do it on display, or else communicate the data in a text format (such as json, although that has its own escaping) and set it as text via the DOM.
My one fear with doing that is that it opens a door I don't really want opened. SMF's ecosystem is populated with those who write SSI functions and jiggle around with the innards, and I'd really rather not give them any more ways to make it easier to make them insecure.
Hmm. I feel like we had something in other/tools/ that did this and did do topic reattribution, which Compuart I think wrote. Weird.
centralized bug fixes in an unofficial package that I updated every day or two.
I'm the guy who always wanted to remove the calendar from the standard distribution of SMF.
Right now the calendar is seemingly used by the relative minority. A small but vocal group use it for events, and a larger group turn it on because of birthdays.
So, my take on it, specifically, is that birthdays should be separated from the calendar, and then the calendar should either be moved out of the core, or made more useful, so as that more people would find it useful. For example, we've talked about accessing "Today's posts" from the calendar.
While I don't like to point at other systems and go "they do it" (it sort of goes against my "innovate, not imitate" philosophy), I look at WordPress and I see they have it. Different horses for different courses but they have a calendar in the core, enabled by default. Because it suits them to do so.
Given that we have blog support - because it's something we both want and I saw more and more requests for it at SMF (and the fact that if you have Wedge, you shouldn't really need to bridge to WP for a simplistic blog, have one system and be done with it), the calendar takes on a new light - and one that helps justify keeping it.
7155
Off-topic / Re: htmlspecialchars while inserting into DB
« on June 21st, 2011, 11:59 AM »Definitely the br thing was. And I know the nobbc/html thing was just because that was the easiest way.
While I agree, it also means large forums (where's Douglas at anyway?) are a big problem.
Douglas is still running his forums but I think he's moved on from his old job at SolarVPS these days, not sure what he's up to now though.
Sure. But I explicitly don't agree with the "garbage in, garbage out" philosophy when it comes to security. An example: just because you only put valid filenames into the attachments table doesn't mean you shouldn't validate that assumption on the way out too. This also improves upgrade scenarios anyway (if done right.)
The big example is parse_bbc. It is the single slowest component in SMF and Wedge, and it is a scary monster because it's expressly checking that what's coming out is safe, with certain protections carried out separately in preparsecode (like sanitising nobbc and html). I think it's important that some element of that is continued to be done, but how far do you go with it? At what point is it acceptable to trade off performance for security?
On the flip side, there are things like topic subjects, which aren't resanitised on display, what's in the DB is assumed to be safe.
Well, at work, this is a pretty common mantra. I'm even holding back changes on a project right now for this reason.
One that really annoyed me is the reattribute posts function. It reattributes old posts to a new user, e.g. new account after deletion, but it doesn't do topic ownership at the same time. Someone demonstrated the query, it works and I don't see anything about it that would cause regression. But it was deferred.
At the same time, the dreaded "IE 8 jumping text bug", which they fixed late in the day, and that DID cause a regression which they sort of fixed in the templates without fixing the root cause - the editor component now can't be given a width from the create function, so any mod that invokes the editor and asks for the default of 70% width doesn't get it, it just fills the maximum available width, and it's "acceptable" to rewrite it in the template to set the width there, rather than setting it when it's invoked. Which means, I had to go fix all the instances of this in SimpleDesk, because they fixed one bug and made another, which to me is a regression.
Back whenever ago, I posted the same basic thing on simplemachines.org; if you need more RCs, release more RCs. There's no shame in releasing RCs
You know something's wrong when they're accepting mods on the mod site that are bug-fix mods. I only wish I were kidding.
Well, jQuery is fine. Most people who take the test, even if they add jQuery, leave that line in and don't touch it; they clearly don't understand it. That's what's a bad sign.
I'm willing to bet that not everyone who left it alone did so because they didn't understand it - but I won't argue that it's certainly the more likely reason.
My problem with it is that it's got a lot of features, most of which I don't need, and doesn't provide most of the features I do.
It's a convenient go-to place, but like everything else it has pros and cons that need to be weighed up for the project at hand. For us, it's convenient: it provides a consistent XHR object with some nice conveniences, animation which can be nice for UX if done properly, and it makes DOM manipulation easy for the sorts of thing we're doing.
Plus it's available via CDN, so it's not like users probably won't have it cached.
And, it's popular. But yes, it can encourage bad behaviours - but so too does PHP ;)