Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Arantor
7141
Development blog / Package Manager, how we won't miss thee
« on June 22nd, 2011, 06:12 PM »
The question about the package manager, SMF mods in general, still haunts us. I recently proposed what I want to do about it to the private board, and now I'd like to expand on that and put it in general terms for everyone; my proposal was more technical than general in nature. Besides, I figure that you want to see not only where we are but a few thoughts and insight into where we're going. (That's what the blog's for, right? :lol:)

SMF's package manager is an interesting artefact, and certainly more than a nod to what came before. But we're bidding it farewell in Wedge, to be replaced with a very different beast, even if it looks similar on the surface.


The Add-on Manager. It sounds nicer than package manager, and it implies its purpose better; packages don't just have to be new functions even in SMF, they can be packs of avatars, language packs and so on, whereas an add-on is physically a new thing added on to the core.

In many ways it's more than just a change of name, it's also a change of mindset: the add-on manager does not allow add-ons to modify files AT ALL. I'm not just removing the functionality from the existing package manager, I'm actually implementing a whole new branch of code to replace it, every aspect is being gutted, torched and rebuilt. It means you're never trying to break old habits, you start out by learning better ones from the word go. It also means that SMF mods will have to be rewritten, but honestly, that's a blessing rather than a curse.

No permissions issues where you have to open up all your files to higher permissions than they needed. You'll even be able to add and remove packages simply by uploading folders and removing them, without risking it just breaking randomly - but you'll have to visit the admin panel to enable it, of course.

It's also much smarter: instead of this version emulate madness and tying it to versions, the process is much closer to feature detection - add-ons declare the facilities they need to be able to use, and if those needs are met, it can be activated. (Add-ons can even, if they so wish, declare that they provide certain facilities, and other add-ons can indicate they want to use them, so you can build add-ons that have implicit dependencies)

Also if a plugin has specific requirements such as obscure PHP extensions, these can be indicated in the package itself preventing installation without them.

Naturally, all this stuff will be available through whatever we come up with as an add-on repository, so it becomes much clearer as to what's needed and what works with what.

Just for an encore, the add-on management will be able to figure out what languages are supported in an add-on, so we can also display that on the add-on repo, as well as displaying it in the core.


Add-ons will live in their own folder, which I expect to call Addons (and removing the old Packages folder), one folder per add-on. That way, a plugin's functionality is self-contained rather than filling up the Sources and Themes/default/languages folders with stuff.

Now, all this sounds wonderful, too good to be true in fact... except it isn't. It's not only workable, but a slimmer, less refined version is even built in to SimpleDesk 2.0 and has been since I prototyped it a year ago. I've learnt some lessons since then, some refinements to that design, and some of the changes we've made in Wedge make it possible to really do it justice.


I won't get into the innards right now, other than that Modifications.*.php files will be disappearing, that the add-ons still use XML to relate their details, much like package-info.xml but nicer, and that it'll have more in common with WordPress' plugins area than SMF's package manager.

I'll release a few more details once I'm comfortable and can show you a bit more about how it works.
7142
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 05:48 PM »
That's the thing: the admin panel is still very much in flux. There will be a LOT more in it than there is in SMF's, simply because we've added more to it than we have taken out.

Right now it is basically just SMF's admin panel with half a dozen things gone and plenty more added. So, thus, my question stands: you've said that you see a problem with the look and layout options. What problem do you see? How would you resolve it?

I have my own ideas, some of which are fairly controversial on the subject of pruning the admin panel, so let's start on a smaller area...
7143
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 04:21 PM »
Oh, that's a good lecture.
Quote
as how the users perceive the site
So, what is it you're asking for?
7144
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 03:56 PM »
I'm not sure I know exactly what you mean about 'making things appear/disappear'. But I certainly won't disagree that the admin panel is a clusterfuck of stuff that doesn't really make sense. 1.1 was better, 2.0 took everything in 1.1 and added more "stuff" to it and it's confused a lot of people, particularly the way permissions were done.

That said, you can't, realistically, have configuration options for every little thing: the admin panel already exceeds 100 1st and 2nd level menu items (which means you're basically talking ~90 pages of stuff, all told - first level items just display the first of an item's second level items)

There is a certain degree of value in 'less is more', in fact. Or at least, the perception of less stuff even if there's actually more stuff available to you.
7145
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 03:43 PM »
Quote
...is Wedge really trying to appeal to "noobs"?
No, but it's inevitable that noobs will use it anyway. Hell, before I got actively involved with SMF, I used to provide support for Sphinx, which is a search daemon. Clue's in the title; it's a daemon, which means it's running on shell, as a system service, and you will likely need to compile it yourself.

Yet, I heard people talking about running it on shared hosts and so on. Regularly.

If it turns out well, the name gets about, people will want to use it, noob or not.

The thing is, by making things more logical and run more smoothly, the odds are fairly in favour that it will be more noob friendly than SMF by accident anyway, even though we're not encouraging its use by such.
7146
Features: Forward thinking / Re: Removing deprecated code
« on June 22nd, 2011, 01:36 PM »
There's nothing that we expressly need 5.3 for in Wedge right now, but that's not to say we have to leave it that way. There's plenty of time to change yet :)
7147
Features: Forward thinking / Re: MySQL only!
« on June 22nd, 2011, 10:36 AM »
Except that we removed half of the abstraction layer because it added complexity and a series of kludgy workarounds for things.
7148
Development blog / Re: New directions
« on June 22nd, 2011, 10:31 AM »
Quote from groundup on June 22nd, 2011, 03:00 AM
Hopefully we'll see a revolution of open source forks ;) that will commit to the community instead of an individual or group of individuals.
Low blow. Especially since you clearly don't know the whole story. The move back to closed source is simply so that SMF doesn't start ripping off our code after the way things went down. Once our code is sufficiently far away from SMF to be useful, I suspect we'll reopen it again later.
7149
Features: Forward thinking / Re: MySQL only!
« on June 22nd, 2011, 01:16 AM »
Yes, I saw. I'm not entirely sure what the benefits would be of using prepared statements vs what we're already doing (and how that's only going to change going forward anyway), and looking at the documentation it seems that we'd have to use the SQL syntax rather than the more elegant method (or converting everything to use MySQLi) which really seems to eradicate any benefits we could get from it.

Most hosts still provide ext/mysql and that's fine for now. Changing to async queries and the OOP style doesn't really achieve all that much so we don't really stand to gain much from changing to MySQLi; in any case the biggest DB overhead is one that MySQLi can't optimise for and that schema changes need to come first.
7150
The Pub / Re: So this is where you folks have been hiding
« on June 22nd, 2011, 12:14 AM »
Quote from YogiBear on June 22nd, 2011, 12:00 AM
Sooner or later SMF High Command will find out where we dug the escape tunnel.  :huh:
Oh, they've known for a while. It's not like it's a secret or anything.
7151
Other software / Re: Excuse me while I vomit.......
« on June 21st, 2011, 11:51 PM »
Sure there are, though such techniques aren't that prevalent any more, not when you can bulk buy batches for 1USD per 1000 solved. It's idiotically cheap now, such that it would actually be cheaper to do that than to mess about developing your own framework to do so.
7152
Off-topic / Re: Salary in web development
« on June 21st, 2011, 07:59 PM »
I doubt it, don't really know. Though if it's a really well known university it could make a difference, e.g. Harvard or Yale comes to mind.
7153
Off-topic / Re: Salary in web development
« on June 21st, 2011, 07:23 PM »
I have what's called an "A" Level in Computing, it's a UK qualification, roughly like a US high school diploma, nothing higher. A degree does give you a little more leverage, but experience and portfolio is key.
7154
Off-topic / Re: htmlspecialchars while inserting into DB
« on June 21st, 2011, 06:04 PM »
Quote
This wasn't done for security reasons (although it's not necessarily a bad thing security wise.)
Oh? Well, I never went as far back as YaBB or YaBBSE, so I'm looking at it all from SMF 1.1 and later's perspective, and what I've seen said. My understanding was that SMF 1.0 did bbc parsing through regexp, on display, but that a vicious ReDoS could take it out because it wasn't protected against that, hence it was doing through string parsing and done on a regurgitation basis, and that security was one of the major factors.

But yes, your other comment about not changing it on upgrading is a good one. In our case, we explicitly allowed for that sort of thing because we were looking at fixing otherwise broken links when coming from non SMF sources.
Quote
And lastly because you need to be able to edit your posts, and we knew there would be bugs if we tried to "reverse the translation" - but were worried about the additional storage requirements of storing two copies of every single post.
Funny you should mention that, that's exactly what the WYSIWYG editor tries to do, badly. So much so that it doesn't bother converting anything other than simpler HTML.
Quote
1. String parsing in PHP is slow.
No argument. Interesting comment there, there were plans to rewrite the bbc parser in C under the banner of smflib. (Not sure if that was after your time or not) but it never really went anywhere. It's still in SMF's SVN, untouched in pretty much forever.

I think whatever happens, we can't really afford to leave PHP, it's not like we can just conjure up a parser in C, and even if we could and did, I'd honestly not want the hassle of support for something like that.
Quote
2. Caching posts needs a bit of a better solution
Correction: it needs *some* solution.

parse_bbc does caching, but only under certain circumstances: if a post takes over a certain length of time to parse, and the parse_tags parameter wasn't given. parse_tags is only applicable when you invoke the parser with a subset of tags to accept, e.g. for signatures that have a different set of rules to normal.

In real practice, only truly huge posts ever hit this limit anyway.
Quote
What I've done is an "indexed cache", which means I mark entries in the cache that are popular, and over time, cache those popular items.
This sounds like a good idea to me. I'm not sure offhand how we'd do it, but it seems more reasonable than just bulk throwing things at cache.
Quote
3. The parse_bbc() routine is far from perfect
I really need to get properly familiar with TOX-G's innards. Is there anything specific you're thinking about how it operates that you'd do differently?
Quote
4. Someone needs to profile parse_bbc()
*nods* It does need to be done properly. I never seem to find the time to do this though :/
Quote
Right.  I think htmlspecialchars is relatively cheap, so I'd either do it on display, or else communicate the data in a text format (such as json, although that has its own escaping) and set it as text via the DOM.
So, then, if we were to htmlspecialchars it on output, presumably we wouldn't do it on saving the data in the first place? Or would we unsanitise it before resanitising it?

My one fear with doing that is that it opens a door I don't really want opened. SMF's ecosystem is populated with those who write SSI functions and jiggle around with the innards, and I'd really rather not give them any more ways to make it easier to make them insecure.
Quote
Hmm.  I feel like we had something in other/tools/ that did this and did do topic reattribution, which Compuart I think wrote.  Weird.
Well, in 2.0, it's in the admin panel. Except it's broken.
Quote
centralized bug fixes in an unofficial package that I updated every day or two.
See, that's neat. And really, I'm not against the idea. I just wish they'd write rules and stick to them; it always used to be the case when I was on the Cust. team that bugfixes weren't allowed. Then they allow these bugfixes, and they even package up the bugfix for the IE 8 "jumping text bug" for 1.1.x as a mod rather than roll it into the update >_<
Quote
I'm the guy who always wanted to remove the calendar from the standard distribution of SMF.
Things like this I'm in one of two minds about - and it's been discussed here too. My take on it is straightforward enough: everything that's in the core of Wedge should be something that a "reasonable percentage of users" are going to use.

Right now the calendar is seemingly used by the relative minority. A small but vocal group use it for events, and a larger group turn it on because of birthdays.

So, my take on it, specifically, is that birthdays should be separated from the calendar, and then the calendar should either be moved out of the core, or made more useful, so as that more people would find it useful. For example, we've talked about accessing "Today's posts" from the calendar.

While I don't like to point at other systems and go "they do it" (it sort of goes against my "innovate, not imitate" philosophy), I look at WordPress and I see they have it. Different horses for different courses but they have a calendar in the core, enabled by default. Because it suits them to do so.

Given that we have blog support - because it's something we both want and I saw more and more requests for it at SMF (and the fact that if you have Wedge, you shouldn't really need to bridge to WP for a simplistic blog, have one system and be done with it), the calendar takes on a new light - and one that helps justify keeping it.
7155
Off-topic / Re: htmlspecialchars while inserting into DB
« on June 21st, 2011, 11:59 AM »
Quote
Definitely the br thing was.  And I know the nobbc/html thing was just because that was the easiest way.
I'm not knocking the nobbc/html thing in any way, frankly I think it's the way to go.
Quote
While I agree, it also means large forums (where's Douglas at anyway?) are a big problem.
Large forums are always going to be a problem when you're moving to anything else, if it does anything differently. I'd rather handle it at conversion time and end up with something cleaner - start as you mean to go on.

Douglas is still running his forums but I think he's moved on from his old job at SolarVPS these days, not sure what he's up to now though.
Quote
Sure.  But I explicitly don't agree with the "garbage in, garbage out" philosophy when it comes to security.  An example: just because you only put valid filenames into the attachments table doesn't mean you shouldn't validate that assumption on the way out too.  This also improves upgrade scenarios anyway (if done right.)
That's why I said it depends what you're talking about. As a minimum, what goes into the DB should be sane/safe. The real question about coming back out is how much effort it is to validate that assumption, which is where the performance/security balance comes back in.

The big example is parse_bbc. It is the single slowest component in SMF and Wedge, and it is a scary monster because it's expressly checking that what's coming out is safe, with certain protections carried out separately in preparsecode (like sanitising nobbc and html). I think it's important that some element of that is continued to be done, but how far do you go with it? At what point is it acceptable to trade off performance for security?

On the flip side, there are things like topic subjects, which aren't resanitised on display, what's in the DB is assumed to be safe.
Quote
Well, at work, this is a pretty common mantra.  I'm even holding back changes on a project right now for this reason.
It's a common mantra, but in the cases I'm thinking of, it's misapplied.

One that really annoyed me is the reattribute posts function. It reattributes old posts to a new user, e.g. new account after deletion, but it doesn't do topic ownership at the same time. Someone demonstrated the query, it works and I don't see anything about it that would cause regression. But it was deferred.

At the same time, the dreaded "IE 8 jumping text bug", which they fixed late in the day, and that DID cause a regression which they sort of fixed in the templates without fixing the root cause - the editor component now can't be given a width from the create function, so any mod that invokes the editor and asks for the default of 70% width doesn't get it, it just fills the maximum available width, and it's "acceptable" to rewrite it in the template to set the width there, rather than setting it when it's invoked. Which means, I had to go fix all the instances of this in SimpleDesk, because they fixed one bug and made another, which to me is a regression.
Quote
Back whenever ago, I posted the same basic thing on simplemachines.org; if you need more RCs, release more RCs.  There's no shame in releasing RCs
To be fair to them, a lot of people were very fed up of a new RC every six months or so that fixed some bugs and not a lot else. The problem is that all the signs hint at 2.0.1 not being a bugfix release but only in the event of security fixes, so some of the deferred bugs will only be solved in 2.1 or higher, which is a butt-ache if they've been on the tracker for a couple of years, especially if there are posted solutions.

You know something's wrong when they're accepting mods on the mod site that are bug-fix mods. I only wish I were kidding.
Quote
Well, jQuery is fine.  Most people who take the test, even if they add jQuery, leave that line in and don't touch it; they clearly don't understand it.  That's what's a bad sign.
On the other hand, if it's there, it's there for a reason, and usually that reason is because other stuff depends on it working exactly that way. If I were to add jQuery to something like that, I'd be flagging up that this other item was to be removed and all instances that use it should proceed to use jQuery.

I'm willing to bet that not everyone who left it alone did so because they didn't understand it - but I won't argue that it's certainly the more likely reason.
Quote
My problem with it is that it's got a lot of features, most of which I don't need, and doesn't provide most of the features I do.
You can say that about a lot of things, though. It's almost the same argument against something like Microsoft Word: just because you might only use 2% of its features, that sounds like an argument for stripping out the rest - until you realise that everyone else uses a different 2% of its features.

It's a convenient go-to place, but like everything else it has pros and cons that need to be weighed up for the project at hand. For us, it's convenient: it provides a consistent XHR object with some nice conveniences, animation which can be nice for UX if done properly, and it makes DOM manipulation easy for the sorts of thing we're doing.

Plus it's available via CDN, so it's not like users probably won't have it cached.

And, it's popular. But yes, it can encourage bad behaviours - but so too does PHP ;)