Package Manager, how we won't miss thee

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Package Manager, how we won't miss thee
« on June 22nd, 2011, 06:12 PM »
The question about the package manager, SMF mods in general, still haunts us. I recently proposed what I want to do about it to the private board, and now I'd like to expand on that and put it in general terms for everyone; my proposal was more technical than general in nature. Besides, I figure that you want to see not only where we are but a few thoughts and insight into where we're going. (That's what the blog's for, right? :lol:)

SMF's package manager is an interesting artefact, and certainly more than a nod to what came before. But we're bidding it farewell in Wedge, to be replaced with a very different beast, even if it looks similar on the surface.


The Add-on Manager. It sounds nicer than package manager, and it implies its purpose better; packages don't just have to be new functions even in SMF, they can be packs of avatars, language packs and so on, whereas an add-on is physically a new thing added on to the core.

In many ways it's more than just a change of name, it's also a change of mindset: the add-on manager does not allow add-ons to modify files AT ALL. I'm not just removing the functionality from the existing package manager, I'm actually implementing a whole new branch of code to replace it, every aspect is being gutted, torched and rebuilt. It means you're never trying to break old habits, you start out by learning better ones from the word go. It also means that SMF mods will have to be rewritten, but honestly, that's a blessing rather than a curse.

No permissions issues where you have to open up all your files to higher permissions than they needed. You'll even be able to add and remove packages simply by uploading folders and removing them, without risking it just breaking randomly - but you'll have to visit the admin panel to enable it, of course.

It's also much smarter: instead of this version emulate madness and tying it to versions, the process is much closer to feature detection - add-ons declare the facilities they need to be able to use, and if those needs are met, it can be activated. (Add-ons can even, if they so wish, declare that they provide certain facilities, and other add-ons can indicate they want to use them, so you can build add-ons that have implicit dependencies)

Also if a plugin has specific requirements such as obscure PHP extensions, these can be indicated in the package itself preventing installation without them.

Naturally, all this stuff will be available through whatever we come up with as an add-on repository, so it becomes much clearer as to what's needed and what works with what.

Just for an encore, the add-on management will be able to figure out what languages are supported in an add-on, so we can also display that on the add-on repo, as well as displaying it in the core.


Add-ons will live in their own folder, which I expect to call Addons (and removing the old Packages folder), one folder per add-on. That way, a plugin's functionality is self-contained rather than filling up the Sources and Themes/default/languages folders with stuff.

Now, all this sounds wonderful, too good to be true in fact... except it isn't. It's not only workable, but a slimmer, less refined version is even built in to SimpleDesk 2.0 and has been since I prototyped it a year ago. I've learnt some lessons since then, some refinements to that design, and some of the changes we've made in Wedge make it possible to really do it justice.


I won't get into the innards right now, other than that Modifications.*.php files will be disappearing, that the add-ons still use XML to relate their details, much like package-info.xml but nicer, and that it'll have more in common with WordPress' plugins area than SMF's package manager.

I'll release a few more details once I'm comfortable and can show you a bit more about how it works.
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Re: Package Manager, how we won't miss thee
« Reply #31, on June 24th, 2011, 12:19 PM »
Another thing I never liked from the SMF Package Manager is it never tells you if your mods are up-to-date or if any of them has been upgraded and you missed it.

Normally when you open your Package Manager you see all your mods with a green light and you could think all of them are up-to-date. It happened to me at the beginning until I noticed I had to subscribe to receive notifications of updates of the mods I was interested in.

Are you planning to change that?

Re: Package Manager, how we won't miss thee
« Reply #32, on June 24th, 2011, 01:04 PM »
That's less a fault of the package manager and more a fault of the package serving architecture.

WP has this function, however, WP has it set up so that you are pretty much obligated to use their architecture for grabbing plugins. Consequently, if you're using their architecture you can use it to keep up to date. But the minute you start having any (and I do literally mean *any*) other service, you're pretty much telling users they have to keep things up to date themselves, as one of the caveats of not using the core architecture.

Now, I'm not exactly enthusiastic about having a single locked-in repo. Partly because of all the arguments about the 'walled garden' (personally, I'm not against it since it benefits users but it doesn't help developers much if those in control of the garden change the ruels)

But without a single master repo - and everything else being 'caveat user', so to speak - there's no really good, or reliable, way to manage that kind of update.

Re: Package Manager, how we won't miss thee
« Reply #33, on June 24th, 2011, 01:24 PM »
Sorry to butt in........


If your idea doesn't give every SM forum admin a stiffy (male OR female) Ill make a big donation to your hosting..... well £50 anyway :)

Billy

Re: Package Manager, how we won't miss thee
« Reply #34, on June 24th, 2011, 01:48 PM »
OK, I'll say it again. And I'll be more blunt about it.

I WILL NOT BUILD A WALLED GARDEN. I will not create an environment that forces a single source of mod production.

There are two reasons for this:
1. I did actually design and build my own mod repository for SMF once upon a time, where I could publish what I wanted without any restrictions imposed by any moral majority. I could, for example, have released a mod that allowed for viewing user PMs, if I so chose. Such a mod would have been forbidden by SMF, but the only rules I set for my site were my own. I would be trampling on the very freedom that Wedge was borne out of if I did not allow for choice.

2. Wedge opens a lot of doors, it represents a shift from the past. One of the things I was never happy about with SMF was its lack of premium resources.

The health of an ecosystem is reflected in the community that grows up around it. While I dislike much about WordPress, I respect that it has a lot of contributions from its community, in the form of themes and plugins, and there is a goodly number of premium resources. The platform is considered sufficiently viable that professionals are contributing to it, that it doesn't just represent a loose-knit collection of individuals contributing in their spare time and augmenting it with those who can justify spending professional-grade time and energy contributing to the ecosystem.

My stance is just that: if a platform and its ecosystem actively discourages professional-grade users from publishing premium resources, the sum total of its ecosystem becomes a collection of hobby contributors producing hobby contributions.

This is not, and should not be, construed as an insult to hobby authors - people just contributing code when they feel like it, is the backbone of any software-related community. But there is a glass ceiling of sorts: those who do it as a hobby invariably do not have the time and energy to commit to contributions in the same way paid professionals can do so. Being a hobby, they may not pursue the finer points of programming, and as such it's likely that - objectively speaking - the quality will be lower.

I should note that this isn't true of all ecosystems, nor is it true of all hobbyist programmers vs premium programmers. There are genius hobbyist programmers out there, just as there are absolutely abysmal premium programmers out there. But as a broad rule, if charging for something, the quality curve tends to be higher.

I do not want to design an environment that casts out the premium contributions. If it were true that people valued price over performance, we'd all be using SMF/phpBB/etc and no-one would be using vBulletin or IPB - because they are paid endeavours, there is much more incentive for them to keep developing, plus more incentive for them to add new things, because there's a reward in it other than the pure satisfaction of the thing.

That's another thing that's quite important: the reason *why* people do it. We're up front about why we're doing Wedge: because we're building it to be the platform we want it to be. When I wrote mods, I wrote them because they fulfilled a task I needed of them - but that was it. I didn't want to be dragged into ever-increasing circles of more features just because people asked me to - there was less joy in doing that, and I wrote things to work how I wanted them to, and I've lost count of the number of times I was asked to change what I did because it didn't suit someone else - if I wanted it to work the way they wanted, I'd have written it like that up front.

Paid contributions to the ecosystem actually swing the balance towards users, rather than leaving the power solely in the hands of the developer in their spare time: if there's a demand for a certain function/feature, it'll be written and if the market can stand it being charged for, market economics will allow it to.

There is an implied third reason that sums up the two together: us running a plugin repository that handles paid plugins puts responsibility on us. It brings up who is responsible for payments, who is responsible for security matters, and potentially would put us in legal liability situations that we have no desire to get into in the first place. This is why I didn't end up running smfmarketplace.com as I once envisaged.

Consequently, having an 'official repo' like sm.org has is nice but it removes the potential for premium resources and tying users into it brings a whole world of hassle in terms of walled garden. The last thing I want to do is shoot developers in the foot who like Wedge as a platform but may not care for myself or Nao in terms of personality (hey, it happens, we are perhaps 'abrasive' people)

That all said, I'm inclined to keep package servers in some form or another, but not necessarily updating magically. Any plugin that really needs it (like SimpleDesk, SimplePortal etc.) can implement their own method readily enough.

Re: Package Manager, how we won't miss thee
« Reply #35, on June 24th, 2011, 10:30 PM »
Would it be considered a walled garden if you went Cydia style?  This would allow Wedge forums to check your repository for updates as well as mine if I were to build one and then let the community know I had a repository that was automagically compatible with Wedge and all they had to do was point their forum at my repository.

Not that I have those skills but for others.

Re: Package Manager, how we won't miss thee
« Reply #36, on June 24th, 2011, 10:35 PM »
What happens, then, if the same plugin is in two different repositories? Which one should be used? Should I have to track where it's come from? What happens if you install it manually (which is possible), how should it phone home then?

I have already spent quite a bit of time thinking about this, I'd already long considered Cydia style too - but because you're getting it via the store in the first place, there is a mechanism for tracking from its source.

It still doesn't solve the issue of paid repositories, either.


Re: Package Manager, how we won't miss thee
« Reply #38, on June 25th, 2011, 06:27 AM »
My thoughts here, and not pushing anything I'm just brainstorming a little so if you've already discarded this or don't wanna talk about it anymore you can tell me to shut up already.  :)

If its built to check whatever repository the forum owner has pointed it to, then it can check all repositories for version numbers and update from anywhere that has an approved package in that repository.  Really, most people with other repositories won't be mirroring you, they'll be doing their own thing for their own projects both paid and free that they want control over.  If they want a paid option then they can figure out access to paid repositories or you can use group level access where paid members go into a group with access to paid repositories or whatever.

Like I said though, I'll shut up if you don't want to think about this any more.  I was just enjoying thinking out something that didn't involve a diaper.  :)

Re: Package Manager, how we won't miss thee
« Reply #39, on June 25th, 2011, 10:11 AM »
Quote
Really, most people with other repositories won't be mirroring you
You'd think so, but the reality does not bear that out. The repository of mods on SMF Hacks.com contains a surprising number of duplications between that and sm.org's repository. That said, the number of package servers for SMF is rather small and doesn't really work that well... off the top of my head the only package servers I can think of, in total:

* sm.org's own
* SMF Hacks.com
* the one I used to run
* Niko runs servers for his projects, one per project I believe
* SimpleDesk has one

And even there was duplication. I already mentioned SMF Hacks, mine had updated versions of some already on sm.org, and SimpleDesk's serves SD plus mods that were contributed by the team as a whole, all of which are on sm.org too.
Quote
Like I said though, I'll shut up if you don't want to think about this any more.
It's not about that - it's simply that I've been over this ground a lot and haven't found any solutions that tick all the boxes, so to speak.


That said, in amongst the headache pain of the last day or so, I did have one idea about how it might work, including even paid stuff, just need to make sure it's provided for in the system, and interestingly it does follow the same sort of idea: instead of doing the Cydia thing and tracking where it comes from, have a package identify itself with a unique ID (which, of course, relies on packages actually pushing a unique ID in the XML, but that's another matter), and query the servers for the packages they hold, matching against that unique ID and version number.

I realised that while I'd included an SMF-style ident in the package, it's not actively being used for anything so I'd be better off using a GUID instead there which for all intents and purposes would be unique and could then be trusted to be used to track whether a given package is available on a given server.

As for paid repositories, certain things hold true with those: most notably that it's done by access groups on a forum. All we do is provide the username and password - suitably hashed, of course - as part of the query. If anything I'd probably make a package server respond to an XML-RPC query rather than just shoving a massive XML file (hint: when you do the browse thing on sm.org, the packages.xml file is served up and processed in your server... last I checked, that topped 1.1MB of XML)

Re: Package Manager, how we won't miss thee
« Reply #40, on June 25th, 2011, 10:53 AM »
Suggestion. Maybe already done.
Have packages put into their XML the URL to either the official package or another XML with the URL and version number.

Re: Package Manager, how we won't miss thee
« Reply #41, on June 25th, 2011, 11:06 AM »
The XML already has the version and name (which it requires), and provides space for the author and the author's site, but rather avoid putting call-home links like that in the package.

While I don't want to limit people making package servers if they want, I am slightly hesitant to encourage the world and his dog to make one, which is principally the problem I see with the Cydia model, you'd end up with mods so fragmented rather than sane distribution methods.

The XML having a GUID and version number means it has some concept of self-awareness (i.e. I am <big long identifier that's unique to me>) and that can be used to query the servers that a given install is about. I'm not averse to a package being able to indicate package servers though.

(And, at the end of all this, I get to remove an SMF table, yay!)

Re: Package Manager, how we won't miss thee
« Reply #42, on June 25th, 2011, 11:12 AM »
Okay, please do not drop version detection entirely. Make it important to have versions, because many times feature detection itself isn't enough. Because one cannot detect a specific version of a mod depending upon feature(It may have same features, but different hooks), or their may be an important bug fix required for the add-on which is based on version detection.

And if a mod has multiple versions in multiple repositories, just show all of them(Highest version first). Simple as that, do not make it important to have a repository hosted mod but please push the repositories to be functional and mainstream. Cydia does handle paid mods and it works quite nicely, one just needs an auth to get in.

Re: Package Manager, how we won't miss thee
« Reply #43, on June 25th, 2011, 11:26 AM »
There is version detection for PHP and MySQL for this exact reason because you can't always validate functionality by existence of given functions there, but I don't see why you need to do version detection otherwise: if a mod provides hooks x and y, competent developers will keep the hooks working the same way other than bug fixes between versions - that's the whole point of having an API by definition, and the whole point is that it should be reasonably abstract, the calling code shouldn't have to worry about the exact definition provided that the API signature is the same.

In all honesty I don't expect many mods to offer hooks anyway.

Give me an actual example, not a contrived one, that validates this point and I'll tweak it but I'm really, really not wanting to go down that road if I can help it: one of the greatest weaknesses of SMF is that mods have version number dependencies when they really shouldn't have it - how many mods exist for 1.1.x that will work on all versions of 1.1.x but because the author hasn't updated the version number, leading to "can someone update this for 1.1.14?" posts? Far, far too many - and I'd rather get away from doing it.

From my perspective, if a mod author is smart enough to be writing their own hooks, they're probably smart enough to figure out how to write version-free code.

Re: Package Manager, how we won't miss thee
« Reply #44, on June 25th, 2011, 11:28 AM »
Will you be registering the hooks that a mod calls for validation purposes?
Posted: June 25th, 2011, 11:28 AM

Oh and do you plan on providing delta updates?

Re: Package Manager, how we won't miss thee
« Reply #45, on June 25th, 2011, 11:53 AM »
Quote
Will you be registering the hooks that a mod calls for validation purposes?
Yes, plus the hooks that a given mod provides.
Quote
Oh and do you plan on providing delta updates?
You mean like SMF's 1.1.13 -> 1.1.14 patches? Probably not. (WordPress doesn't and their mods don't exactly do version checking either.)