Wedge

Public area => The Pub => Plugins => Topic started by: Arantor on January 21st, 2013, 12:00 AM

Title: Plugin server
Post by: Arantor on January 21st, 2013, 12:00 AM
So I've been wrangling with this, and I don't see any way around what needs to be done for this.

Specifically, as nice as it is using Aeva, I don't believe that we can meaningfully use that on a production site, because there's a bunch of meta information that's required, not to mention the fact that the nature of private repositories requires providing a login and ideally without having to actually do the usual run of actually logging in.

Not to mention the nature of serving plugins and so on carries a performance hit that I'd rather not carry on the main server (especially as in the future I plan to have plugins query for updates and that's going to be expensive). We can use a subdomain if we wanted to, I also own wedgeplugins.com/.net/.org just in case.

So, let me explain the entire nature of the plan I have, and then we'll deal with questions.

I plan to run a separate plugin repo, on a separate Wedge install to here. This is to isolate performance issues. In fact, there's no reason why - with what I'm about to say - that it even needs to be on the same physical server. From a performance perspective it may even be better if it is not. I'll come back to this in a minute.

So, plugin server serves three types of requests from repos, it serves: simple list of plugins (be that in categories or otherwise), actually serving the plugin, and searching for plugins.

The architecture I have in mind will even allow for private plugins and so on, quite happily, the user provides their username/password when adding the repo. It sends the details when making a request as part of the request (in a very encoded form, of course), and authentication is done there.

As far as the 'separate server' gig, that's not a huge deal. Some people here will be familiar with the concept of IP.Connect; this is the principle plan I have, to use the second server as a slave using the IP.Connect protocol to connect to this site (this means you have two sites that have conjoined accounts in a sense)

Alternatively if third party sites want to set up their own repos, that's an option too.

Mechanically, we're talking about a plugin that accepts requests to action=plugins and handles the requests from there. Mostly that's straightforward requests, but here's the clever part: we can easily make this a complete 'mod site' as was, including comments/support, approvals or whatever and creating boards/topics as needed.

It just seems to me that doing it on a separate site is actually a stronger proposition than trying to manage it here in the long run, especially in performance terms. Given how much trouble people have with sm.org, it actually seems to me that having proper separation is actually more of an advantage than not separating it.

I'm also thinking this will help shift some of the 'perceived liability' especially if this other site becomes a sort of marketplace of sorts, which is ultimately where I hope to go with it.


Any questions?
Posted: January 20th, 2013, 11:57 PM

I should add, it would not be terribly difficult for me to drop all repo handling and just leave it to people to browse the site etc. But this seems more professional to have it integrated. I would not lose *that* much sleep if the opinion were overwhelmingly against worrying about having integration into the core.

Most of this is stuff that needs to be done either way, though.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 02:01 AM
Hmm, I think I went too big at once. Most of the plugin stuff is going to be fairly similar in nature to SMF's package server mechanically, the authentication side is almost just a sideline in a lot of respects.

The only things that really need discussion are:
1. Using/not using Aeva, though I think I made my position on that fairly clear

2. Creating a separate physical site to handle plugins and so on. Don't know if you've seen sm.org's comments about being busy, but a vast amount of their time/effort is actually spent handling plugins, serving plugin listings etc. for users, and the package parser and it's all attached to the main database there.

I believe having a separate physical site will mitigate the load considerably.
Title: Re: Plugin server
Post by: Auk on January 23rd, 2013, 04:17 AM
I thought Aeva is a media embedding tool... I have no thoughts on it, nice tool though.

As for creating a separate physical site to handle plugins and so on... Separate server/website would also ease unnecessary confusion. I can imagine topics spawning in wrong parts of the forum, the plugin sub-forum would bloat from "Help on how to make plugins" into "Help!! How do you change the flashing color for the New Private message plugin" forum. :S


I hope at the plugin managing software (linux respo style) would load some screenshots of the plugins, as well as url to demo if available. Only on the main respo (not hosted by third parties, but they should too) have a standard for making screen captures available.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 04:26 AM
There's two versions of Aeva - Aeva Lite, and Aeva Media. Aeva Lite is purely the embedder, Aeva Media is the full gallery as well - and both are built into Wedge. Aeva is what has been used here thus far to serve the alphas and all the plugins we already wrote.

The plugin repo site will be similar in a number of ways to SMF's mod site, but with a few other... twists... in mind that will likely be pretty much unique to the main site.

Demo is a bit trickier to pull off, because ain't no way I'm giving out admin access to the admin panel to access things ;) But I can well imagine there being a link to a site where you can see it.

I should add, some of the plugins on the site I have in mind will be paid-for plugins. Exactly how that's going to work, I don't know yet, but that's what's going to happen, e.g. some of the bigger and rarer-used packages I have in mind will be paid-for plugins. But I'm building the add-plugin stuff to allow for repositories where you have to provide a username/password combo anyway.
Title: Re: Plugin server
Post by: Auk on January 23rd, 2013, 05:05 AM
Yes, an external url to their own demo of course. It would be very sloppy if hosted on that server. On that subject, there should be:
1. external url
2. "Invisible" url to a blank file stored on external demo site. (So an automated code can run a monthly demo-url checker which can auto mark demo dead and hide external url, not sure if there is a way to write that code without performance issues.)

For paid for plugins...
You should enable third party (user/password) to connect to your server using an automated script to store a key or something. Like this:
a942-b302c5b0d-1d75fa8-95f9e3d-7de-7c

1. Someone purchases license to use my plugin. My registration form sends this key to your server, also emails the user: a942-b302c5b0d-1d75fa8-95f9e3d-7de-7c
2. The end user who wants to use my plugin connects to your respo using this key.

* Note: This is only a key to simply download plugin... This is not to activate plugin.. The author is responsible of coming up with instructions/howto in regards to software activation. Especially since it would be capable of being hosted on 3rd parties anyway.

The username/password combo should only be left with software authors. The key thing is basically already a username/password, but it's easily disposable so respo owners wouldn't need to worry about it.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 05:15 AM
Quote
Yes, an external url to their own demo of course. It would be very sloppy if hosted on that server. On that subject, there should be:
1. external url
Eh, just ping the server once a week/month/whatever and if it returns a 404, it's dead. No extra effort required.
Quote
For paid for plugins...
You should enable third party (user/password) to connect to your server using an automated script to store a key or something. Like this:
a942-b302c5b0d-1d75fa8-95f9e3d-7de-7c
OK, I did explain how this is going to work, knowing as I do the realities of pirated software.

I can't prevent nulled copies getting out there. It sucks, but it's true. It's not hard to find nulled copies of anything if you try.

On the other hand, I really don't want to get into the rabbit hole of licence key activation for the simple reason that it can and will put legitimate people off. As it is, InvisionPower marks my IPB test site inactive periodically and I have to go and re-validate my licence with them. Not even kidding >_<

It is my belief that it actually limits people unnecessarily more than it prevents piracy. In any case, no support without valid licence. And that's easy enough to deal with as it comes up, just as it is on sm.org with pirated DzinerStudio - ask them why they're not asking for support on the DS board. And lock it when it inevitably becomes clear.

As far as licensing goes, I already figured how I intend to do it. The idea is that you save the username and password in your forum (and they're hashed, of course) for your account on the plugin server, and then your install contacts the plugin server, providing authentication details, and the list of plugins you get back is what you have on your account.

Other stuff, like auto checking plugins for updates... not yet figured out how best to do that.
Title: Re: Plugin server
Post by: Auk on January 23rd, 2013, 05:51 AM
You're thinking of how frequent plugins should be checked for updates eh?  I think it should simply look at version number, or hashed version number (e.g. alpha -> 2c1743a3)

1. Maybe in an interval of every 2-3 weeks (from end user).
2. And... Whenever an admin logs into ACP if that's not an issue (Only counting the first admin who does so), some info in the forum-databse for "last-checked-update" numeric timestamp will be updated and it will never change again until the next day or two.

I believe that's reasonable. Must end users be hurriedly informed of updates anyway?
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 06:00 AM
Quote
You're thinking of how frequent plugins should be checked for updates eh?  I think it should simply look at version number, or hashed version number (e.g. alpha -> 2c1743a3)
It doesn't matter... it *still* has to do that call home but it's not so much the frequently that's the issue. Potentially to multiple plugin servers - and right now it has no way of knowing which plugin came from where, so if it goes specifically asking, especially if it's providing credentials, there's a privacy issue there (because someone operating a third party repository could conceivably then log who is accessing what, which if it checks every site for everything is an issue)

The alternative is for a plugin server to say what plugins it has and their versions and then the local Wedge install queries for what it wants to update.

For small deployments, this isn't a problem. It only becomes a problem when it scales to SMF size, when you're into the multi-MB territory for plugin listings. (The XML file that indicates 'New Feature' plugins on sm.org is a 1.1 MB file listing the hundreds of plugins) Now, I can mitigate it somewhat, but it isn't going to be pretty. There's also potential issues in terms of memory use while doing this.

This is why, interestingly enough, WP only really allows for its own repo, specifically to avoid the privacy issue of reporting what plugins are installed on a server.

Title: Re: Plugin server
Post by: Nao on January 23rd, 2013, 02:48 PM
- IPS Connect: I'm aware of this technology but am not sure how it works, especially when it comes to security. Doesn't this amount to simply transmitting authentication data between two servers...?

- Having another server for it: it doesn't prevent us from putting its data into 'plugins.wedge.org', for instance. A nice place to look at... Also, it potentially fixes cookie issues, although I liked the idea of having subdomain-specific cookies, allowing for cookieless subdomains to be designed. Although I guess it's better suited for a specific extra domain like 'wedgecdn' or whatever...

- Using AeMe or not: unsurprisingly, I'd push for the use of AeMe, if anything because I can help improving the software as well (I haven't touched it in a long time, but just because I'm rusty doesn't mean I can't be efficient with it). What's the alternative? Using SSI? To me that's mostly a waste of our valuable time if we have to rewrite a complete file server when we already have something good with AeMe, *and* it gives up an opportunity to improve AeMe's download system as we find new bugs or limitations by using it to distribute plugins.

Hope I didn't forget anything... I'm not very well versed in plugins as you know, so I tend to shamelessly avoid anything with the name 'plugin' in the topic title... :-/
Title: Re: Plugin user interfaces
Post by: Nao on January 23rd, 2013, 03:13 PM
For mental health safety reasons, I'd strongly suggest going for the post-review process.
i.e.: (1) author gets to upload plugin directly, (2) any users (or maybe a limited group of users, but not 'Customize team' :P) get to try it out and share comments or give a rating, (3) in case any issues (security mostly) are found, report to moderators.

It would be more... realistic. Although in the beginning, I should assume there'll be less plugins to play with.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 03:14 PM
It is a bit more than that, you have a master and 1+ slaves, where the slaves look for a local account and failing that query for the master, and create an account, plus keeping them in sync when there are password changes. Having a separate server within the same subdomain doesn't solve auth issues, as such, since the second install still has to cope with dual DB stuff. The only way around that is to run it off this server, which for stated reasons I do not want to do.

I did buy wedgeplugins.com etc just in case but I'm not averse to plugins.wedge.org.

Of a plugin server, very little of it is actually related to the files themselves. All the effort is around metadata, stuff that Aeva not only doesn't need to do, but should never have to even consider. In theory I could solve it by making Aeva drastically more hook able but that's not really the point either. I'd still need to write a ton of handling anyway. The workload is about the same in either case. I do not expect to find much, if anything, in the way of improvements on top of Aeva with this stuff.
Title: Re: Plugin server
Post by: Farjo on January 23rd, 2013, 03:29 PM
Er, I don't really understand any of what's been written so far. I'd just like to push for something that makes searching easier. On SMF I've bookmaked http://custom.simplemachines.org/mods/index.php?action=search;smf_versions[]=68;bool=and;sort=modified;desc to get the latest 2.0.3 mods (clever ain't I :eheh:). And it would be good to be able to search for those that are genuinely new rather than merely updated, or those that are now available for my version. Also searching on a word or phrase is hit-and-miss. (Sorry if this don't belong here)
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 03:31 PM
That's the other part of it, searching needs to be part of the deal, and for things Aeva doesn't really apply to.
Title: Re: Plugin server
Post by: Nao on January 23rd, 2013, 06:29 PM
Re: search, I think AeMe can search for items, including in custom tags..? (fox.noisen.com uses AeMe and as you can see, it's a full-featured music database. I think all of the code is handled by AeMe...)
Quote from Arantor on January 23rd, 2013, 03:14 PM
It is a bit more than that, you have a master and 1+ slaves, where the slaves look for a local account and failing that query for the master, and create an account, plus keeping them in sync when there are password changes. Having a separate server within the same subdomain doesn't solve auth issues,
It solves cookie issues, at the very least. I thought you meant having the same database for both sites... But if you're going for two different databases...
Quote from Arantor on January 23rd, 2013, 03:14 PM
Of a plugin server, very little of it is actually related to the files themselves. All the effort is around metadata, stuff that Aeva not only doesn't need to do, but should never have to even consider. In theory I could solve it by making Aeva drastically more hook able but that's not really the point either.
Oh, that's precisely the point... I'd love for AeMe to REALLY be part of the Wedge ecosystem, unlike now where it's been forcibly wedged into it.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 06:33 PM
No, I physically meant two separate servers for load balancing purposes.

Yes, Aeva can search, but it does not have any of the metadata processing I require, nor can the search search that metadata, and I know I can build it to run faster with a custom geared database designed for that purpose.
Title: Re: Plugin server
Post by: Nao on January 23rd, 2013, 06:42 PM
Quote from Arantor on January 23rd, 2013, 06:33 PM
No, I physically meant two separate servers for load balancing purposes.
I'm okay with that, but only if you're responsible for the plugin server :P
Quote
Yes, Aeva can search, but it does not have any of the metadata processing I require, nor can the search search that metadata, and I know I can build it to run faster with a custom geared database designed for that purpose.
I built AeMe around Foxprog, you know :P I already had a 'working' website before I plugged AeMe into it, and it didn't take long to do. It's very doable. I'm just saying, no need to waste time on this when it's already there. Building a site from scratch is... not fun.
But it all depends on whether you WANT to do it.
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 06:45 PM
As I said, there's still a ton of architecture to implement around it anyway, whether I use Aeva or not.

It would be quicker for me to just do it, plus I can set up proper mod support areas that are more than just a single topic... Without it impacting here ;)
Title: Re: Plugin server
Post by: Nao on January 23rd, 2013, 06:51 PM
As you wish... ^^
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 06:54 PM
Inconceivable![1]
 1. The Princess Bride, if you're wondering.
Title: Re: Plugin server
Post by: MultiformeIngegno on January 23rd, 2013, 07:45 PM
We could also have wedge.org/plugins and /plugins is on a separate server (at least with nginx that's possible)..
Title: Re: Plugin server
Post by: Arantor on January 23rd, 2013, 08:04 PM
The URL is the least of the problems, really ;)
Title: Re: Plugin user interfaces
Post by: Arantor on January 24th, 2013, 03:46 AM
Quote from Nao on January 23rd, 2013, 03:13 PM
For mental health safety reasons, I'd strongly suggest going for the post-review process.
i.e.: (1) author gets to upload plugin directly, (2) any users (or maybe a limited group of users, but not 'Customize team' :P) get to try it out and share comments or give a rating, (3) in case any issues (security mostly) are found, report to moderators.

It would be more... realistic. Although in the beginning, I should assume there'll be less plugins to play with.
I should add, I split this off from the plugin interfaces topic, it's more relevant here.

I can see it both ways, but I sort of need to make the decision now because it's all part of the DB structure.

On the one hand, pre-review means there is a better than nothing chance of preventing nasties. It also creates more work, primarily for me. But I know how long it takes to vet plugins, and there's all kinds of neat tricks that can be done to help with that. We can do all kinds of fun like validating plugins on upload, checking the plugin-info.xml file is valid, we can check the hooks that it requires, we can check the files for syntax errors, you name it, it's all doable.

And that's before we even get to human review. In fact, I think I'd be inclined to enforce that even if we go to post review so that nothing appears on the site to regular users if it hasn't at least passed a basic validation.

Pre-review does also put some weight on whoever is reviewing to do a decent job, but it does require also discipline in doing it regularly, not to mention having naturally high skill requirements.

Post-review lowers the workload considerably, of course, plus removes the liability ("but the team should have checked for that") but it raises the chance of someone getting something nasty through the doors.

I'm reasonably amenable to either, though in the case of post-review, I would make it clear that I would be moderating after the fact and reserve the right to remove plugins that are dangerous, and I suspect I will even be doing some post reviews at some point myself, to validate what's going on. Of course, that leads to having some kind of badge related to 'vetted by' status to provide some kind of stability to people who want that kind of thing.

Of course, plugins pushed out under the team banner (i.e. official plugins) will automagically have something special to indicate it.

OK, so let's go with the theory of post-review rather than pre-review. What should happen with beta plugins? Plugins that people want others to test prior to going live, that is - should they automatically just push them to the main repo, but perhaps with a 'beta' tag on them?
Title: Re: Plugin server
Post by: Nao on February 24th, 2013, 02:38 PM
Post-review is the way to go... Always has been.
With a badge system for plugins. Always has been.

'Official plugin' for team plugins
'Outstanding plugin' for non-team plugins the team loves
'Reviewed plugin' for non-team plugins the team has tested and can attest will work, but do not particularly recommend over anything else.

Of course we don't care about the badge names, it's just that we need to differentiate them...
Title: Re: Plugin server
Post by: Arantor on February 24th, 2013, 02:52 PM
Works for me.

I do know one thing. I already know that the plugin server has a code name. Wedge is the core software, Inclined Planes is the plugin server :D
Title: Re: Plugin server
Post by: Nao on February 24th, 2013, 02:57 PM
Now that's a silly name... :niark:
Title: Re: Plugin server
Post by: Arantor on February 24th, 2013, 03:01 PM
It seems perfectly logical to me, it is yet another 'simple machine' ;) I couldn't decide whether I preferred 'Pulley' or 'Lever' for the translation tool on this site. Now THAT's a silly name :lol:
Title: Re: Plugin server
Post by: Dragooon on March 17th, 2013, 09:23 AM
Not sure if you're still looking for opinions but I'll throw mine around anyway. Paid plugins has always been a little grey area when it came to open source projects, do you plan on officially supporting them?
Quote from Nao on February 24th, 2013, 02:38 PM
Post-review is the way to go... Always has been.
With a badge system for plugins. Always has been.

'Official plugin' for team plugins
'Outstanding plugin' for non-team plugins the team loves
'Reviewed plugin' for non-team plugins the team has tested and can attest will work, but do not particularly recommend over anything else.

Of course we don't care about the badge names, it's just that we need to differentiate them...
^ This.
Title: Re: Plugin server
Post by: Arantor on March 17th, 2013, 02:58 PM
Yes, I do plan on officially supporting them. Most of the headaches around the plugin server stuff revolve around authentication to the plugin server to validate that you can download them.

I intend to write some, too.