Plugin servers / getting plugins to a system

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Plugin servers / getting plugins to a system
« on October 13th, 2011, 06:19 PM »
So, this is one of the things I've been thinking about lately, is how to make plugin servers work, on a technical level.

This is going to be something of a stream of consciousness rather than a focused discussion, to start with. It's more about trying to make sense of all my thoughts, and if you have anything to add, please do.

Let's look at what SMF has (not what it actually uses). It's all in the Packages > Download Packages page.

Up front that's stupid. You click a page called Download, so that you can upload something. It's almost as backward as the logic of clicking a Start Button to shut down, though at least they have the argument of 'starting the shutdown process'.

Idiot question of the day... should uploading plugins from your computer be a separate page to downloading from a plugin server? Gut instinct says yes, it should, because they're totally different options and processes.

That also leads in nicely to something Dragooon mentioned, about plugins having an 'app storefront' kind of deal. I've mentioned before that I'd like to be able to support the environment where creators can publish paid plugins, and still make use of something like the package server facility, currently not possible in SMF. (Incidentally I was thinking about that almost 18 months ago, if not further back)

Hmm. Seems to me that the actual process of getting a list of plugins is at least as dependent on what interface to use to display, as anything else. SMF provides the ability to accept a list of plugins, and/or a list of categories, wherein you can browse categories. Thing is, that most people don't realise, is that you can also segment the package server list by category so you only need serve a list of what categories there are, then what's in each one - as opposed to the behemoth served by SMF currently that actually breaks some installs with low memory limits.

Then again, that's when you have a public list. If the list is potentially private or semi private, or in any way filterable, it needs to be accommodated by the server, not the client, which means that just serving a packages.xml type file isn't an answer. Hmm. I'm sensing that you'd make a request on a given page much you do with feeds, and make it essentially similar to a RESTful style implementation, that the URL http://example.com/forum/index.php?action=plugins would provide a list of plugins available for download, and adding ;cat=x would filter the list by the specified category. Of course, that would be a detail provided by the server.

I'm a little wary of providing something that amounts to a storefront structure but I'm OK with the architecture providing lists of what's available and so on, especially as for anything that's premium, you'd have to be a member of their site anyway and have access to the relevant downloads.

Hmm. Should a plugin be able to list the plugin server or servers where it can be found?

Also, what exactly is the process for plugins requesting updates? I'm sensing it would be a daily process job, where the system goes to all the plugin servers it knows, with whatever credentials it has, and supplies a list of plugins. (Active plugins, all plugins?) The server itself should be doing the work on filtering down plugins, not the client. Though, if the plugin itself indicates a server or servers where it can be found, presumably only those need to be queried?

One last thing. Do we really need to support both .zip and .tar.gz? Gut instinct says no, just zip will cover the bulk of what's actually needed? (even if it does provide a shade of inconvenience for those who use *nix / OS X that write plugins... which is going to be a limited enough group anyway?)
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

live627

  • Should five per cent appear too small / Be thankful I don't take it all / 'Cause I'm the taxman, yeah I'm the taxman
  • Posts: 1,670
Re: Plugin servers / getting plugins to a system
« Reply #1, on October 13th, 2011, 07:52 PM »
Quote
Do we really need to support both .zip and .tar.gz? Gut instinct says no, just zip will cover the bulk of what's actually needed?
I use the gzipped tars because SMF randomly has problems with ZIPs.
A confident man keeps quiet.whereas a frightened man keeps talking, hiding his fear.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278

live627

  • Should five per cent appear too small / Be thankful I don't take it all / 'Cause I'm the taxman, yeah I'm the taxman
  • Posts: 1,670

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Plugin servers / getting plugins to a system
« Reply #4, on October 13th, 2011, 09:20 PM »
Yeah, that's something that concerned me. That said, there is a PEAR library equivalent that can be safely used if needed.

What about the rest of my stream of quasi-nonsense? :P

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: Plugin servers / getting plugins to a system
« Reply #5, on October 13th, 2011, 11:26 PM »
Quote from live627 on October 13th, 2011, 07:52 PM
Quote
Do we really need to support both .zip and .tar.gz? Gut instinct says no, just zip will cover the bulk of what's actually needed?
I use the gzipped tars because SMF randomly has problems with ZIPs.
How so? read_tgz_file/read_zip_file has always worked for me...

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Plugin servers / getting plugins to a system
« Reply #6, on October 13th, 2011, 11:48 PM »
There were issues with it concerning 64 bit builds, specifically there was a bug in the way it tried to load zips made on a 64 bit build where phantom offsets were introduced and used.

I was of the understanding that they'd been fixed though.
Re: Plugin servers / getting plugins to a system
« Reply #7, on October 14th, 2011, 01:24 AM »
OK, let's recap.

.zip vs .tar.gz.
There are more rugged solutions than read_zip_data() out there, one of which is ZipArchive. It's not the only solution, however, and it won't be that hard to find such a solution.

My theory of using only one file type in general just means that it simplifies the process and the code, and minimises the overall number of hassles people have, IMHO.

Should uploading plugins from your computer be a separate page to downloading from a plugin server?
I'm thinking that getting from a plugin server would be a 'Find' page, and uploading from your computer would be an 'Upload' page, as Plugins > Find and Plugins > Upload.

Should a plugin be able to list the plugin server or servers where it can be found?
This one is totally dependent on the expected prevalence of plugin servers. There are worryingly few for SMF, which has pretty much always amounted to people running their own for large projects. I think it's still in single digits.

If there's a (relatively easy) way to specify a plugin server it would help matters, but the more I think about it, the more I think a plugin shouldn't specify its home as a target for lookups, not directly. (As in: we can't expect plugin authors to expressly put in the Wedge plugin server generally, but the plugin system can query all the servers it knows for updates)


I think I almost have to just go away and start building before I can really answer some of these questions, that I won't answer them ahead of time.

(Just a quick note, I took a look today at how WP handles plugin updates on the fly. Interestingly, it's practically the same as I'd outlined Wedge would have to be doing, that the download occurs, the old plugin is disabled, the new one enabled, and the old plugin then deleted. I still think the user should perform those steps, though, rather than entirely automating it. I don't know. Might have to experiment with how well that works out for us.)
Posted: October 14th, 2011, 01:15 AM

Oh, and I'm going to have to write a plugin server of my own before long, as I've always (always!) kept my webserver's folder as a copy of the files, rather than directly hosting SVN files. In this case, C:/Dev/wedge_plugins currently holds all my plugins as a SVN tree, but C:/Dev/public_html/smf/wedge/ is where my testing site is... would be nice to have a solution for grabbing plugins that was quicker than just copying folders (and, as a side matter, didn't include .svn folders)
Re: Plugin servers / getting plugins to a system
« Reply #8, on October 16th, 2011, 11:53 PM »
Bump for feedback. I've been doing work on other parts of the system that will feed into this... before long I'm going to have to implement this and would rather do it once and do it right...

DarkLite

  • Posts: 3
Re: Plugin servers / getting plugins to a system
« Reply #9, on October 17th, 2011, 12:15 AM »
I've spent a depressing amount of time lurking as a guest just to follow Wedge's progress, and thought I'd throw in my £0.02 here.

While I agree that in principle downloads and uploads are two very different things and would seem to require two pages, I don't think the upload page has enough content to merit a separate page. I can't envision anything other than a single "choose file" dialog (but there may well be other possibilities I haven't thought of?).

As a result, is it really worth splitting it off into a page with one button and no options? It'd look somewhat odd to the average end user, given that no other admin page contains so little stuff (even the Languages->Settings page has some options).

What about rebranding "Download packages" as "Add a package", rather than splitting it up? I think that would cover both "upload" and "download" scenarios, with the uploads being handled by a button at the top and the downloads by the server browsing system below.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Plugin servers / getting plugins to a system
« Reply #10, on October 17th, 2011, 12:30 AM »
That makes sense, probably more than splitting them up, though the download process is likely to be more powerful than before, so I was also debating putting the whole 'package server' management into its own page, because it does need a shade more in future.

There is actually one page that does have so few things on it - the caching page. For most of the options therein, you don't get any configuration other than to pick caching type.

(Also, welcome to Wedge :))

DarkLite

  • Posts: 3
Re: Plugin servers / getting plugins to a system
« Reply #11, on October 17th, 2011, 12:51 AM »
Quote from Arantor on October 17th, 2011, 12:30 AM
That makes sense, probably more than splitting them up, though the download process is likely to be more powerful than before, so I was also debating putting the whole 'package server' management into its own page, because it does need a shade more in future.
The upload system wouldn't take up much screen real estate: would there really not be room for a "choose file" button at the top of the page?
Quote
There is actually one page that does have so few things on it - the caching page. For most of the options therein, you don't get any configuration other than to pick caching type.
In the case of the caching page, it's got options. A page consisting of a single button offers no choice, and I don't think most people would see a reason for the page to exist (particularly when there are other closely related pages it could be merged into).
Quote
(Also, welcome to Wedge :))
Thanks!  :D

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: Plugin servers / getting plugins to a system
« Reply #12, on October 17th, 2011, 12:55 AM »
Quote
The upload system wouldn't take up much screen real estate: would there really not be room for a "choose file" button at the top of the page?
Take a look how big the current download page is. It doesn't fit on the screen as it is, and it's going to get more complex in future, especially with the browse stuff being intentionally more prominent.

Plus, I do want to put some notes on the page about people uploading things manually (since there's little real difference, but some people are funny like that, and it's possible that the normal code won't be able to run for whatever reason, however unlikely)
Quote
In the case of the caching page, it's got options. A page consisting of a single button offers no choice, and I don't think most people would see a reason for the page to exist (particularly when there are other closely related pages it could be merged into).
A page consisting of a single button might not be too much use, but there must be at least two buttons (a pick-files and a submit button). I did also consider making it possible to upload multiple plugins at once should that be so desired.

Nao

  • Dadman with a boy
  • Posts: 16,082

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278