Post by: Arantor on October 21st, 2011, 12:03 PM
I'm not quite at the stage of implementing this yet so it's not like I can't change it but something occurred to me that I would like some feedback on.
Imagine a large repository, of a couple of thousand plugins - like sm.org has. It's realistic to expect that Wedge will get to that point sooner or later (preferably sooner!), and at that point it starts to be unviable to be requesting a list of everything the server has - it's not good for the server, it's not great for the client either.
So I conceived of the notion of having the request actually include a list of plugins that it has available, and the server responds with a list of what it has in it that matches by id, so you know whether to update or not. This idea is not unknown, WordPress does it, for example.
The difference is that WordPress only supports its own repository, not third party repos, in the core code - we expressly support third parties.
Here's the problem. While I can guarantee that my own server(s) won't log requests[1], I can't guarantee other servers won't do the same, so that if a list of plugins is sent, it would be possible to log what plugins you have.
And, naturally, there is a privacy concern here as a result.
I should note what I'm planning on logging as a core feature of the plugin server system (the actual server facility itself) - it should be able to log downloads (total and per day downloads), and if a plugin is secured by group access, i.e. it's not open to guests or regular members, I think it should also log the name and time of download as well as the user who downloaded it.
I think the above logging should be provided because it's useful to see how many people are downloading plugins, whether they're still being downloaded currently (or just historically), and if it's secured behind some kind of authorisation, it should be able to log who's downloading for the simple reason that you're offering a product behind some kind of restriction and you should be able to identify an account or accounts that are using that facility.
Thoughts?
| 1. | This should include wedge.org, though there is a valid logic behind doing it here too, which if done anonymously allows us to see which plugins are popular, not just downloaded but what is downloaded *and still being used*. |
Post by: Nao on October 23rd, 2011, 12:00 PM
I'd tend to be cool about all of this and just don't bother with third parties logging stuff.
I mean it's their choice... Might simply need to add a warning about that when adding a new server, I dunno.
BTW, can plugins specify their 'owner' server? e.g. you download a plugin from the Wedge server, but all subsequent versions are downloaded from the other server.
Also, we really need to determine a way to easily upgrade Wedge for users... I'd like to have some script that goes through the svn repository and stores a list of file additions, removals and modifications for each revision, and sends the latest files to a server after being provided a current revision ID...
Post by: Arantor on October 23rd, 2011, 06:08 PM
On the whole concept of looking up updates, that could actually be done separately by just requesting a list of plugins and versions available with their required hooks, and letting the client figure out what can be updated. It doesn't have the same privacy issue but the bandwidth use is naturally increased.
That said, if it is simply a list of author:pluginid:version:required-hooks, one per line, that might not be too bad. Still better than requesting a (large) XML block, but it would be done more frequently, I guess, since it would be done daily or so.
Plugins do not currently have any support for "home" servers. I suppose they could provide alternatives but I don't see that it can justifiably replace the home site.
Let me explain. There aren't many use cases for third party sites. The practical applications thereof are premium sites (who won't have their plugins on the core site), people who want to distribute plugins that don't sit well with us, either ones we have moral or practical objections too or that they have issues with us for whatever reason[1], and lastly for sites issuing dev builds.
In all of these cases, there may or may not be overlaps with the core site. For example I might decide to offer some WD plugins that have overlap into the core, and offer them on both sites, and some might choose to only trust the core site even if I specify WD's site there.
The other real case where there's overlay is for dev builds. It might be best to leave the core site for stable builds and offer dev builds through a third party site. In that case you wouldn't necessarily want the third party "home" site to be considered in priority. Though you probably wouldn't offer that as a default server in the package.
I don't really know what to suggest for the best. Would really appreciate input from users who would create plugin servers, who would use plugin servers for premium resources and dev resources etc...
| 1. | Like arantormods existed to distribute plugins without any contribution going back to sm.org |
Post by: Asgard on October 23rd, 2011, 08:32 PM
The only other aspect I can possibly see causing harm is if a plugin causes a security compromise on a server and someone uses a logged list of servers which has that plugin in order to compromise all of the logged servers.
But from a pure privacy standpoint, I really don't see it being a practical issue for someone else to have a log of plugins, though I suppose that might depend on what exact plugins were on your server and if they themselves were someone how privacy compromises... (in which case I likely wouldn't have them installed in the first place...)
Post by: Arantor on October 23rd, 2011, 09:31 PM
The idea that a server could track what plugins my server says it has doesn't really bother me at all, now if it grabbed other non-plugin related information I could see that being an issue.
The only other aspect I can possibly see causing harm is if a plugin causes a security compromise on a server and someone uses a logged list of servers which has that plugin in order to compromise all of the logged servers.
The only safe option around that is to not provide a list of plugins, and simply request any and all, and filter it client-side. It's not as nice bandwidth-wise (on either side) but it avoids exposing any details the client might have.
I really don't see it being a practical issue for someone else to have a log of plugins, though I suppose that might depend on what exact plugins were on your server and if they themselves were someone how privacy compromises... (in which case I likely wouldn't have them installed in the first place...)
The specific case I'm thinking about is when you have a site that compiles a third-party compendium of packages. There is one such site I know of for SMF, where it provides its own premium mods but also has a large collection of free mods that others have submitted - and I doubt it's that thoroughly vetted. You could add that list (or, as the owner's mods do, add it in automatically) - the problem is you're risking security that way, because while I'm sure we can keep wedge.org from logging anything it shouldn't, we cannot say the same for other sites, so maybe there is some kind of risk there - anyone who could write for a plugin server could trivially enough modify said server.
Maybe I'm being paranoid here, but from a privacy and security standpoint, being paranoid is the way to be...
There is a related concern that most people don't necessarily realise: while we all assume SMF installs are unmodified, it is not unheard of for site owners to modify the setup to not hash passwords.[1]
Can you trust the security of any site you give a password to? (I don't. For sites that I'm not particularly interested in other than support, I either get a test account or create a test account with a deliberately low strength password, and will remove the account when I'm done...)
| 1. | Rare, but it HAS happened. Only once, to my knowledge, but I'd be surprised if it were only the once. |
Post by: Arantor on November 3rd, 2011, 04:31 PM
What the fuck is the point of a publicly accessible API, that's in use every day, that doesn't have documentation?[1] That assumes, of course, I already had that information to hand, but no, a trawl through the source and eventually just doing a search on wordpress.org told me what I needed to start off with, the URL of the repository. Lots of Google searches and code searches ensued.
Of relevance, is http://markjaquith.wordpress.com/2009/12/14/excluding-your-plugin-or-theme-from-update-checks/ and as a bit more background, http://w-shadow.com/blog/2010/09/02/automatic-updates-for-any-plugin/ (I don't expect anyone to read these, they're pretty WP centric, but the point should be made that underscores the very concerns I have had here, that private plugin data could be submitted to WP's master servers)
Interestingly the performance aspect of dealing with all the requests[2] is ignored.
Anyway to recap. They consider it a problem that plugins could call home and ask for updates, and want an opt-out, whether that's for privacy or practicality,[3] and it would be nice to do it in a better form than having to write a several-line function to be called from a hook before doing an update.[4]
Now, several suggestions have been put forward in how this might be dealt with, including plugins specifying a 'home' repository but I think we have to do more than that.
The typical case is a plugin that will reside on Wedge.org itself in some fashion, and nowhere else, no mirrors, no private repositories etc. Since that's the default I see no reason to actually alter it, so plugins that are just bog standard off-the-main-site ones need do nothing. And come the daily check for updates, those should be requested.
The next case is a plugin that maintains a third party repository but is also distributed on the main site. In SMF's case, this is SimpleDesk, SimplePortal and others, projects that feel they are too big in size or scope to remain totally on the home site and form their own.
Now, in those cases, it's likely that they'll have updates ahead of the main site so the plugin can specify its own repository, but that it should not automatically use that; I think it should be opt-in to use third party repositories (unless you add them manually) and there are people who won't use plugins that aren't on the official site anyway, but if they opt in, it should use the third party repository in priority to the official site.
Then you have the case of plugins that might be available on third party sites, but that won't be on the main site for whatever reason. Again, I think the adding of details should be opt-in but the plugin itself needs to indicate an opt-out for the main site.
For example, there's no need for WedgeDesk to phone wedge.org for updates because it won't get any.
The last thing to note: if a plugin contacts a third party and a response is given for that plugin, it shouldn't phone wedge.org even if the opt-out isn't used. So if a plugin originally contacts a third party but the third party is defunct, it should query wedge.org for that plugin. If the opt-out of using wedge.org is specified, it should be made clear to the user that either they use the third party server or check manually.
It's tricky!
Posted: November 3rd, 2011, 04:30 PM
Heh just after I hit the post button, a separate thing occurred to me.
Instead of automating look-ups, we can indicate sources for plugins instead, and have an opt-out of updates for that plugin, and when opting-in or -out for updates for a plugin, it can indicate and sort out where the plugin should be queried for. That seems more practical to me, actually, because it gives users the control they want and doesn't seem to me to have any other problems.
| 1. | Go to http://api.wordpress.org/plugins/info/1.0/ and click on the link for "API Docs". I did, later, find some documentation on the API, on the blog of 'DD32' but that's another story. |
| 2. | And by criminy there will be a lot of requests, as this happens every day or so from every outbound-calling WP install. |
| 3. | And I just considered the privacy angle. There are practical angles too, such as modified updates that you wouldn't necessarily want to be notified about, but if you're doing that, you're aware of the implications of such. |
| 4. | See the link to Mark Jaquith's blog, where he discusses doing that in WP. It's not pretty, and FWIW if Mark is saying it, it's probably the best there is for WP, I've encountered posts from him before. Note also that doing it from a hook implies the plugin is active, when it might not be. |
Post by: jpb on November 3rd, 2011, 05:44 PM
jpb
Post by: Arantor on November 3rd, 2011, 05:50 PM
(Also, please be aware that I have personally put hundreds of hours already into Wedge's plugin system, including writing plugins for it. I'm not exactly enthusiastic about rebuilding major parts of it.)
Post by: jpb on November 3rd, 2011, 08:35 PM
jpb
Post by: Arantor on November 3rd, 2011, 08:48 PM
I am just a user. The contao plugin-manger is very easy to use an very comfortable. That's it. I am not a developer,
Is it possible you could attach a screenshot here, please? Mind you, I have posted screenshots of what I've built, and I find it pretty easy to use...
Post by: jpb on November 3rd, 2011, 09:20 PM
jpb
Post by: Arantor on November 3rd, 2011, 09:31 PM
OK, so, the listing of plugins looks more cluttered and full of stuff that doesn't really make sense to show off, IMO. For comparison the equivalent page in Wedge is:
 |
The plugin details page is interesting but I'm not sure all that is exactly needed. In our case the situation is a lot more complicated which was the purpose of this thread (and I really should split this because the point I raised earlier today that is very important has now been utterly buried)
I do imagine our page will be similar without so much clutter.
The search page is a bit nicer, and again the points raised earlier today have bearing on that.
As far as the last item is concerned, we've actually removed things like SMF's backup function because there are better third party tools to do the job.
All in all, it's comparable to what Wedge will have when I finish it all, because as you can probably tell, it's important to me, but there's honestly nothing new there that I either hadn't seen before or considered for implementation.
Post by: jpb on November 3rd, 2011, 09:50 PM
I can't wait to install the first beta-version of wedge!
Thanks Arantor and Nao for developing wedge!
jpb