Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - Arantor
5581
Plugins / Re: Plugin manager, phoning home and privacy
« on November 3rd, 2011, 04:31 PM »
OK, so I've spent most of my day thus far understanding WTF WordPress does. If I didn't already hate WordPress, I really hate it now.

What the fuck is the point of a publicly accessible API, that's in use every day, that doesn't have documentation?[1] That assumes, of course, I already had that information to hand, but no, a trawl through the source and eventually just doing a search on wordpress.org told me what I needed to start off with, the URL of the repository. Lots of Google searches and code searches ensued.

Of relevance, is http://markjaquith.wordpress.com/2009/12/14/excluding-your-plugin-or-theme-from-update-checks/ and as a bit more background, http://w-shadow.com/blog/2010/09/02/automatic-updates-for-any-plugin/ (I don't expect anyone to read these, they're pretty WP centric, but the point should be made that underscores the very concerns I have had here, that private plugin data could be submitted to WP's master servers)

Interestingly the performance aspect of dealing with all the requests[2] is ignored.

Anyway to recap. They consider it a problem that plugins could call home and ask for updates, and want an opt-out, whether that's for privacy or practicality,[3] and it would be nice to do it in a better form than having to write a several-line function to be called from a hook before doing an update.[4]

Now, several suggestions have been put forward in how this might be dealt with, including plugins specifying a 'home' repository but I think we have to do more than that.

The typical case is a plugin that will reside on Wedge.org itself in some fashion, and nowhere else, no mirrors, no private repositories etc. Since that's the default I see no reason to actually alter it, so plugins that are just bog standard off-the-main-site ones need do nothing. And come the daily check for updates, those should be requested.

The next case is a plugin that maintains a third party repository but is also distributed on the main site. In SMF's case, this is SimpleDesk, SimplePortal and others, projects that feel they are too big in size or scope to remain totally on the home site and form their own.

Now, in those cases, it's likely that they'll have updates ahead of the main site so the plugin can specify its own repository, but that it should not automatically use that; I think it should be opt-in to use third party repositories (unless you add them manually) and there are people who won't use plugins that aren't on the official site anyway, but if they opt in, it should use the third party repository in priority to the official site.

Then you have the case of plugins that might be available on third party sites, but that won't be on the main site for whatever reason. Again, I think the adding of details should be opt-in but the plugin itself needs to indicate an opt-out for the main site.

For example, there's no need for WedgeDesk to phone wedge.org for updates because it won't get any.

The last thing to note: if a plugin contacts a third party and a response is given for that plugin, it shouldn't phone wedge.org even if the opt-out isn't used. So if a plugin originally contacts a third party but the third party is defunct, it should query wedge.org for that plugin. If the opt-out of using wedge.org is specified, it should be made clear to the user that either they use the third party server or check manually.

It's tricky!
Posted: November 3rd, 2011, 04:30 PM

Heh just after I hit the post button, a separate thing occurred to me.

Instead of automating look-ups, we can indicate sources for plugins instead, and have an opt-out of updates for that plugin, and when opting-in or -out for updates for a plugin, it can indicate and sort out where the plugin should be queried for. That seems more practical to me, actually, because it gives users the control they want and doesn't seem to me to have any other problems.
 1. Go to http://api.wordpress.org/plugins/info/1.0/ and click on the link for "API Docs". I did, later, find some documentation on the API, on the blog of 'DD32' but that's another story.
 2. And by criminy there will be a lot of requests, as this happens every day or so from every outbound-calling WP install.
 3. And I just considered the privacy angle. There are practical angles too, such as modified updates that you wouldn't necessarily want to be notified about, but if you're doing that, you're aware of the implications of such.
 4. See the link to Mark Jaquith's blog, where he discusses doing that in WP. It's not pretty, and FWIW if Mark is saying it, it's probably the best there is for WP, I've encountered posts from him before. Note also that doing it from a hook implies the plugin is active, when it might not be.
5582
Features / Re: Ideas from the oddest places
« on November 3rd, 2011, 04:21 PM »
That's the thing I'm most surprised at, actually. I took a cruise through their anti-spam plugins, and I find very few actually new ideas, many of which I'd already thought of and either implemented or rejected because they aren't viable for widespread use.

I'm also slightly surprised to note that there's only 16,000 plugins; given the relative dominance of WP over SMF, you'd think there would be more than 10x the plugins...
5583
Features / Re: Ideas from the oddest places
« on November 3rd, 2011, 04:08 PM »
Quote
The only thing that will make a difference to me when it comes to avatars, is finally getting rid of any possibility of loading them through PHP
Yeah, we need to do that, soon.

I just noticed it and thought I'd share. I didn't expect it to be greeted with mass enthusiasm, because it doesn't really work in our case, but I figured I'd share just in case.
5584
Features / Re: Thought system
« on November 3rd, 2011, 04:07 PM »
Quote
Any issues with the field being called 'data', btw...? That's what it is called in my local install. It's not very original and can be confusing, but at least it makes for a nice $user_info['data'] array.
'data' is a little generic, but it works.
Quote
Well there's definitely some overhead too for the AeMe cache and it's stored in the profile table...
There's overhead whatever you do, the only way not to have overhead is to not do anything to the menu in the first place.[1]

But in WD's case, the overhead is actually more trivial; it's not just looking up last-read-id vs last-actual-id or similar, there's a whole bunch of determination of ticket status vs author vs permissions to figure out how many tickets should be shown if the user is staff, and how many if they're a regular user. I spent far too long fighting it and it would save so much effort to just put it in the members table where it can be updated as needed rather than queried and regenerated at best every few minutes (and at worst far more often)

Horses for courses. Do what works best for what you're doing, and in my case I realised too late that I should have put it in the members table.
Quote
Yeah, that's the thing with the data field: it's only for when you need to update one member at a time... Which is why the custom function I added for it (updateMyData(), can be renamed later) doesn't take a $members param -- it uses $user_info['id'] directly, at least for now.
Yup, and that's cool - provided that you are only updating one member at a time. The data field makes that a requirement, not a contextual thing. (You can update multiple rows to SET column = column + 1 in a single query, but you can't update multiple rows based on multiple criteria)
Quote
Do you have any stats, honestly.....?
http://www.simplemachines.org/community/index.php?board=9.0 and http://www.simplemachines.org/community/index.php?board=147.0 searching for 'ftp' and 'chmod'. Those are the users who can't do it out of the box. Pretty much all crap shared hosting (which to be fair, is almost all of it) runs the web server as www-data or nobody or apache, which means it's not writable.

And while there is an argument to be made about using chmod, the idea of making the Plugins folder a+rw doesn't fill me with confidence, especially because it then becomes an attack vector (especially on shared hosts) - it's not like the attachments or avatars folders, where you're expressly locking down content from being executed, the contents of Plugins and its related folders absolutely *must* remain executable. Which means elevating privileges, uploading, then dropping privileges again.
 1. The only winning move is not to play.
5585
Other software / Re: So, SMF has declared the cold war...
« on November 3rd, 2011, 03:52 PM »
Quote
Which is why we need to start documenting Wedge ASAP... There are so many features we added that are barely mentioned in these forums. It'd be a shame if they weren't put to use by plugins!
Now you know why I made a more sane interface for them through the plugin manager :lol: And, yes, I did document the entire plugin manager's behaviour as far as managing plugins went because authors would need to use it. It's a biiiiiiig post.
Quote
I think it is a great feature, which should continue to get attention and improvements in SMF. Just a couple of general-purpose routines would save mod authors from having to roll their own timers.  I'll suggest it to the devs.
Better still, abstract it away. Mod authors should not need to understand what the mechanics are in order to use it. All they need to understand is what is needed to make a task, the system should deal with physical creation and management of that. The plugin manager in Wedge does this already, and I documented how it works from the user point of view but I'll just grab the content (it's already public, I'm just spotlighting the relevant bit)
Quote from Arantor on September 20th, 2011, 03:01 AM
More hokey magic: scheduled tasks

I don't think this is something that's going to come up that often, but it's convoluted enough to do manually that I wanted to make it easier.

Creating a scheduled task in the system is as simple as adding this block:
Code: [Select]
<scheduledtasks>
<task runevery="1" runfreq="day" name="shd_scheduled" file="$addondir/src/WedgeDesk-Scheduled" />
</scheduledtasks>

One task per <task> block, and it should be fairly obvious that again, you're indicating how often the task should run, what function to call and a file to load that contains that function.

(NB: Right now the task won't receive a name in the admin panel properly, I haven't yet quite decided how I want to fix that, but rest assured, I'll provide examples once I figure it out.)
That's in our equivalent of package-info.xml. The plugin manager[1] adds the relevant row on enable, turns it off on disable, removes it entirely on uninstall. The mod author never has to worry or care about inserting a row, or cleaning up after themselves.


There is also a secondary subsystem in Wedge that is available to authors though not through this interface, whereby they can schedule future-dated one-off tasks to be carried out. Worth bearing in mind (and again, it's mentioned in the changelog, so it's not like I'm giving away any secrets)
 1. At the time the post was written it was still called the add-on manager.
5586
Features / Ideas from the oddest places
« on November 3rd, 2011, 03:44 PM »
In the depths of my research into how WordPress updates (for plugins) work, I bumped into a separate site entirely.

In particular, this post caught my eye: http://w-shadow.com/blog/2011/01/31/lazy-load-avatars/

Now, it makes more difference on WordPress where the bulk of content will be without avatar, and only the comments are, but it strikes me that it could make a difference in terms of perceived load performance if applied to avatars in Wedge.

Thoughts?
5587
Other software / Re: So, SMF has declared the cold war...
« on November 3rd, 2011, 03:18 PM »
Quote
As for the task scheduler -- just like Pete said. Being backwards compatible means making choices. (Plus, I never really got the hang of the task scheduler.)
The list of people who know and understand how to use the task scheduler is still in single digits, as far as I'm aware.
5588
Features / Re: Thought system
« on November 3rd, 2011, 03:17 PM »
Quote
Looking at some of your code in updateMemberData...
That was one of mine? Egad, looks ugly. Smells more like something that was there before and that I changed.
Quote
I take it you agree that a data field is a good idea then
I'm a fan of such things, and I'm going to be making use of such when I get to implementing the per-user sanctions (e.g. through warnings and having permissions removed etc.)
Quote
I didn't know that instant_messages was calculated when sending a PM. I thought it was done when going to your PM area or something... So, scrapped.
unread_messages is probably in a similar situation.
Yeah, it's done at point of sending and updated when you read messages, the idea being to minimise the effort required in gathering the data for the purposes of updating something that appears on every page. No sense requerying when you can just update it at the point you're physically increasing the number of items that it relates to. The downside is that it is possible to get it out of whack, though it isn't that common.
Quote
Clearly, though, I'll be doing the data field because it's a good opportunity for plugins to store profile-specific data without having to add their own database fields (or custom fields).
Well... it sort of depends. If it's going to be a mostly passive thing like timezone, it's cool to do. But anything that is primarily a form of cache (like unread_messages) shouldn't be put there. Of course, there's no reason why you couldn't use the themes table if you're careful to store per-user settings anyway (since it's already abused for the custom profile fields)

Let me put it another way. Right now, one of the things I came to regret in WedgeDesk was the way the number of tickets is cached and not stored in the profile of the user. The overhead, despite caching, for that is surprising - and it has other consequences too that I won't get into.[1] So for the next version I'm going to be putting it into the members table.

While at first blush it would make sense to put it into the cache you're proposing or into the themes table, the best solution for me is a new column in the members table. From my perspective, the value is obtained immediately (though as I discussed elsewhere, I don't really like the fact that $user_settings is persisted and contains the *entire* row, regardless of what $user_info has), and I can update it when necessary regardless for users with a single UPDATE, which I can't do in the case of either the cache (because I then have to do one update per user) or the themes table (because I have to establish the value exists to UPDATE it or do an INSERT first)

It's useful for data that's per user but only accessed and updated by that user, or at least, very infrequently by other users.
Quote
I always wondered about that but didn't want to break it.
At least, that's my understanding of it. I can't imagine any other reason.
Quote
Ah, yes, that was it... I remember now how I was frustrated having to cancel my upcoming commit about that...
Quote
We could also simply say -- if your upload process doesn't work, use FTP and don't bother us with that! Just outputting a nice error message explaining what happened is okay to me...
Which means for a fraction of users, they have a cushy ride and everyone else is told to hit the highway. To be honest, I don't feel that's particularly acceptable, it's basically saying "I couldn't be bothered to make this work well for you." in my head.
 1. Suffice to say, SSI has fun with it, and the logic around cache purging when tickets are updated is ridiculous.
5589
Other software / Re: So, SMF has declared the cold war...
« on November 3rd, 2011, 02:03 PM »
It also doesn't help that there's not a good, easy way to add scheduled tasks. (The only way to do it is a manual DB query during mod install, I did document it many months ago)

Still, there is a lesson to be learned and it's made me go back and think about other things too...
5590
Other software / Re: So, SMF has declared the cold war...
« on November 3rd, 2011, 01:29 PM »
Ah, a case of always working until it doesn't, and then the house of cards falls. Unfortunately been there too many times :(

The task scheduler actually isn't really an option, without creating a separate 1.1.x and 2.0 branch of Aeva, which considering its current status is somewhere between unnecessary and futile, as 1.1.x has no task scheduling facilities.
5591
Features / Re: Thought system
« on November 3rd, 2011, 01:12 PM »
Quote
id_thought and thought can be in it, because it's only going to be used in the member's sidebar.
Works for me.
Quote
Things like smiley_set or id_theme/skin could also be in it (although the latter would prevent easily giving stats in the Change Skin page.)
I'm not fussed about smiley set,[1] though theme/skin would probably be better to keep out for reporting purposes.
Quote
Maybe timezone and language stuff, too. (It's used in a user's profile, though. But we could have some kind of function that gets the serialized array and extracts some data from it. As long as it's not used for more than *two* users at a time (the viewer and the viewee), it's probably safe.)
Timezone doesn't bother me, unless we want to do some kind of display of user timezones. Language, however, has other consequences, like when sending email templates, it actually tries to fetch the right one for the language of the user the email is going to.
Quote
message_labels maybe... But it's a text field so it could end up eating the new data field's space. We could make the data field a LONGTEXT but it could end up wasting a lot of time unserializing data that's never used (outside the PM area.)
Offhand, I'm not sure where message_labels is evaluated in terms of how it figures out how to label a PM (given that it's essentially a rules filter)
Quote
instant_messages and unread_messages are good candidates, too.
Not they're not. If you send someone a PM, do you really want to have to query their member rows, unserialize each one, increment, then update each row? If you send 10 people a PM, that's 11 queries. Now imagine sending a newsletter to each member via PM. As opposed to a single query that updates all of those things at once (which means only one table lock)
Quote
Well... What do you think? I for one would sure enjoy seeing a stripped down member table!
And TE's importer can easily deal with serializing imported fields.
Keeping the table lean is important, provided you don't end up compromising other things like above.
Quote
Oh, while looking at the member table, I found a couple of definitions ending with default 0 instead of default '0'... I turned them to '0' but I've always wondered the point of using a string when an integer is perfectly fine... Maybe it was for compatibility with SQLite or PGSQL or something...? Maybe we should just go ahead and use integers....
The dump is from either mysqldump or phpMyAdmin (not sure which) which escapes everything, anything that's without quotes is one added manually somewhere.

It wasn't compatibility, SQLite and PGSQL got their own install SQL scripts.
Quote
Yes we did. And like with so many conversations, I guess nobody thought of adding this to their to-do list...
On top of that we were talking about using wesqlQuery with it too, which is probably why no-one did it, because it's hellishly complicated (and important to get right)
Quote
I completely agree with your view on FTP, but there's really nothing we can do.
I'd tend to say, "to hell with people who use strange filesystems", but the more we say that kind of thing, the less users we'll have...
It depends on your definition of strange. From my point of view, the majority of filesystems are strange :P I did consider avoiding the whole upload process entirely and forcing users to use FTP but that's not very nice, now, is it?
 1. Honestly, how many sites actually do the whole multi-smiley-set thing? Most people neither know nor care. But I don't have enough reason to remove that.
5592
Off-topic / Re: Battlefield 3
« on November 3rd, 2011, 12:01 PM »
In other news, I've been playing Voxatron from the current http://www.humblebundle.com/ package (I bought it when it first came out, so I didn't have to do anything special to get the Binding of Isaac and Blocks That Matter, though I already had that one anyway)
5593
Features / Re: Thought system
« on November 3rd, 2011, 11:50 AM »
Quote
Well, I'm thinking we may have to do a join on the display page... I don't like the idea, but... Well, I don't know. I'd like more opinions.
Technically, it's in loadMemberData where the join is. Hmm, lMD needs more branches than just minimal, normal, profile. Didn't we already talk about that before? :/
Quote
That plugin feature is a bitch eh?
The plugin management itself is pretty straightforward. It's all the crap that goes with it, like dealing with file permissions[1] and figuring out how the update process should work.
 1. Seriously, while I understand the principles and so on of the way Linux/Unix file permissions are done, the whole lot of having to use FTP to elevate permissions on most systems instead of allowing the PHP process to run as the right user is just so mindnumbingly broken...
5594
Features / Re: Thought system
« on November 3rd, 2011, 10:58 AM »
Oh, I see, you always need the latest, public or not, for any given user - even if you're going to be serving the public one inside topic views.

There's no good way around that. Either you have to store both in the members table, or you have to store them both in the main thoughts table and query for them appropriately. Or one of each.
Quote
I'm feeling like everytime I want to tackle a small task like thoughts, I end up spending ten times longer on it than I first expected.
I know exactly what you mean :/
Posted: November 3rd, 2011, 10:58 AM

As for progress, commit what you're happy with.
5595
Features / Re: Template skeleton!
« on November 3rd, 2011, 01:51 AM »
Hmm, I think I found another bug, this time in wetem::layer. Just want to confirm I'm using it correctly or not as the case may be.

This one's from WedgeDesk.

Code: [Select]
wetem::load(
array(
'shd_post_nojs',
'post_navigation',
'preview',
'ticket_info',
'ticket_subjectbox',
'ticket_meta',
'ticket_postbox',
'ticket_footer',
'ticket_pageend',
'ticket_proxy_js',
),
'default',
'replace'
);

if (!empty($context['ticket_form']['do_replies']))
{
wetem::layer('ticket_replies', 'ticket_pageend', 'before');
wetem::load('ticket_do_replies', 'ticket_replies', 'add');
}

It's a long story as to why that's a replace up in the first load call, but that's what it has to be for the time being until I fix other things. This is based against r1142, but I have something to add in a moment about r1143.

Anyway, my understanding is that I can dynamically create a layer through wetem::layer and have it positioned where I want in the list, and I want to add it dynamically based on a variable that I know to exist and be set correctly. Except that although the if() is executed, wetem::layer never adds ticket_replies as a layer before ticket_pageend, and do-replies is never added to the list.

I know I can work around it by declaring the layer up front in the main load call and then dynamically removing it if the relevant variable is empty, but that seems clumsy to me.

Also, r1143 will break any instance of array_insert; all places it was called (e.g. the template skeleton) relied on it modifying $array by reference but now it's returning a value - and wetem (amongst other places) isn't picking that up. I'll fix it tomorrow if you don't get to it first.