« Once upon a time...
So, SMF has declared the cold war...
Change of code in order to suit the SSL version »

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #165, on November 3rd, 2011, 12:17 AM »
Quote
I don't believe we'll ever reach the number of users that SMF has
That's as maybe but there's no harm in planning for the worst, is there?
Quote
Wedge will only retrieve a few bytes (plus headers).
Depends on push vs pull and the exact characteristics of each. Potentially, the pull could be quite expensive, not so much in terms of bandwidth but raw grunt on the server. If I were, for example, to have the query contain a list of plugin-ids, that's a list that has to be queried on the server. While it's not an enormous and expensive query, it's a lot heavier than just serving a static file which would be the preference there.

Doing it as a notification doesn't have the same burden on us, but it does put more information under our control to look after, i.e. a list of what plugins a user has.
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Nao

  • Dadman with a boy
  • Posts: 16,082

AngelinaBelle

  • Still thinking...
  • Posts: 92
Re: So, SMF has declared the cold war...
« Reply #167, on November 3rd, 2011, 01:22 PM »
Quote from Nao on November 2nd, 2011, 09:12 PM
BTW Pete, Aeva Media was already (since last April) modified to remove the auto-update code.
I mean I was expecting that if it was a load on the server, I should have received a request about it when I was active on Aeva -- i.e. over a year ago.
I don't think it was a problem on the server, when all these AEVAs were checking 1 time a day.
The problem arose because the server was offline for a day. By that point, all these AEVAS had begun checking for updates.
And not once a day. Multiple times per second. In every call to aeva_parse_bbc2. Because when the check fails, the timer is not reset. With all of them trying at once, even when the server came up, none could succeed. So they are all still trying and failing. Day and night. On every spider traverse. The hail of AEVA requests is still falling on the servers, but now the site and server teams have opened an umbrella to block the user agent and protect the file server.

Epic.

The task scheduler is always an option for future "phone home" activity. The task would run once a day, whether it succeeded or failed. The task could even log its success or failure, so that the user could find out what is going on.
I'm an SMF doc writer.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #168, on November 3rd, 2011, 01:29 PM »
Ah, a case of always working until it doesn't, and then the house of cards falls. Unfortunately been there too many times :(

The task scheduler actually isn't really an option, without creating a separate 1.1.x and 2.0 branch of Aeva, which considering its current status is somewhere between unnecessary and futile, as 1.1.x has no task scheduling facilities.

AngelinaBelle

  • Still thinking...
  • Posts: 92
Re: So, SMF has declared the cold war...
« Reply #169, on November 3rd, 2011, 01:50 PM »
Quote from Arantor on November 3rd, 2011, 01:29 PM
The task scheduler actually isn't really an option, without creating a separate 1.1.x and 2.0 branch of Aeva,
True. That's why I mentioned it for future mods.

I understand that Nao has fixed the problem in AEVA light, and it is much appreciated.
One of these days, the currently-installed AEVA lights will be uninstalled, and the AEVA hail will begin to slow down.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #170, on November 3rd, 2011, 02:03 PM »
It also doesn't help that there's not a good, easy way to add scheduled tasks. (The only way to do it is a manual DB query during mod install, I did document it many months ago)

Still, there is a lesson to be learned and it's made me go back and think about other things too...

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: So, SMF has declared the cold war...
« Reply #171, on November 3rd, 2011, 03:09 PM »
Oh, my... I didn't even take this into account. Well, at the time I wrote the feature, the sm.org server was very solid so I never experienced a downtime and didn't bother to check further.
I'm deeply sorry about this oversight.[1]

As for the task scheduler -- just like Pete said. Being backwards compatible means making choices. (Plus, I never really got the hang of the task scheduler.)
 1. Somehow, it feels like a DDOS attack sent by a guy that was post-banned long ago and wanted a late revenge.... :^^;:

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #172, on November 3rd, 2011, 03:18 PM »
Quote
As for the task scheduler -- just like Pete said. Being backwards compatible means making choices. (Plus, I never really got the hang of the task scheduler.)
The list of people who know and understand how to use the task scheduler is still in single digits, as far as I'm aware.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: So, SMF has declared the cold war...
« Reply #173, on November 3rd, 2011, 03:22 PM »
Quote from Arantor on November 3rd, 2011, 03:18 PM
The list of people who know and understand how to use the task scheduler is still in single digits, as far as I'm aware.
Which is why we need to start documenting Wedge ASAP... There are so many features we added that are barely mentioned in these forums. It'd be a shame if they weren't put to use by plugins!

AngelinaBelle

  • Still thinking...
  • Posts: 92
Re: So, SMF has declared the cold war...
« Reply #174, on November 3rd, 2011, 03:39 PM »
I was thinking that, too.  Everybody on the SMF team seems to recognize that this was unintentional. I think that, initially, the server team saw it as DDOS.  I think your unique choice of user-agent may have helped narrow down the source of the problem.

The simplemachines.org servers do seem to be pretty reliable, but even the most expensive business-critical servers have downtime on occasion (remember the recent Blackberry outage?).  In this case, a well-planned, fairly routine hardware upgrade was complicated by a coincidental RAID controller malfunction on the file server.
So  a job which should have taken a couple of hours wound up taking something more like 24 hours.
http://www.simplemachines.org/community/index.php?topic=457093.msg3198013#msg3198013

I hear your point, both of you, about the task scheduler not being so easy to use. I think it is a great feature, which should continue to get attention and improvements in SMF. Just a couple of general-purpose routines would save mod authors from having to roll their own timers.  I'll suggest it to the devs.

------------------------------

(we're typing at the same time)
And in Wedge too, of course.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #175, on November 3rd, 2011, 03:52 PM »
Quote
Which is why we need to start documenting Wedge ASAP... There are so many features we added that are barely mentioned in these forums. It'd be a shame if they weren't put to use by plugins!
Now you know why I made a more sane interface for them through the plugin manager :lol: And, yes, I did document the entire plugin manager's behaviour as far as managing plugins went because authors would need to use it. It's a biiiiiiig post.
Quote
I think it is a great feature, which should continue to get attention and improvements in SMF. Just a couple of general-purpose routines would save mod authors from having to roll their own timers.  I'll suggest it to the devs.
Better still, abstract it away. Mod authors should not need to understand what the mechanics are in order to use it. All they need to understand is what is needed to make a task, the system should deal with physical creation and management of that. The plugin manager in Wedge does this already, and I documented how it works from the user point of view but I'll just grab the content (it's already public, I'm just spotlighting the relevant bit)
Quote from Arantor on September 20th, 2011, 03:01 AM
More hokey magic: scheduled tasks

I don't think this is something that's going to come up that often, but it's convoluted enough to do manually that I wanted to make it easier.

Creating a scheduled task in the system is as simple as adding this block:
Code: [Select]
<scheduledtasks>
<task runevery="1" runfreq="day" name="shd_scheduled" file="$addondir/src/WedgeDesk-Scheduled" />
</scheduledtasks>

One task per <task> block, and it should be fairly obvious that again, you're indicating how often the task should run, what function to call and a file to load that contains that function.

(NB: Right now the task won't receive a name in the admin panel properly, I haven't yet quite decided how I want to fix that, but rest assured, I'll provide examples once I figure it out.)
That's in our equivalent of package-info.xml. The plugin manager[1] adds the relevant row on enable, turns it off on disable, removes it entirely on uninstall. The mod author never has to worry or care about inserting a row, or cleaning up after themselves.


There is also a secondary subsystem in Wedge that is available to authors though not through this interface, whereby they can schedule future-dated one-off tasks to be carried out. Worth bearing in mind (and again, it's mentioned in the changelog, so it's not like I'm giving away any secrets)
 1. At the time the post was written it was still called the add-on manager.

Nao

  • Dadman with a boy
  • Posts: 16,082
Re: So, SMF has declared the cold war...
« Reply #176, on November 3rd, 2011, 03:55 PM »
Quote from AngelinaBelle on November 3rd, 2011, 03:39 PM
I was thinking that, too.  Everybody on the SMF team seems to recognize that this was unintentional. I think that, initially, the server team saw it as DDOS.  I think your unique choice of user-agent may have helped narrow down the source of the problem.
A unique user agent is always the responsible thing to do when your software connects to some other server. (I'm just upset that I didn't think to suggest banning it when I heard about the server load issue. I only thought of banning the destination URL...)

AngelinaBelle

  • Still thinking...
  • Posts: 92
Re: So, SMF has declared the cold war...
« Reply #177, on November 3rd, 2011, 07:06 PM »Last edited on November 3rd, 2011, 07:35 PM by AngelinaBelle
The plugin manager xml interface to task manager seems like a good idea.
I also like the fact that you've got an "enable/disable hook" for every installed plugin.

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Re: So, SMF has declared the cold war...
« Reply #178, on November 3rd, 2011, 07:19 PM »
Oh, it runs so much deeper than that. I originally implemented add/remove hook as an extension to the existing packman code, months and months ago, but I realised that it wasn't how I wanted things to operate.

So I tore the rulebook up and designed a structure that really pushes and emphasises hooks. There's no version checking done, for example, unless file edits are done, because the plugin system works on the principle that the hook's existence is sufficient. You don't need to worry about version numbers etc. if the hooks are present and tests are done at install time to validate that the required hooks are available. On top of that, a plugin can indicate that it provides hooks of its own giving you an implicit dependency system.

On top of that, the plugin file also indicating things like settings means that the plugin manager can clean them up on uninstall. As I learned the hard way, authors don't generally do things thoroughly, and mods that do proper housekeeping to clean up after themselves are unfortunately rare - so I made sure to take as much away from that as possible.

Default values of settings are declared (though due to updateSettings()'s behaviour, default values of 0 aren't updated to the system) and so can be used to clean up on uninstall, plus it's possible to intimate language support - multi-language readmes were supported in SMF 2.0 RC2 onwards but I don't think anyone ever used them. But now, if multi-language readmes are given, they're firmly used (so it encourages people to use them)

Each element of the plugin manager was designed around things I observed as flaws with SMF's mods, and things I learned from doing SimpleDesk.[1] I'm not saying it will work for everyone but I feel like it's working for us. I've not found many problems in writing plugins and I find the way everything is held together to be more reliable in the long run.
 1. There are some serious low-level tweaks in there like modifying the language editor to list the mod language files.

Nao

  • Dadman with a boy
  • Posts: 16,082
« Once upon a time...
Change of code in order to suit the SSL version »