Wedge

Public area => Development blog => Topic started by: Arantor on July 12th, 2011, 12:56 PM

Title: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 12:56 PM
While I'm still trying to figure out how to incorporate all the feedback from the package manager changes(!), I thought I'd talk about what I want to do with the ban system. Sorry in advance, this is going to be a bit of a novel: it's a big change, it's probably at least as controversial, and something about it is necessary anyway, so let's dive in.

The ban system as implemented is functional, as in it works but it's not overly elegant, it doesn't support IPv6 and I take the view that it doesn't solve the problem at hand, not one bit.

Let me deal with the IPv6 problem first, before I tackle the other stuff. The current system works on IPv4 addresses, which are x.y.z.a addresses, and whatever you put a ban (on IP address) on, it resolves to a range internally for each of the blocks. So a ban on 1.2.*.* becomes a ban internally on 1.2.0-255.0-255. Structurally, that makes sense, but IPv6 is much larger - instead of 4 blocks in the range of 0-255, you have 16 to contend with, though they're not written in decimal, nor written in the same way, but written as aaaa:bbbb:cccc:dddd:eeee:ffff:0000:1111 and similar.

There is one thing to consider, that addresses are divided in half in IPv6, the first half is for a 'network' and the second half for machines in that network, and it sounds that on the surface you could get away with just barring based on the first half only. Whether that will be successful or practical remains to be seen, but something tells me it's not that practical. It's not even that practical from a technical standpoint because if you're keeping that approach, you're not just comparing 4 values against ranges, but doing it for at least 8 - and you need to handle the high/low values, which is what SMF's and Wedge's system does right now.

I didn't implement IPv6 in Wedge in a way that would make this particular easy to implement for, because I took the view that it was the wrong way to be going about it, that any minor change extending the current direction of implementation to fit either 2x or 4x larger scope was an unnecessary performance headache, as well as a logistical one.

So, I sat back and thought about what I'd really like to be able to use in the ban system, and that lead me to my normal approach of trying to figure out what it is the ban system should be needed for, and what it should be able to do.


What is the ban system for? Primarily it's for getting rid of miscreants, and troublemakers. It isn't really a spam-solving solution, and it's not really for keeping users out that you're not interested in - it's for keeping users out that you don't want, which isn't the same thing.

Now before you start, I'm well aware that users do currently use it for keeping users out that they're not interested in, but on a variety of levels, I'm just not sure how viable that is, but we'll get on that in a minute.

So, dealing with troublemakers. The ban system lets you ban a user by name, email, IP or hostname. So you ban them, they come back under a new name through a proxy. Doesn't solve the problem much. For dealing with trolls and so on, there are better ways of dealing with them instead of slamming the door in their face - the tools used by Annoy User for example, to lock off certain features, plus the warning system that allows you to control whether they can post or whether their posts are moderated.

Of course, none of those things will solve the proxy problem, but the ban system wouldn't anyway. No, the solution is to gently turn up the heat so they don't realise that they're being pushed out, or at least discouraged from posting for whatever reason, and without it being obvious - so that they go and do something somewhere else.

If anything, the face-slam of the door is probably worse, not better, at making them go away - because what happens is that they don't have closure, they're not leaving of (kind of!) their own will, so you get all kinds of hassle as a result.


As for banning on email address, what is the hope of that? If you have miscreants who have their own domain, they can create as many emails as they like, so you just restrict the entire domain - it won't prevent them re-registering, though. So you get the extra account, you ban the entire domain, they try to register a third time and they still register - but this time they're banned and will take the hint. The problem is you've still got more accounts than you wanted in the first place.

Instead, then, how about limiting the email addresses up front? Put in the ability to restrict emails based on domain, either whitelisting or blacklisting certain domains as necessary. I know a number of users that restrict signups from mail.ru because of spam - if the domain is blacklisted, they can't even register (which is better than banning it).

There is, interestingly, a performance consideration here - and one for the better. If you ban based on email, the ban has to be evaluated more frequently than just locking it down at registration/change email time. In fact, that's going to be true of all bans - the more bans you have, the more you have to evaluate, and it has even a per-page consequence. By removing that query, you remove the performance hit, especially on long-term sites that have many bans, most of which aren't needed any longer.


Then we have IP addresses. Hello, darkness, my old friend. Putting aside the considerations of above with IPv6 addresses, the simple fact is that IP bans are really not that effective at keeping out miscreants because of proxies. That said, if you apply any of the measures in something like Annoy User, such users will likely notice it when they log out, or if they use another computer after logging out (so you can't even really use cookies on their computer) - not to mention the fact that IP addresses are shot to bits if you use mobile devices on 3G connections and similar. It's not like you can even reliably block proxy connections here.

With all that, IP bans are basically useless, except to the most technically inept of users - and they certainly don't keep out spammers, there are better ways of doing that which don't require tracking IP addresses, which are only going to be more and more useless for tracking in future as IPv6 goes mainstream.

The only salvage then is hostname, but even that... well, it's typically disabled in a lot of cases because of sluggish performance (usually because hosted machines are behind a laggy rDNS) meaning it's not a lot of use to you, and even if it wasn't, most of the time bans are not carried out on hostnames but on IP addresses, when really, hostnames would be more useful.

The solution then, might be to be able to blacklist certain hostnames if lookups are enabled and functioning, but to use it at a deeper level than keeping the conventional bans on it (there are performance considerations too), and then you could use it only if you needed it. What I might do is integrate that into our Bad Behaviour implementation, making it look like (to the user, anyway) as if their computer has a problem rather than anything else.


That wraps it up for the problems with the ban system and how they can be mitigated, but let's go further: dealing with miscreants needn't stop at fixing the current setup.

So, user-level problems, we deal with at the user level, not some global administrative level. I'm thinking we can expand the warning system as a result. Right now users can be watched, moderated or muted. It's trivial to expand that to full-on banned, and it would be useful to expand how the tail-off works. Right now you can set how quickly the warning level drops for all users (in points per day), but making that per user would make more sense, so that users who just need a time-out can be given one, and it can be done per user, rather than something across the board.

I'm also thinking we could influence other permissions, such as losing avatar and signature if the warning is over a certain level.


Just for fun, there's another subsystem I've been thinking about, that will debut in some form. Specifically, it will allow you to add rules to certain parts of the system, e.g. things to do when a post is made - so you can check the contents of a post, and if it contains words you don't like, it gets moderated and the user can be warned automatically.


Too long, didn't read (tl:dr;) summary:
* Removing the ban system as it is
* Making post moderation more prominent, probably even enabled by default (but with performance tweaks to make it run more efficiently)
* Email blacklist/whitelist on registration/change email, instead of the old method of banning
* Add hostnames to the possible rules that will be checked in our Bad Behaviour setup, so that instead of getting a 'banned warning', it looks like problems with their computer
* Replacing user-level bans with the warning system and making it more granular rather than as coarse as it is right now
* Adding functionality from my old Annoy User mod to encourage bad users to go away
* Expanding the warning system to more gradually remove powers, than just moderated and muted

I don't think I missed anything but if I did, I'm sure you're going to let me know about it!

And please, before telling me you need the ban system as it is, really stop and think about what you use in it and why you use it, then before complaining at me for breaking what you think is an essential feature, think about if there's actually a better way of doing it, like the above. Banning is not a particularly wonderful technique as explained - it doesn't solve any problem, it solves some of the symptoms. I'm trying to solve the deeper problems. Just because something is what it is, doesn't mean you have to accept it.

Oh, one more thing I forgot.

I want to introduce a 'Banned' membergroup that users go into. Not only does it have a visual consideration but a permissions one: it would let you reduce access to boards. I don't know yet whether I want to make that an on/off thing (like banning is now, except it would turn off some boards and maybe show others) or a gradual thing (as you get more warnings, you slowly see fewer and fewer boards)

But that would certainly make life interesting!
Title: Re: Banning, and what I want to do with it
Post by: snoopy-virtual on July 12th, 2011, 02:36 PM
I have been reading it all and cannot see anything missed. I think you have covered there any possible aspect of the problem and also that you have found the perfect solutions for each one of them.

In fact I stopped using the ban system in SMF 2 or 3 years ago because I thought it wasn't working as it should and I already have more or less some of the ideas you have been talking about here (but in a more subliminal level, not as clear as you have expressed them here).

Congratulations.  :cool:
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 12th, 2011, 05:02 PM
Well, it seems to me that a solution to spam attacks would provide a solution to your garden variety bad user because all the tools available to spammers (IP spoofing, proxy, multiple email accounts) are also available to the miscreant, and a good number of them will use those.

In banning IPs I've accumulated quite a long, long list of ban triggers (mainly spammer/hackers who attempt to create an account without visiting the board), and when I periodically check on those for how many times they get hit, I find that the majority of the IPs have been used only once, and that was to attempt to hack into the board. Others, just a handful of times, so I'm skeptical about how effective IP banning is on the whole, as even your garden variety miscreant can change IPs.

And a spammer spoofing the IP of a legit user is something else again, although I haven't seen that happen at all this year. A couple of years ago, though, I found that I was banning legit users when I went after the spammers.

That Annoy User is probably the better way to deal with miscreants...or simply restrict that User's Permissions down to nothing. IMHO. IP banning has gotten increasingly more useless over the years, and it has certainly gotten increasingly high-maintenance, taking up a huge chunk of management time (and bandwidth when the list is long?)

I've explored the .htaccess idea, but when you have a long list of IPs, the work factor doesn't diminish. Banning an IP via .htaccess is pretty darn drastic at the get-go.   And I do like the blacklist/whitelist idea.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 05:29 PM
That's the thing: once you've swung the ban hammer, either they come back with new details - which a ban doesn't solve - or they leave you alone. Either way the ban isn't worth keeping around because it just clogs up the system.
Quote
mainly spammer/hackers who attempt to create an account without visiting the board
Depends what you mean by 'without visiting the board' - registration requires that you go through the right channels, i.e. through the action=register stuff, you can't just go direct to action=register2, it shouldn't let you (and if it does, it's broken!)
Quote
Others, just a handful of times, so I'm skeptical about how effective IP banning is on the whole, as even your garden variety miscreant can change IPs.
That's why I'm actively proposing doing away with it. Hostname blocks are a slightly different kettle of fish because you could block a hostname at the top-most level and be able to block more than a discrete IP range, without worrying about the side effects of over-sized ranges. The only issues then are if you have a poor rDNS, and/or IP spoofing but you'd have those anyway, regardless of anything else.

The key thing here is that I'm taking something that's largely a technical solution to a non-technical problem away, and replacing it with more sociologically-aware tools. Some of which are still technical, but they're not done for the sake of technical convenience.
Quote
That Annoy User is probably the better way to deal with miscreants...or simply restrict that User's Permissions down to nothing.
That's the general plan of mine, yes.
Title: Re: Banning, and what I want to do with it
Post by: spoogs on July 12th, 2011, 05:29 PM
Nice post Arantor

As mentioned elsewhere I already turned to using banned groups in SMF some time ago, 3 for the mainly public sections of the forum and a few others for the specialized sections. Each ban group denies certain permissions while 1 in particular denies all permissions (view/enter board permissions made my setup even more tightly controlled).

You make very good points:
Troubled users generally want closure before they leave a site[1], so much so that 1 of the features of my banning system is that banned users see a specific board where they can appeal the ban levied on them... I actually find that doing this causes them to get out what their issue is and my staff can double check to see if the ban was warranted or not. If a user is placed in the highest ban group all they can do is login if the want to but that's all they get. Their username and email is now hostage they can register with different ones if they choose but I find they generally are much less of an ass if they do.

I've only known of 1 IP-ban issued on my forum (pre-SMF anyway) and the consequences for us were brutal. This guy started using a proxy made our lives a living hell for almost 2 months IIRC, as pissed as we were we could only laugh about it until we just gave up.

There's and SMF mod for white-/black-listing domains but it was never extended to email addresses.

I've never really had much of a spam problem so I can't relate much from experience but your proposal still seems pretty solid overall.
 1. at least that's how it seems to me
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 05:34 PM
Quote
Each ban group denies certain permissions while 1 in particular denies all permissions (view/enter board permissions made my setup even more tightly controlled)
That's really what the ban group proposal is for, for this very reason: to ensure that certain facilities can be revoked if desired.
Quote
Troubled users generally want closure before they leave a site
Yup. Or, sometimes, to argue their defence, depending on the situation. Either way, it's not a simple black and white line, nor should it be.
Quote
Their username and email is now hostage they can register with different ones if they choose but I find they generally are much less of an ass if they do.
Yup. That's something in particular about why I don't delete accounts, nor ban them. If they're a nuisance, I issue a 100% warning and go from there.
Quote
This guy started using a proxy made our lives a living hell for almost 2 months IIRC, as pissed as we were we could only laugh about it until we just gave up.
This is why IP banning is no longer effective ;)
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 12th, 2011, 06:55 PM
Yup, it sounds like you've got a solid handle on it, Arantor.

And to explain this...
Quote from Arantor on July 12th, 2011, 05:29 PM
Depends what you mean by 'without visiting the board' - registration requires that you go through the right channels, i.e. through the action=register stuff, you can't just go direct to action=register2, it shouldn't let you (and if it does, it's broken!)
I get newly registered Member accounts with the Last Active status as Never. In other discussions about spammer/hackers elsewhere in cyberspace, it was brought to my attention that there exists registration-bot software and I strongly suspect this is what's been happening: registration-bot registrations.  If SMF is supposed to be designed to not allow that, then somebody needs to re-examine how registration-bots still get in.

Additionally, I have a mod installed that I consider vital: Default Membergroup On Registration. All registered newbies are automatically put in Restricted Newbie Membergroup with access to only one board, not visible to regular Members, requirement of which is to make one post of introduction so that I can tell if they're real people (or miscreant).  Along the lines of Annoy User, I can always put a miscreant in the Restricted Newbie membergroup.

==========================

I should add that my board's traditional policy has been to not use email verification, and if I can avoid changing that policy, I would prefer to avoid it.
Title: Re: Banning, and what I want to do with it
Post by: spoogs on July 12th, 2011, 07:13 PM
I use that mod as well, but you could accomplish what you use it for without that mod by being rather careful with board access and permissions.

I use that mod only for the fact that I don't like having the post count group as a primary group.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 07:21 PM
OK, I can explain the Last Never behaviour though, it's where they registered but never go back into the forum after the registration completes, usually because of badly written bots.

The whole "default" member group thing is something that we probably should deal with, but I think a wider change is needed to make post count groups not special.
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 12th, 2011, 07:29 PM
Quote from Arantor on July 12th, 2011, 07:21 PM
OK, I can explain the Last Never behaviour though, it's where they registered but never go back into the forum after the registration completes, usually because of badly written bots.

The whole "default" member group thing is something that we probably should deal with, but I think a wider change is needed to make post count groups not special.
A good beginning point would be to get rid of the automatic "(no primary member group)" arrangement and automatically put them in a specific default member group, IMHO.

And IMHO, if the registrant was last active on the board by registering, the Last Active status should read the date of the registration.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 07:31 PM
They are in a specific member group. Just it's a special one you can't ever edit, though you are physically attached to it (it is not possible to be in less than two groups, ever, without mashing the members table)
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 12th, 2011, 07:32 PM
Quote from spoogs on July 12th, 2011, 07:13 PM
I use that mod as well, but you could accomplish what you use it for without that mod by being rather careful with board access and permissions.

I use that mod only for the fact that I don't like having the post count group as a primary group.
The mod is automatic which means I don't have to babysit my board every livin' minute.  I got a life. ;)
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 12th, 2011, 07:32 PM
You can do everything you said automatically, just with a different configuration of groups.
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 12th, 2011, 07:36 PM
Quote from Clara Listensprechen on July 12th, 2011, 07:32 PM
The mod is automatic which means I don't have to babysit my board every livin' minute.  I got a life. ;)
Well, that's the problem. The deal with having a "no primary member group" is that you can set up membergroups based on postcount. I'm saying that shouldn't be the default membergroup a new Member should be in, or any other Member who is not subject to postcount membergroups. "No primary member group" should instead be "Postcount membergroup" and in that one can set up groups dependent on postcount.
Posted: July 12th, 2011, 07:35 PM
Quote from Arantor on July 12th, 2011, 07:32 PM
You can do everything you said automatically, just with a different configuration of groups.
Be that as it may, I swear by the mod. I luvz that mod. I ain't letting it go. so THERE. :P
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 13th, 2011, 12:14 AM
There is one side effect that occurs to me, though I'm not convinced it's a major one, not enough to drastically alter plans, anyway.

There is, specifically, one edge case in SMF, where you ban on email address, do so on a domain or other wildcard level, and it ends up banning one or more other users because of the ban.

Under the new proposal, this would simply allow them to remain until they updated their email next, so I'm going to have to factor into the process some method whereby when the admin adds a blacklisted email, it checks existing emails and lets the admin do something about it (e.g. sending the user a PM, or a warning or whatever)
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 13th, 2011, 01:48 AM
There's that, and I wonder if there's a way to check if an email is a valid address. I've had spammers use amazonaws.com as an email address and I've had no problems banning *@amazonaws.com because the deceit is obvious. In other cases, the deceit isn't as obvious, ie @gmail.com . But sometimes when one uses WHOIS, one can establish that the Hostname is indeed in commercial usage for spamming and so an IP range then becomes effective.

And so I want to put in a good word for the WHOIS tool on IP tracking. That's something else I consider vital.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 13th, 2011, 01:59 AM
Quote
There's that, and I wonder if there's a way to check if an email is a valid address.
That's where email validation comes in. If the address isn't valid up front the account doesn't get opened.
Quote
But sometimes when one uses WHOIS, one can establish that the Hostname is indeed in commercial usage for spamming and so an IP range then becomes effective.
Which is why I'm OK with leaving in facilities for blocking hostnames and email address (at the individual, wildcard or domain levels) but not IP addresses.
Quote
And so I want to put in a good word for the WHOIS tool on IP tracking. That's something else I consider vital.
It won't be vital for long. Not only with all the comments raised above but with the ongoing exhaustion of IPv4 and the real lack of fully deployed IPv6 services, there are going to be some creative mashups in the mean time, most importantly I think we're going to see IP address trading, which means WHOIS blocks may become stale.

There is a very distinct reason that I'm pushing out the IP facilities the way I am. If you can't ban on IP addresses, what use is the IP tracking facility, exactly? Especially when you will see an address like 3ffe:1900:4545:3:200:f8ff:fe21:67cf being thrown at you. And that's a slightly shortened (but entirely valid) IPv6 address.[1]

As I said: "With all that, IP bans are basically useless, except to the most technically inept of users"
 1. Any field that has leading zeroes can be have the leading zeroes truncated. The full address here would be 3ffe:1900:4545:0003:0200:f8ff:fe21:67cf.
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 13th, 2011, 02:02 AM
Dang if that don't resemble an ethernet NIC address...colon delimiters instead of periods.  Okie dokie, then. Carry on.

========================

On second thought, what IPv6 is starting to look like, is really dispensing with IPs altogether and using the NIC address exclusively. That's gonna put the kibosh on portability, machine to machine, if that's so. O_O
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 13th, 2011, 02:07 AM
Quote
Dang if that don't resemble an ethernet NIC address.
Only longer, and it's likely they chose to reuse the MAC format for readability and cutting back on how much will have to be done in terms of handling IPv6 addresses. But really it doesn't make much difference when MAC/NIC addresses are only 48 bits, IPv6 is 128. Ultimately yeah, it soon becomes apparent that we cannot continue to rely on dealing with IP bans when the potential range is so large.

The entire IPv4 spectrum is contained within one /64 subnet of that - as in, 0000:0000:0000:0000:0000:00ff:1234:5678 where 12345678 is the hex version of the address (each two digits is a single part of the x.y.z.a unit) - I must find the piece I saw recently explaining the relative size changes, it had references to how many addresses there were relative to the number of blades of grass on your lawn and stuff like that.
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 13th, 2011, 02:13 AM
I was thinking that the size change may be due to the same reason the phone company needed more area codes...running out of the number of possible unique numerical combinations without repeating because of all those unique different individual machines/subnets demanding online access. That number may very well be akin to the number of blades of grass in one's lawn.

(For the record: I'm fluent in hexadecimal)
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 13th, 2011, 02:24 AM
Quote
.running out of the number of possible unique numerical combinations without repeating because of all those unique different individual machines/subnets demanding online access
Oh, that's essentially what's happened, IPv4 became exhausted at the most coarse levels and is slowly filling up at the most granular of levels. 4.3 billion addresses, coupled with some logistical constraints that cause it to be reduced to a fraction of its potential doesn't take long to be exhausted at a very broad level, when blocks were handed out like candy.

But IPv6 is stupendously large, one might dare say future-proof large. It's split at the 64 bit boundary, so you have the first half for networks and the second half for machines on that network.

That means the system can cope with 18,446,744,073,709,551,615 unique networks (ignoring the all-0 combination that's used to contain the entirety of IPv4 space), and a similar number of machines per network.

Every human being on the planet would need to have billions of devices before that space was overrun. But for all this capacity, it means that we have to start thinking in new ways to deal with it.
Title: Re: Banning, and what I want to do with it
Post by: Clara Listensprechen on July 13th, 2011, 02:32 AM
Once upon a time, broadcasting networks figured on one TV set per household. I'm wondering... if there's such a guy as regularly uses 200 tabs open in his browser, then  there can indeed be such a guy that has a billion machines under one roof, given that some refrigerators, microwave ovens, security systems, and house temperature controls get on the internet.  After all...

"640k should be enough for anybody." --Bill Gates, 1981.

We'll see.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on July 24th, 2011, 04:10 PM
I just realised, there is a strange irony in what I'm proposing here: since half the stuff I'm suggesting revolves around per user permissions of a sort (or, at least, per user restrictions), that would actually require some of the effort of user-specific permissions that I've always been so against.

Though, to be honest, if you think about it, it's not really a problem: the only use cases anyone could ever give me about per user permissions were always per user restrictions, not per user permissions generally.

I've been mulling over exactly how I want to handle this on the 3 hour journey back home, and the conclusion I've come to is this: two slightly different ways of managing it.

The first is to set the generic stuff - that % warning means removing permissions generally. This would be for the main stuff, would see the removal of 'watched' status... seriously how many people actually USE that?

To clarify, the current warning levels are watched, moderated and muted (plus, naturally, banned) - I'm looking at making the 0-100% scale offer the following (in no particular order): post moderation, post/reply muting, PM muting[1], removal of avatar, removal of signature, media item post/comment muting and finally revoking ability to log in (typically at 95+%)

Additionally, the rules on how quickly the warning level drops should be altered; they shouldn't just drop at the same level across the board - it should be configurable per person really.

Then, just for fun, it should be configurable per person as to specific exceptions; you might have it so that 35% is the point at which post moderation cuts in, but that a certain person who is being a pain should be set to moderated regardless of anything else.


Lastly, and probably most importantly: I'm not managing post moderation through the permissions system, I fully intend to rip that out and replace it. While there is a certain elegance of using permissions to identify who can and can't post with/without moderation, the fact is it is actually a pain. Trying to explain to people how it should be set up is ridiculous, and bitter experience teaches me that the approach while convenient from a programming perspective, it's inconvenient from a user's perspective. So I need to rethink that too.
 1. I want to keep muting on posts and muting on PMs separate, so that people who have been muted can plead their case if the admin so wants.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 28th, 2011, 09:17 PM
As a diehard user of SMF who has used SMF for everything from a normal forum to various content management systems on dozens of sites across the internet one of the most irksome things I've had to deal with is the banning system.

Let's take for instance a site that's being hammered and I mean absolutely hammered by Chinese spam bots. I ban an IP Range, good, don't need to see that one here again, move on to the next. and so on and so forth until I've got a list of ban triggers (A concept which I love) 3 to 4 pages long.

I go to another forum I host and I see it having the same problem. Maybe it's running a slightly different version of SMF or maybe it's already got a few bans in the system or what-have-you, but now I'm forced to re-enter all those bans again, by hand. It's so time consuming I had to write a system that'd let me inject them into the list directly via the database.

I would die, 10 fold if the system were as easy as that, if I could just edit a list of CIDR's like this
Code: [Select]
1.0.1.0/24
1.0.2.0/23
1.0.8.0/21
1.0.32.0/19
1.1.0.0/24
1.1.2.0/23
1.1.4.0/22
1.1.8.0/21
1.1.16.0/20
1.1.32.0/19
1.2.0.0/23
1.2.2.0/24
1.2.4.0/22
1.2.8.0/21
1.2.16.0/20
1.2.32.0/19
1.2.64.0/18
1.3.0.0/16
And just add another CIDR to the list, it would save me hours of work.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 28th, 2011, 09:28 PM
I'm still not convinced about IP blocking being that useful, however if it's done it would be matched by CIDR rather than older style ranges. The one problem is converting IPv4 CIDR blocks into something meaningful for our IPv6 implementation.

But note that the CIDR blocking that I'm thinking about would be for bulk blacklisting as you're talking about, not per user banning.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 28th, 2011, 09:34 PM
Right, the per user banning system actually functions quite well already. I understand IPV6 will become an issue in the future and I like that you're considering it now.

As it stands the only thing keeping me from simply blocking them at either the .htaccess level or iptables is that I like to have a log of who's attempting to access what in the individual forums administrative interface, either for my records or other admins.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 28th, 2011, 09:57 PM
No, I think you might misunderstand me. The per user banning is going away in its present form, and per user banning will instead be handled as account level sanctions rather than being looked up against a general list of bans, especially because of the IP bans being unreliable and trivial to get around, but blocking at bigger levels becomes useful for now.

Implementing bans in the current ban system is, frankly, laughable.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 28th, 2011, 10:13 PM
Oh no I got it I was referring to the tools I've got available to me right now. The idea of replacing bans with account level sanctions is great. So long as I have a way to effectively hide that user from the rest of the population. The ability to ghost all his or her posts/threads, make them not show up in the member list to unsanctioned members etc.

If those types of sanctions are something they can't see themselves they are more likely to just go away on their own accord because they appear to be ignored rather than raising a stink and getting the reaction they want from the current members.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 28th, 2011, 10:44 PM
Hmm, I'll see what I can do, global ignore like that wasn't quite part of my plan, but other account level sanctions certainly were.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 28th, 2011, 11:14 PM
Ultimately a troll has to go away and you want to erase all record of that person having been there. Even if you don't get rid of that persons posts you don't want them gumming up your member list/count and if you have to delete their account to do it then those user level sanctions cease to be in effect.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 28th, 2011, 11:21 PM
Very true, yes. But at the same time, I'd rather not slow down forum-wide operations. I mean, we're already going to be seeing a gain because of not having the banning query per page (which is deceptively expensive), so my job is to figure out a way to make that work as desired and be efficient.

It isn't that I don't want to do it - it would be a smart way to get rid of trolls for sure or at least clean up after them - but I guarantee you, selecting a list just based on 'the first ten of those in this topic' is significantly faster than 'the first ten of those in this topic, excluding this other information in this other table'. Even post moderation is actually hurtful in performance as it stands, so maybe we'll end up looking at doing it in a similar fashion to how post moderation is looking to be expanded upon internally.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 28th, 2011, 11:53 PM
Ouch that's true I can see it getting very bogged down very quickly. I wish I had a good answer for that. There's the evercookie approach but ultimately that'd fail to a persistent user.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 12:00 AM
No, you'd have to do it server-side, and frankly the notion of embedding an evercookie just feels wrong. I may want to get rid of persistent pain users, but I don't want to do it by 'below the belt' means if at all possible.

I mean, we still have the same inherent problem with post moderation. Essentially I think it's going to have to be an extension of the post moderation system, just adding a flag to unhide it (and hide all traces of it being hidden, as it were) to the person it affects.

On a side note, would administrators continue to see these posts? What about moderators?
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 29th, 2011, 12:20 AM
Administrators would see ghosted posts and I suppose moderators would be granted the ability to see them or not depending on the admin's preferences.

I like the idea of ghosted posts being collapsed ala reddit where a post is so downvoted it simply falls off the thread or on the wow forums a post that garners enough dislikes is greyed out and replaced with comment deleted for X.

something like this(http://i.imgur.com/NrG07.png).

I was thinking, if I were designing the system it might be worth it to simply remove ghosted posts from the actual table altogether, saving copy to another table and modding the way the forum displays threads on an account level basis. That way the forum isn't doing any extra work for regular users at all, it shows them the actual content of the thread and the only time it does any extra work is when an admin, moderator or the affected individual see a thread.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 12:25 AM
Well, we're looking at implementing deletions much like that anyway (as opposed to the current recycle bin approach) which means it's just another flag to play with. Essentially, the current flag used for approved/not approved is looking to be renamed to a general status flag for that post.

So approved/not approved, deleted, and essentially ghosted, are all variants on the post's status, and can be excluded as such. In fact, thinking about it, that would mean we could create a SET for it, and apply it as a filter, and if it's suitably indexed it shouldn't be too much of a problem to work with, though benchmarking will of course need to be done (on a big enough scale to validate that it isn't going to hurt too much compared to where we are now, though it may even improve things, you never know)

There's an awful lot of other stuff going on at the moment that means it's going to be a little while before I attack this in particular but it is going to get done :)

On a related anti-troll note, it is (and has been for some time) to deny problem groups from given boards (something not readily possible in SMF) so that you can create a 'Sin Bin' or similar group and just give them only a single board to vent in, perhaps to appeal their sanctions.
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 29th, 2011, 12:35 AM
That would be pretty great. The ability to individually flag posts could have a great many uses. Flagging a post as correct in a thread of particularly useful which would change it's style to draw attention could be cool. The ability to jump to certain types of flagged posts in a thread similar to how on the wow forums you can jump ahead in a 60 page thread to one posted by a community manager for instance.

Could be a good way to help users quickly find information they might be looking for in a large thread instead of wading through post after post to find it. Moderators could flag a certain post and it'd maybe throw up a button at the top of the thread somewhere with a custom title that'd take you to said post.

A hundred years ago when I worked on starmen.net's forums we had perhaps 500 active users daily it was necessary to jail some users. We were using FudForum at the time and I had created a Jail'd user group which would take priority over all other permissions aside from administrative permissions and effectively jail a user. Giving them access to the Jail forum that no one else would normally see where they could plead their case and a moderator would answer.

It was extremely effective. We went from banning 2 or 3 people a week to 2 or 3 people a month.

Edit: maybe it was more like 100 active users, but it was a lot :P
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 12:53 AM
Well, the different ways to flag posts raises all kinds of other issues that I don't really want to get into at this stage, as it has all kinds of interesting side effects (not least how a mod would extend it and still have it be done efficiently; with the method I have in mind, there would be a limit of 64 tags that could be attached to a post, but configurability would be... limited)

Actually, thinking about it, this would be useful in solving another issue that I had; I want to be able to provide a style hook of some description to say 'this post was made by the admin' or 'this post was made by a moderator', and I'd never been able to figure out a way of doing it before that made sense (without querying for the users and their permissions each page load) - so while the base might not make use of it, it would be trivial to style a staff member's post much as Blizzard's forums do.

The downside to that method is that it has to be maintained manually and it causes trouble if you demote someone, but it is certainly food for thought (and has yet other consequences that I can envisage)

Note that I'm not thinking of an 'arbitrary' set of tags. It would, for the sake of efficiency, be up to 64 pre-defined tags in the system that would be attached to a post, and adding new ones potentially would cause big issues. (There's a field type in MySQL called SET. It lets you set 64 tags which are either attached or not attached in that value. But it compresses to use the minimum number of bytes. 1-8 fields = 1 byte, but 9 fields = 2 bytes. If you add that ninth field after the table is created, you have to go through and shunt all the data for every row along to fit it in... Hence not making it particularly extensible. But it would be fast.)

And yeah, the Banned group is something I've long thought about implementing. There's a bunch of tricks needed internally to make it work properly (and to make it function meaningfully as a banned group, like not letting you delete it, for one thing) and that users would likely go in and out of it at a different rate to normal assignment, so it would be managed by the system instead. It's complex!
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 29th, 2011, 01:04 AM
I could see that creating an additional problem of which value takes precedence over another. Say a post is flagged a ghosted, posted by a moderator and selected as some other flag like "correct answer". Which style would the post receive?
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 01:09 AM
It doesn't really matter how you did it, if you can attach two or more states to a message that are not expressly exclusive, precedence is going to be a problem. (Even attaching arbitrary tags would have the same problem)

If it's ghosted, I'd argue that should be the most prominent one. After that, moderator. However note that I don't see 'correct answer' being a core item, which does make me think about making it arbitrary states and putting it in a separate table - though that is irrelevant as far as precedence is concerned.

Generally, selectivity is the usual criteria for determining precedence (that something more selective takes precedence over something general) - I'd argue that ghosting is going to be more selective than moderator/admin posting and so should take precedence. (Though, why would you ghost either a moderator post, or a correct answer?)
Title: Re: Banning, and what I want to do with it
Post by: zushiba on October 29th, 2011, 01:55 AM
Perhaps you have a Moderator gone rogue and had to action him or her in a manner that caused a post to be ghosted. Though that's easily taken care of by having ghost set as highest priority. I guess I'm thinking more along the lines of secondary tags not what would be considered a core item stacking up.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 01:59 AM
Yes, I realise that's the problem, as soon as it was mentioned that having multiple tags on a message was on the cards, I realised it would be a problem, being core or not is actually totally irrelevant.
Title: Re: Banning, and what I want to do with it
Post by: Nao on October 29th, 2011, 07:25 PM
Just wanna say one thing about banning and stuff...

I think that in the last 3 months, I've only got one spam message here on wedge.org.
And that's after defining a couple of questions at install time, and using Pete's captcha. I could actually do without the ban system...
I don't think that anything can beat custom questions anyway!
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 07:30 PM
Let me draw your attention to one thing, from the OP.
Quote
What is the ban system for? Primarily it's for getting rid of miscreants, and troublemakers. It isn't really a spam-solving solution, and it's not really for keeping users out that you're not interested in - it's for keeping users out that you don't want, which isn't the same thing.

Now before you start, I'm well aware that users do currently use it for keeping users out that they're not interested in, but on a variety of levels, I'm just not sure how viable that is, but we'll get on that in a minute.
I never intended to overhaul the ban system to keep out spammers because of that very assertion, it's for the trolls etc, that don't take no for an answer and don't give up at a little bit of resistance. People like our friend Clara, for example. The idea is that instead of having a generic blocking system, it should be people-centric to curtail their activities as necessary.

Custom questions do a lot of it, and the custom CAPTCHA is effective in that it hasn't yet gone mainstream and thus not yet been examined or broken by the likes of xrumer, because it's not worth their time and effort to do so - yet. It will, though, as and when Wedge gets popular, however the custom question angle does make a lot of difference.
Title: Re: Banning, and what I want to do with it
Post by: Nao on October 29th, 2011, 07:51 PM
It was mainly in answer to zushiba ;)
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 07:54 PM
*nods* But that's only in terms of spammers actually getting as far as registration - spammers hammering away trying to get in is still a silent issue that's present, and is really what butchs' Forum Firewall was designed to solve.[1]

Mind you, people are realising that Baidu is a problem even though it's a 'legitimate' site, it's a bit heavy on the accessing, so I'm not even sure I want to unblock it from the current exclusions in our Bad Behaviour implementation.
 1. Though, I find it hilarious that he's written in a 'no SMF fork can make use of any of the ideas in this mod' which was written almost entirely because he thought I was trolling him. Sad really, the guy has some good ideas, but has a lot of bad ideas about execution of them.
Title: Re: Banning, and what I want to do with it
Post by: live627 on October 29th, 2011, 11:25 PM
Quote from Arantor on October 29th, 2011, 07:54 PM
Mind you, people are realising that Baidu is a problem even though it's a 'legitimate' site, it's a bit heavy on the accessing, so I'm not even sure I want to unblock it from the current exclusions in our Bad Behaviour implementation.
Google hammers my site more than Baidu, which seems... interesting.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 29th, 2011, 11:26 PM
Quote
Google hammers my site more than Baidu, which seems... interesting.
You have Bad Behaviour, right?

Baidu doesn't typically send an Accept header so BB should be kicking it out.
Title: Re: Banning, and what I want to do with it
Post by: live627 on October 29th, 2011, 11:32 PM
Oh... yeah, I do. But some Baidu engines still get through. Is it possible that some Baidu spiders do pass the Accept header?
Title: Re: Banning, and what I want to do with it
Post by: Nao on October 29th, 2011, 11:39 PM
You can't own ideas anyway. As annoying as it can be, any other fork can get inspiration from our own original ideas. Not code though.

Not that I look into others' work. So I don't bother.
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 30th, 2011, 12:23 AM
Quote
Oh... yeah, I do. But some Baidu engines still get through. Is it possible that some Baidu spiders do pass the Accept header?
Not that I've seen, put it that way.
Quote
You can't own ideas anyway.
Of course you can't. Copyright law doesn't protect ideas, only the implementation of those ideas.
Quote
Not that I look into others' work. So I don't bother.
I look around from time to time to see what's out there, but I don't ever implement anything wholesale without a serious amount of refining and adding my own doses of magic to the mix.
Title: Re: Banning, and what I want to do with it
Post by: Nao on October 30th, 2011, 01:56 AM
Looking at other's work always annoys me.
Either it's crap, and then I'll feel like I've wasted my time.
Or it's excellent, and I'll be jealous.

(From memory, taken from Hemingway in 'Midnight in Paris'. :P)
Title: Re: Banning, and what I want to do with it
Post by: Arantor on October 30th, 2011, 02:06 AM
Ah, I look less at the substance and more of the form. I tend not to look at the code of others unless I have to, but I do look at what is provided and how it operates.