Banning, and what I want to do with it

Arantor

  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Banning, and what I want to do with it
« on July 12th, 2011, 12:56 PM »Last edited on July 12th, 2011, 05:15 PM by Nao/Gilles
While I'm still trying to figure out how to incorporate all the feedback from the package manager changes(!), I thought I'd talk about what I want to do with the ban system. Sorry in advance, this is going to be a bit of a novel: it's a big change, it's probably at least as controversial, and something about it is necessary anyway, so let's dive in.

The ban system as implemented is functional, as in it works but it's not overly elegant, it doesn't support IPv6 and I take the view that it doesn't solve the problem at hand, not one bit.

Let me deal with the IPv6 problem first, before I tackle the other stuff. The current system works on IPv4 addresses, which are x.y.z.a addresses, and whatever you put a ban (on IP address) on, it resolves to a range internally for each of the blocks. So a ban on 1.2.*.* becomes a ban internally on 1.2.0-255.0-255. Structurally, that makes sense, but IPv6 is much larger - instead of 4 blocks in the range of 0-255, you have 16 to contend with, though they're not written in decimal, nor written in the same way, but written as aaaa:bbbb:cccc:dddd:eeee:ffff:0000:1111 and similar.

There is one thing to consider, that addresses are divided in half in IPv6, the first half is for a 'network' and the second half for machines in that network, and it sounds that on the surface you could get away with just barring based on the first half only. Whether that will be successful or practical remains to be seen, but something tells me it's not that practical. It's not even that practical from a technical standpoint because if you're keeping that approach, you're not just comparing 4 values against ranges, but doing it for at least 8 - and you need to handle the high/low values, which is what SMF's and Wedge's system does right now.

I didn't implement IPv6 in Wedge in a way that would make this particular easy to implement for, because I took the view that it was the wrong way to be going about it, that any minor change extending the current direction of implementation to fit either 2x or 4x larger scope was an unnecessary performance headache, as well as a logistical one.

So, I sat back and thought about what I'd really like to be able to use in the ban system, and that lead me to my normal approach of trying to figure out what it is the ban system should be needed for, and what it should be able to do.


What is the ban system for? Primarily it's for getting rid of miscreants, and troublemakers. It isn't really a spam-solving solution, and it's not really for keeping users out that you're not interested in - it's for keeping users out that you don't want, which isn't the same thing.

Now before you start, I'm well aware that users do currently use it for keeping users out that they're not interested in, but on a variety of levels, I'm just not sure how viable that is, but we'll get on that in a minute.

So, dealing with troublemakers. The ban system lets you ban a user by name, email, IP or hostname. So you ban them, they come back under a new name through a proxy. Doesn't solve the problem much. For dealing with trolls and so on, there are better ways of dealing with them instead of slamming the door in their face - the tools used by Annoy User for example, to lock off certain features, plus the warning system that allows you to control whether they can post or whether their posts are moderated.

Of course, none of those things will solve the proxy problem, but the ban system wouldn't anyway. No, the solution is to gently turn up the heat so they don't realise that they're being pushed out, or at least discouraged from posting for whatever reason, and without it being obvious - so that they go and do something somewhere else.

If anything, the face-slam of the door is probably worse, not better, at making them go away - because what happens is that they don't have closure, they're not leaving of (kind of!) their own will, so you get all kinds of hassle as a result.


As for banning on email address, what is the hope of that? If you have miscreants who have their own domain, they can create as many emails as they like, so you just restrict the entire domain - it won't prevent them re-registering, though. So you get the extra account, you ban the entire domain, they try to register a third time and they still register - but this time they're banned and will take the hint. The problem is you've still got more accounts than you wanted in the first place.

Instead, then, how about limiting the email addresses up front? Put in the ability to restrict emails based on domain, either whitelisting or blacklisting certain domains as necessary. I know a number of users that restrict signups from mail.ru because of spam - if the domain is blacklisted, they can't even register (which is better than banning it).

There is, interestingly, a performance consideration here - and one for the better. If you ban based on email, the ban has to be evaluated more frequently than just locking it down at registration/change email time. In fact, that's going to be true of all bans - the more bans you have, the more you have to evaluate, and it has even a per-page consequence. By removing that query, you remove the performance hit, especially on long-term sites that have many bans, most of which aren't needed any longer.


Then we have IP addresses. Hello, darkness, my old friend. Putting aside the considerations of above with IPv6 addresses, the simple fact is that IP bans are really not that effective at keeping out miscreants because of proxies. That said, if you apply any of the measures in something like Annoy User, such users will likely notice it when they log out, or if they use another computer after logging out (so you can't even really use cookies on their computer) - not to mention the fact that IP addresses are shot to bits if you use mobile devices on 3G connections and similar. It's not like you can even reliably block proxy connections here.

With all that, IP bans are basically useless, except to the most technically inept of users - and they certainly don't keep out spammers, there are better ways of doing that which don't require tracking IP addresses, which are only going to be more and more useless for tracking in future as IPv6 goes mainstream.

The only salvage then is hostname, but even that... well, it's typically disabled in a lot of cases because of sluggish performance (usually because hosted machines are behind a laggy rDNS) meaning it's not a lot of use to you, and even if it wasn't, most of the time bans are not carried out on hostnames but on IP addresses, when really, hostnames would be more useful.

The solution then, might be to be able to blacklist certain hostnames if lookups are enabled and functioning, but to use it at a deeper level than keeping the conventional bans on it (there are performance considerations too), and then you could use it only if you needed it. What I might do is integrate that into our Bad Behaviour implementation, making it look like (to the user, anyway) as if their computer has a problem rather than anything else.


That wraps it up for the problems with the ban system and how they can be mitigated, but let's go further: dealing with miscreants needn't stop at fixing the current setup.

So, user-level problems, we deal with at the user level, not some global administrative level. I'm thinking we can expand the warning system as a result. Right now users can be watched, moderated or muted. It's trivial to expand that to full-on banned, and it would be useful to expand how the tail-off works. Right now you can set how quickly the warning level drops for all users (in points per day), but making that per user would make more sense, so that users who just need a time-out can be given one, and it can be done per user, rather than something across the board.

I'm also thinking we could influence other permissions, such as losing avatar and signature if the warning is over a certain level.


Just for fun, there's another subsystem I've been thinking about, that will debut in some form. Specifically, it will allow you to add rules to certain parts of the system, e.g. things to do when a post is made - so you can check the contents of a post, and if it contains words you don't like, it gets moderated and the user can be warned automatically.


Too long, didn't read (tl:dr;) summary:
* Removing the ban system as it is
* Making post moderation more prominent, probably even enabled by default (but with performance tweaks to make it run more efficiently)
* Email blacklist/whitelist on registration/change email, instead of the old method of banning
* Add hostnames to the possible rules that will be checked in our Bad Behaviour setup, so that instead of getting a 'banned warning', it looks like problems with their computer
* Replacing user-level bans with the warning system and making it more granular rather than as coarse as it is right now
* Adding functionality from my old Annoy User mod to encourage bad users to go away
* Expanding the warning system to more gradually remove powers, than just moderated and muted

I don't think I missed anything but if I did, I'm sure you're going to let me know about it!

And please, before telling me you need the ban system as it is, really stop and think about what you use in it and why you use it, then before complaining at me for breaking what you think is an essential feature, think about if there's actually a better way of doing it, like the above. Banning is not a particularly wonderful technique as explained - it doesn't solve any problem, it solves some of the symptoms. I'm trying to solve the deeper problems. Just because something is what it is, doesn't mean you have to accept it.

Oh, one more thing I forgot.

I want to introduce a 'Banned' membergroup that users go into. Not only does it have a visual consideration but a permissions one: it would let you reduce access to boards. I don't know yet whether I want to make that an on/off thing (like banning is now, except it would turn off some boards and maybe show others) or a gradual thing (as you get more warnings, you slowly see fewer and fewer boards)

But that would certainly make life interesting!
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Re: Banning, and what I want to do with it
« Reply #1, on July 12th, 2011, 02:36 PM »
I have been reading it all and cannot see anything missed. I think you have covered there any possible aspect of the problem and also that you have found the perfect solutions for each one of them.

In fact I stopped using the ban system in SMF 2 or 3 years ago because I thought it wasn't working as it should and I already have more or less some of the ideas you have been talking about here (but in a more subliminal level, not as clear as you have expressed them here).

Congratulations.  :cool:

Re: Banning, and what I want to do with it
« Reply #2, on July 12th, 2011, 05:02 PM »Last edited on July 12th, 2011, 05:08 PM by Clara Listensprechen
Well, it seems to me that a solution to spam attacks would provide a solution to your garden variety bad user because all the tools available to spammers (IP spoofing, proxy, multiple email accounts) are also available to the miscreant, and a good number of them will use those.

In banning IPs I've accumulated quite a long, long list of ban triggers (mainly spammer/hackers who attempt to create an account without visiting the board), and when I periodically check on those for how many times they get hit, I find that the majority of the IPs have been used only once, and that was to attempt to hack into the board. Others, just a handful of times, so I'm skeptical about how effective IP banning is on the whole, as even your garden variety miscreant can change IPs.

And a spammer spoofing the IP of a legit user is something else again, although I haven't seen that happen at all this year. A couple of years ago, though, I found that I was banning legit users when I went after the spammers.

That Annoy User is probably the better way to deal with miscreants...or simply restrict that User's Permissions down to nothing. IMHO. IP banning has gotten increasingly more useless over the years, and it has certainly gotten increasingly high-maintenance, taking up a huge chunk of management time (and bandwidth when the list is long?)

I've explored the .htaccess idea, but when you have a long list of IPs, the work factor doesn't diminish. Banning an IP via .htaccess is pretty darn drastic at the get-go.   And I do like the blacklist/whitelist idea.

Re: Banning, and what I want to do with it
« Reply #3, on July 12th, 2011, 05:29 PM »
That's the thing: once you've swung the ban hammer, either they come back with new details - which a ban doesn't solve - or they leave you alone. Either way the ban isn't worth keeping around because it just clogs up the system.
Quote
mainly spammer/hackers who attempt to create an account without visiting the board
Depends what you mean by 'without visiting the board' - registration requires that you go through the right channels, i.e. through the action=register stuff, you can't just go direct to action=register2, it shouldn't let you (and if it does, it's broken!)
Quote
Others, just a handful of times, so I'm skeptical about how effective IP banning is on the whole, as even your garden variety miscreant can change IPs.
That's why I'm actively proposing doing away with it. Hostname blocks are a slightly different kettle of fish because you could block a hostname at the top-most level and be able to block more than a discrete IP range, without worrying about the side effects of over-sized ranges. The only issues then are if you have a poor rDNS, and/or IP spoofing but you'd have those anyway, regardless of anything else.

The key thing here is that I'm taking something that's largely a technical solution to a non-technical problem away, and replacing it with more sociologically-aware tools. Some of which are still technical, but they're not done for the sake of technical convenience.
Quote
That Annoy User is probably the better way to deal with miscreants...or simply restrict that User's Permissions down to nothing.
That's the general plan of mine, yes.

Re: Banning, and what I want to do with it
« Reply #4, on July 12th, 2011, 05:29 PM »
Nice post Arantor

As mentioned elsewhere I already turned to using banned groups in SMF some time ago, 3 for the mainly public sections of the forum and a few others for the specialized sections. Each ban group denies certain permissions while 1 in particular denies all permissions (view/enter board permissions made my setup even more tightly controlled).

You make very good points:
Troubled users generally want closure before they leave a site[1], so much so that 1 of the features of my banning system is that banned users see a specific board where they can appeal the ban levied on them... I actually find that doing this causes them to get out what their issue is and my staff can double check to see if the ban was warranted or not. If a user is placed in the highest ban group all they can do is login if the want to but that's all they get. Their username and email is now hostage they can register with different ones if they choose but I find they generally are much less of an ass if they do.

I've only known of 1 IP-ban issued on my forum (pre-SMF anyway) and the consequences for us were brutal. This guy started using a proxy made our lives a living hell for almost 2 months IIRC, as pissed as we were we could only laugh about it until we just gave up.

There's and SMF mod for white-/black-listing domains but it was never extended to email addresses.

I've never really had much of a spam problem so I can't relate much from experience but your proposal still seems pretty solid overall.
 1. at least that's how it seems to me

Re: Banning, and what I want to do with it
« Reply #5, on July 12th, 2011, 05:34 PM »
Quote
Each ban group denies certain permissions while 1 in particular denies all permissions (view/enter board permissions made my setup even more tightly controlled)
That's really what the ban group proposal is for, for this very reason: to ensure that certain facilities can be revoked if desired.
Quote
Troubled users generally want closure before they leave a site
Yup. Or, sometimes, to argue their defence, depending on the situation. Either way, it's not a simple black and white line, nor should it be.
Quote
Their username and email is now hostage they can register with different ones if they choose but I find they generally are much less of an ass if they do.
Yup. That's something in particular about why I don't delete accounts, nor ban them. If they're a nuisance, I issue a 100% warning and go from there.
Quote
This guy started using a proxy made our lives a living hell for almost 2 months IIRC, as pissed as we were we could only laugh about it until we just gave up.
This is why IP banning is no longer effective ;)

Re: Banning, and what I want to do with it
« Reply #6, on July 12th, 2011, 06:55 PM »Last edited on July 12th, 2011, 07:09 PM by Clara Listensprechen
Yup, it sounds like you've got a solid handle on it, Arantor.

And to explain this...
Quote from Arantor on July 12th, 2011, 05:29 PM
Depends what you mean by 'without visiting the board' - registration requires that you go through the right channels, i.e. through the action=register stuff, you can't just go direct to action=register2, it shouldn't let you (and if it does, it's broken!)
I get newly registered Member accounts with the Last Active status as Never. In other discussions about spammer/hackers elsewhere in cyberspace, it was brought to my attention that there exists registration-bot software and I strongly suspect this is what's been happening: registration-bot registrations.  If SMF is supposed to be designed to not allow that, then somebody needs to re-examine how registration-bots still get in.

Additionally, I have a mod installed that I consider vital: Default Membergroup On Registration. All registered newbies are automatically put in Restricted Newbie Membergroup with access to only one board, not visible to regular Members, requirement of which is to make one post of introduction so that I can tell if they're real people (or miscreant).  Along the lines of Annoy User, I can always put a miscreant in the Restricted Newbie membergroup.

==========================

I should add that my board's traditional policy has been to not use email verification, and if I can avoid changing that policy, I would prefer to avoid it.

Re: Banning, and what I want to do with it
« Reply #7, on July 12th, 2011, 07:13 PM »
I use that mod as well, but you could accomplish what you use it for without that mod by being rather careful with board access and permissions.

I use that mod only for the fact that I don't like having the post count group as a primary group.

Re: Banning, and what I want to do with it
« Reply #8, on July 12th, 2011, 07:21 PM »
OK, I can explain the Last Never behaviour though, it's where they registered but never go back into the forum after the registration completes, usually because of badly written bots.

The whole "default" member group thing is something that we probably should deal with, but I think a wider change is needed to make post count groups not special.

Re: Banning, and what I want to do with it
« Reply #9, on July 12th, 2011, 07:29 PM »
Quote from Arantor on July 12th, 2011, 07:21 PM
OK, I can explain the Last Never behaviour though, it's where they registered but never go back into the forum after the registration completes, usually because of badly written bots.

The whole "default" member group thing is something that we probably should deal with, but I think a wider change is needed to make post count groups not special.
A good beginning point would be to get rid of the automatic "(no primary member group)" arrangement and automatically put them in a specific default member group, IMHO.

And IMHO, if the registrant was last active on the board by registering, the Last Active status should read the date of the registration.

Re: Banning, and what I want to do with it
« Reply #10, on July 12th, 2011, 07:31 PM »
They are in a specific member group. Just it's a special one you can't ever edit, though you are physically attached to it (it is not possible to be in less than two groups, ever, without mashing the members table)

Re: Banning, and what I want to do with it
« Reply #11, on July 12th, 2011, 07:32 PM »
Quote from spoogs on July 12th, 2011, 07:13 PM
I use that mod as well, but you could accomplish what you use it for without that mod by being rather careful with board access and permissions.

I use that mod only for the fact that I don't like having the post count group as a primary group.
The mod is automatic which means I don't have to babysit my board every livin' minute.  I got a life. ;)


Re: Banning, and what I want to do with it
« Reply #13, on July 12th, 2011, 07:36 PM »
Quote from Clara Listensprechen on July 12th, 2011, 07:32 PM
The mod is automatic which means I don't have to babysit my board every livin' minute.  I got a life. ;)
Well, that's the problem. The deal with having a "no primary member group" is that you can set up membergroups based on postcount. I'm saying that shouldn't be the default membergroup a new Member should be in, or any other Member who is not subject to postcount membergroups. "No primary member group" should instead be "Postcount membergroup" and in that one can set up groups dependent on postcount.
Posted: July 12th, 2011, 07:35 PM
Quote from Arantor on July 12th, 2011, 07:32 PM
You can do everything you said automatically, just with a different configuration of groups.
Be that as it may, I swear by the mod. I luvz that mod. I ain't letting it go. so THERE. :P

Re: Banning, and what I want to do with it
« Reply #14, on July 13th, 2011, 12:14 AM »
There is one side effect that occurs to me, though I'm not convinced it's a major one, not enough to drastically alter plans, anyway.

There is, specifically, one edge case in SMF, where you ban on email address, do so on a domain or other wildcard level, and it ends up banning one or more other users because of the ban.

Under the new proposal, this would simply allow them to remain until they updated their email next, so I'm going to have to factor into the process some method whereby when the admin adds a blacklisted email, it checks existing emails and lets the admin do something about it (e.g. sending the user a PM, or a warning or whatever)

Re: Banning, and what I want to do with it
« Reply #15, on July 13th, 2011, 01:48 AM »
There's that, and I wonder if there's a way to check if an email is a valid address. I've had spammers use amazonaws.com as an email address and I've had no problems banning *@amazonaws.com because the deceit is obvious. In other cases, the deceit isn't as obvious, ie @gmail.com . But sometimes when one uses WHOIS, one can establish that the Hostname is indeed in commercial usage for spamming and so an IP range then becomes effective.

And so I want to put in a good word for the WHOIS tool on IP tracking. That's something else I consider vital.