Banning, and what I want to do with it


  • As powerful as possible, as complex as necessary.
  • Posts: 14,278
Banning, and what I want to do with it
« on July 12th, 2011, 12:56 PM »Last edited on July 12th, 2011, 05:15 PM by Nao/Gilles
While I'm still trying to figure out how to incorporate all the feedback from the package manager changes(!), I thought I'd talk about what I want to do with the ban system. Sorry in advance, this is going to be a bit of a novel: it's a big change, it's probably at least as controversial, and something about it is necessary anyway, so let's dive in.

The ban system as implemented is functional, as in it works but it's not overly elegant, it doesn't support IPv6 and I take the view that it doesn't solve the problem at hand, not one bit.

Let me deal with the IPv6 problem first, before I tackle the other stuff. The current system works on IPv4 addresses, which are x.y.z.a addresses, and whatever you put a ban (on IP address) on, it resolves to a range internally for each of the blocks. So a ban on 1.2.*.* becomes a ban internally on 1.2.0-255.0-255. Structurally, that makes sense, but IPv6 is much larger - instead of 4 blocks in the range of 0-255, you have 16 to contend with, though they're not written in decimal, nor written in the same way, but written as aaaa:bbbb:cccc:dddd:eeee:ffff:0000:1111 and similar.

There is one thing to consider, that addresses are divided in half in IPv6, the first half is for a 'network' and the second half for machines in that network, and it sounds that on the surface you could get away with just barring based on the first half only. Whether that will be successful or practical remains to be seen, but something tells me it's not that practical. It's not even that practical from a technical standpoint because if you're keeping that approach, you're not just comparing 4 values against ranges, but doing it for at least 8 - and you need to handle the high/low values, which is what SMF's and Wedge's system does right now.

I didn't implement IPv6 in Wedge in a way that would make this particular easy to implement for, because I took the view that it was the wrong way to be going about it, that any minor change extending the current direction of implementation to fit either 2x or 4x larger scope was an unnecessary performance headache, as well as a logistical one.

So, I sat back and thought about what I'd really like to be able to use in the ban system, and that lead me to my normal approach of trying to figure out what it is the ban system should be needed for, and what it should be able to do.

What is the ban system for? Primarily it's for getting rid of miscreants, and troublemakers. It isn't really a spam-solving solution, and it's not really for keeping users out that you're not interested in - it's for keeping users out that you don't want, which isn't the same thing.

Now before you start, I'm well aware that users do currently use it for keeping users out that they're not interested in, but on a variety of levels, I'm just not sure how viable that is, but we'll get on that in a minute.

So, dealing with troublemakers. The ban system lets you ban a user by name, email, IP or hostname. So you ban them, they come back under a new name through a proxy. Doesn't solve the problem much. For dealing with trolls and so on, there are better ways of dealing with them instead of slamming the door in their face - the tools used by Annoy User for example, to lock off certain features, plus the warning system that allows you to control whether they can post or whether their posts are moderated.

Of course, none of those things will solve the proxy problem, but the ban system wouldn't anyway. No, the solution is to gently turn up the heat so they don't realise that they're being pushed out, or at least discouraged from posting for whatever reason, and without it being obvious - so that they go and do something somewhere else.

If anything, the face-slam of the door is probably worse, not better, at making them go away - because what happens is that they don't have closure, they're not leaving of (kind of!) their own will, so you get all kinds of hassle as a result.

As for banning on email address, what is the hope of that? If you have miscreants who have their own domain, they can create as many emails as they like, so you just restrict the entire domain - it won't prevent them re-registering, though. So you get the extra account, you ban the entire domain, they try to register a third time and they still register - but this time they're banned and will take the hint. The problem is you've still got more accounts than you wanted in the first place.

Instead, then, how about limiting the email addresses up front? Put in the ability to restrict emails based on domain, either whitelisting or blacklisting certain domains as necessary. I know a number of users that restrict signups from because of spam - if the domain is blacklisted, they can't even register (which is better than banning it).

There is, interestingly, a performance consideration here - and one for the better. If you ban based on email, the ban has to be evaluated more frequently than just locking it down at registration/change email time. In fact, that's going to be true of all bans - the more bans you have, the more you have to evaluate, and it has even a per-page consequence. By removing that query, you remove the performance hit, especially on long-term sites that have many bans, most of which aren't needed any longer.

Then we have IP addresses. Hello, darkness, my old friend. Putting aside the considerations of above with IPv6 addresses, the simple fact is that IP bans are really not that effective at keeping out miscreants because of proxies. That said, if you apply any of the measures in something like Annoy User, such users will likely notice it when they log out, or if they use another computer after logging out (so you can't even really use cookies on their computer) - not to mention the fact that IP addresses are shot to bits if you use mobile devices on 3G connections and similar. It's not like you can even reliably block proxy connections here.

With all that, IP bans are basically useless, except to the most technically inept of users - and they certainly don't keep out spammers, there are better ways of doing that which don't require tracking IP addresses, which are only going to be more and more useless for tracking in future as IPv6 goes mainstream.

The only salvage then is hostname, but even that... well, it's typically disabled in a lot of cases because of sluggish performance (usually because hosted machines are behind a laggy rDNS) meaning it's not a lot of use to you, and even if it wasn't, most of the time bans are not carried out on hostnames but on IP addresses, when really, hostnames would be more useful.

The solution then, might be to be able to blacklist certain hostnames if lookups are enabled and functioning, but to use it at a deeper level than keeping the conventional bans on it (there are performance considerations too), and then you could use it only if you needed it. What I might do is integrate that into our Bad Behaviour implementation, making it look like (to the user, anyway) as if their computer has a problem rather than anything else.

That wraps it up for the problems with the ban system and how they can be mitigated, but let's go further: dealing with miscreants needn't stop at fixing the current setup.

So, user-level problems, we deal with at the user level, not some global administrative level. I'm thinking we can expand the warning system as a result. Right now users can be watched, moderated or muted. It's trivial to expand that to full-on banned, and it would be useful to expand how the tail-off works. Right now you can set how quickly the warning level drops for all users (in points per day), but making that per user would make more sense, so that users who just need a time-out can be given one, and it can be done per user, rather than something across the board.

I'm also thinking we could influence other permissions, such as losing avatar and signature if the warning is over a certain level.

Just for fun, there's another subsystem I've been thinking about, that will debut in some form. Specifically, it will allow you to add rules to certain parts of the system, e.g. things to do when a post is made - so you can check the contents of a post, and if it contains words you don't like, it gets moderated and the user can be warned automatically.

Too long, didn't read (tl:dr;) summary:
* Removing the ban system as it is
* Making post moderation more prominent, probably even enabled by default (but with performance tweaks to make it run more efficiently)
* Email blacklist/whitelist on registration/change email, instead of the old method of banning
* Add hostnames to the possible rules that will be checked in our Bad Behaviour setup, so that instead of getting a 'banned warning', it looks like problems with their computer
* Replacing user-level bans with the warning system and making it more granular rather than as coarse as it is right now
* Adding functionality from my old Annoy User mod to encourage bad users to go away
* Expanding the warning system to more gradually remove powers, than just moderated and muted

I don't think I missed anything but if I did, I'm sure you're going to let me know about it!

And please, before telling me you need the ban system as it is, really stop and think about what you use in it and why you use it, then before complaining at me for breaking what you think is an essential feature, think about if there's actually a better way of doing it, like the above. Banning is not a particularly wonderful technique as explained - it doesn't solve any problem, it solves some of the symptoms. I'm trying to solve the deeper problems. Just because something is what it is, doesn't mean you have to accept it.

Oh, one more thing I forgot.

I want to introduce a 'Banned' membergroup that users go into. Not only does it have a visual consideration but a permissions one: it would let you reduce access to boards. I don't know yet whether I want to make that an on/off thing (like banning is now, except it would turn off some boards and maybe show others) or a gradual thing (as you get more warnings, you slowly see fewer and fewer boards)

But that would certainly make life interesting!
When we unite against a common enemy that attacks our ethos, it nurtures group solidarity. Trolls are sensational, yes, but we keep everyone honest. | Game Memorial

Re: Banning, and what I want to do with it
« Reply #16, on July 13th, 2011, 01:59 AM »
There's that, and I wonder if there's a way to check if an email is a valid address.
That's where email validation comes in. If the address isn't valid up front the account doesn't get opened.
But sometimes when one uses WHOIS, one can establish that the Hostname is indeed in commercial usage for spamming and so an IP range then becomes effective.
Which is why I'm OK with leaving in facilities for blocking hostnames and email address (at the individual, wildcard or domain levels) but not IP addresses.
And so I want to put in a good word for the WHOIS tool on IP tracking. That's something else I consider vital.
It won't be vital for long. Not only with all the comments raised above but with the ongoing exhaustion of IPv4 and the real lack of fully deployed IPv6 services, there are going to be some creative mashups in the mean time, most importantly I think we're going to see IP address trading, which means WHOIS blocks may become stale.

There is a very distinct reason that I'm pushing out the IP facilities the way I am. If you can't ban on IP addresses, what use is the IP tracking facility, exactly? Especially when you will see an address like 3ffe:1900:4545:3:200:f8ff:fe21:67cf being thrown at you. And that's a slightly shortened (but entirely valid) IPv6 address.[1]

As I said: "With all that, IP bans are basically useless, except to the most technically inept of users"
 1. Any field that has leading zeroes can be have the leading zeroes truncated. The full address here would be 3ffe:1900:4545:0003:0200:f8ff:fe21:67cf.

Re: Banning, and what I want to do with it
« Reply #17, on July 13th, 2011, 02:02 AM »
Dang if that don't resemble an ethernet NIC address...colon delimiters instead of periods.  Okie dokie, then. Carry on.


On second thought, what IPv6 is starting to look like, is really dispensing with IPs altogether and using the NIC address exclusively. That's gonna put the kibosh on portability, machine to machine, if that's so. O_O

Re: Banning, and what I want to do with it
« Reply #18, on July 13th, 2011, 02:07 AM »
Dang if that don't resemble an ethernet NIC address.
Only longer, and it's likely they chose to reuse the MAC format for readability and cutting back on how much will have to be done in terms of handling IPv6 addresses. But really it doesn't make much difference when MAC/NIC addresses are only 48 bits, IPv6 is 128. Ultimately yeah, it soon becomes apparent that we cannot continue to rely on dealing with IP bans when the potential range is so large.

The entire IPv4 spectrum is contained within one /64 subnet of that - as in, 0000:0000:0000:0000:0000:00ff:1234:5678 where 12345678 is the hex version of the address (each two digits is a single part of the x.y.z.a unit) - I must find the piece I saw recently explaining the relative size changes, it had references to how many addresses there were relative to the number of blades of grass on your lawn and stuff like that.

Re: Banning, and what I want to do with it
« Reply #19, on July 13th, 2011, 02:13 AM »
I was thinking that the size change may be due to the same reason the phone company needed more area codes...running out of the number of possible unique numerical combinations without repeating because of all those unique different individual machines/subnets demanding online access. That number may very well be akin to the number of blades of grass in one's lawn.

(For the record: I'm fluent in hexadecimal)

Re: Banning, and what I want to do with it
« Reply #20, on July 13th, 2011, 02:24 AM »
.running out of the number of possible unique numerical combinations without repeating because of all those unique different individual machines/subnets demanding online access
Oh, that's essentially what's happened, IPv4 became exhausted at the most coarse levels and is slowly filling up at the most granular of levels. 4.3 billion addresses, coupled with some logistical constraints that cause it to be reduced to a fraction of its potential doesn't take long to be exhausted at a very broad level, when blocks were handed out like candy.

But IPv6 is stupendously large, one might dare say future-proof large. It's split at the 64 bit boundary, so you have the first half for networks and the second half for machines on that network.

That means the system can cope with 18,446,744,073,709,551,615 unique networks (ignoring the all-0 combination that's used to contain the entirety of IPv4 space), and a similar number of machines per network.

Every human being on the planet would need to have billions of devices before that space was overrun. But for all this capacity, it means that we have to start thinking in new ways to deal with it.

Re: Banning, and what I want to do with it
« Reply #21, on July 13th, 2011, 02:32 AM »
Once upon a time, broadcasting networks figured on one TV set per household. I'm wondering... if there's such a guy as regularly uses 200 tabs open in his browser, then  there can indeed be such a guy that has a billion machines under one roof, given that some refrigerators, microwave ovens, security systems, and house temperature controls get on the internet.  After all...

"640k should be enough for anybody." --Bill Gates, 1981.

We'll see.

Re: Banning, and what I want to do with it
« Reply #22, on July 24th, 2011, 04:10 PM »
I just realised, there is a strange irony in what I'm proposing here: since half the stuff I'm suggesting revolves around per user permissions of a sort (or, at least, per user restrictions), that would actually require some of the effort of user-specific permissions that I've always been so against.

Though, to be honest, if you think about it, it's not really a problem: the only use cases anyone could ever give me about per user permissions were always per user restrictions, not per user permissions generally.

I've been mulling over exactly how I want to handle this on the 3 hour journey back home, and the conclusion I've come to is this: two slightly different ways of managing it.

The first is to set the generic stuff - that % warning means removing permissions generally. This would be for the main stuff, would see the removal of 'watched' status... seriously how many people actually USE that?

To clarify, the current warning levels are watched, moderated and muted (plus, naturally, banned) - I'm looking at making the 0-100% scale offer the following (in no particular order): post moderation, post/reply muting, PM muting[1], removal of avatar, removal of signature, media item post/comment muting and finally revoking ability to log in (typically at 95+%)

Additionally, the rules on how quickly the warning level drops should be altered; they shouldn't just drop at the same level across the board - it should be configurable per person really.

Then, just for fun, it should be configurable per person as to specific exceptions; you might have it so that 35% is the point at which post moderation cuts in, but that a certain person who is being a pain should be set to moderated regardless of anything else.

Lastly, and probably most importantly: I'm not managing post moderation through the permissions system, I fully intend to rip that out and replace it. While there is a certain elegance of using permissions to identify who can and can't post with/without moderation, the fact is it is actually a pain. Trying to explain to people how it should be set up is ridiculous, and bitter experience teaches me that the approach while convenient from a programming perspective, it's inconvenient from a user's perspective. So I need to rethink that too.
 1. I want to keep muting on posts and muting on PMs separate, so that people who have been muted can plead their case if the admin so wants.

Re: Banning, and what I want to do with it
« Reply #23, on October 28th, 2011, 09:17 PM »
As a diehard user of SMF who has used SMF for everything from a normal forum to various content management systems on dozens of sites across the internet one of the most irksome things I've had to deal with is the banning system.

Let's take for instance a site that's being hammered and I mean absolutely hammered by Chinese spam bots. I ban an IP Range, good, don't need to see that one here again, move on to the next. and so on and so forth until I've got a list of ban triggers (A concept which I love) 3 to 4 pages long.

I go to another forum I host and I see it having the same problem. Maybe it's running a slightly different version of SMF or maybe it's already got a few bans in the system or what-have-you, but now I'm forced to re-enter all those bans again, by hand. It's so time consuming I had to write a system that'd let me inject them into the list directly via the database.

I would die, 10 fold if the system were as easy as that, if I could just edit a list of CIDR's like this
Code: [Select]
And just add another CIDR to the list, it would save me hours of work.

Re: Banning, and what I want to do with it
« Reply #24, on October 28th, 2011, 09:28 PM »
I'm still not convinced about IP blocking being that useful, however if it's done it would be matched by CIDR rather than older style ranges. The one problem is converting IPv4 CIDR blocks into something meaningful for our IPv6 implementation.

But note that the CIDR blocking that I'm thinking about would be for bulk blacklisting as you're talking about, not per user banning.

Re: Banning, and what I want to do with it
« Reply #25, on October 28th, 2011, 09:34 PM »
Right, the per user banning system actually functions quite well already. I understand IPV6 will become an issue in the future and I like that you're considering it now.

As it stands the only thing keeping me from simply blocking them at either the .htaccess level or iptables is that I like to have a log of who's attempting to access what in the individual forums administrative interface, either for my records or other admins.

Re: Banning, and what I want to do with it
« Reply #26, on October 28th, 2011, 09:57 PM »
No, I think you might misunderstand me. The per user banning is going away in its present form, and per user banning will instead be handled as account level sanctions rather than being looked up against a general list of bans, especially because of the IP bans being unreliable and trivial to get around, but blocking at bigger levels becomes useful for now.

Implementing bans in the current ban system is, frankly, laughable.

Re: Banning, and what I want to do with it
« Reply #27, on October 28th, 2011, 10:13 PM »
Oh no I got it I was referring to the tools I've got available to me right now. The idea of replacing bans with account level sanctions is great. So long as I have a way to effectively hide that user from the rest of the population. The ability to ghost all his or her posts/threads, make them not show up in the member list to unsanctioned members etc.

If those types of sanctions are something they can't see themselves they are more likely to just go away on their own accord because they appear to be ignored rather than raising a stink and getting the reaction they want from the current members.

Re: Banning, and what I want to do with it
« Reply #28, on October 28th, 2011, 10:44 PM »
Hmm, I'll see what I can do, global ignore like that wasn't quite part of my plan, but other account level sanctions certainly were.

Re: Banning, and what I want to do with it
« Reply #29, on October 28th, 2011, 11:14 PM »
Ultimately a troll has to go away and you want to erase all record of that person having been there. Even if you don't get rid of that persons posts you don't want them gumming up your member list/count and if you have to delete their account to do it then those user level sanctions cease to be in effect.

Re: Banning, and what I want to do with it
« Reply #30, on October 28th, 2011, 11:21 PM »
Very true, yes. But at the same time, I'd rather not slow down forum-wide operations. I mean, we're already going to be seeing a gain because of not having the banning query per page (which is deceptively expensive), so my job is to figure out a way to make that work as desired and be efficient.

It isn't that I don't want to do it - it would be a smart way to get rid of trolls for sure or at least clean up after them - but I guarantee you, selecting a list just based on 'the first ten of those in this topic' is significantly faster than 'the first ten of those in this topic, excluding this other information in this other table'. Even post moderation is actually hurtful in performance as it stands, so maybe we'll end up looking at doing it in a similar fashion to how post moderation is looking to be expanded upon internally.