I’m guessing you’re not a programmer yourself? Because it’s really really not that east to /just/ detect in the server side, hacks can be super sofisticsted these days and there are often many client side exploits that you simply cannot detect serverside.
Using rootkit anti-cheat is a shortcut that reduces cost for both dev time and hosting time at the expense of your customers’ security and CPU. You also have to lay your cards on the table for those who are attacking you. It is not the right solution for this problem.
Authoritative servers.
Never trust the client, especially with information the player shouldn’t have right now.
Look at behaviors and group players based on if you think they cheat or not - let the cheaters play together, no need to spoil their fun and let them realize you know they cheat.
People do some or all of this on the server now, but root kitting all machines to try to solve this problem to play video games is one of the dumbest approaches ever and we will realize it one day when a state level actor pops their zero day against a big install base.
This. Having worked on some in-house anti-cheat solutions myself, it absolutely is just offsetting the processing and security cost to the players. The attack vector of having such a rootkit running on so many devices is just not even close to be worth the trade off of catching marginally (if really measurably at all?) more cheaters.
But… have you considered having control of 0-ring software that runs on hundreds of millions of computers, that can perform targetted updates to change behaviour on just a select few computers, even interact with the network adapters unbeknownst to the OS.
I’m not talking about zero days popping up for this. But rather, this being part of the design?
A less nefarious application: The root kit anti cheats already continuously monitor processes. Say it finds a crypto mining one. It can request the instructions needed to search for a wallet and snatch that off.
A more nefarious one: RK is known to be in the device owned by the kid of a military contractor. Etc.
Trusting the client is a fools errand. So we are in complete agreement. I never understood why the effort isn’t placed on server side. People are very good at knowing when others have cheated. They know this from information that exists on the server side, so with the correct classifier, the server should also be able to know this.
Never trust the client, especially with information the player shouldn’t have right now.
This is a big part of the problem, but it’s not the only problem. If you do all of that stuff right, you can’t build a responsive first person shooter. There’s some level of trust you need to put in the client.
Disclaimer: This is based on my experience playing shooters and as a programmer. I have not worked on anticheat systems hands on.
We see less and less of the “god mode” hacks where players can send the packet for a carpet bomb and the server just blindly trusts it. Or the ludicrous spinbots that spin at an extreme speed and headshot anyone that comes into line of sight.
What we’re seeing is increasingly sophisticated cheats that provide “buffs” to a player’s ability. An AI enhanced aimbot that when you click gently nudges your hand to “auto correct” the shot and then clicks is borderline impossible to detect server side. It looks just like a player moved the mouse and fired.
The “best” method to prevent these folks from cheating seems to be to detect the system or the game has been tampered with.
Maybe the way to deal with that is to just let it happen and deal with smurfs down ranking… So these “soft” cheaters just exist in the “pro tier” where the pros can possibly stand a chance.
One strategy I have seen that I wish more developers would do is sending “honeypot” information to the game client (like a player on the other side of the wall that isn’t really there but an aimbot or a wall hack might incorrectly expose).
Maybe the increasing presence of hardware cheats will result in new strategies that make these things unnecessary. I keep wondering if a TPM could be used to solve this problem someday… But I’m not sure exactly how/we may need faster TPMs.
I think by the end of your message you were starting to arc around a little bit to the right way you need to think about clients: as outside your security envelope. (TPM is a joke in my mind, just like client side anti-cheat.)
There are many ways to try to identify and stop cheating on the server side that have not been explored because executives have directed use of off-the-shelf anti-cheat because they do not understand why it is snake oil.
I thought this at first as well, but they have an interesting property.
They have a manufacturer signed private key. If you get the public key from the manufacturer of the TPM, you can actually verify that the TPM as it was designed by the manufacturer performed the work.
That’s a really interesting property because for the first time there’s a way to verify what hardware is doing over the network via cryptography.
Hmmm… I was going to say no because it’s asymmetric crypto, but you’re right if you are somehow able to extract the signed private key, you can still lie… Good point
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
Even on consoles after a while it becomes obvious that you cannot control the hardware, let alone the software on the client side. Those are the very best argument for this kind of approach and they get cracked eventually.
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
That’s true, if the player suddenly has higher kill rates. However, that doesn’t work if they’ve been using the cheat from the start on that account. A sufficiently advanced AI powered aim bot would also be nearly indistinguishable from a professional player. Kind of similar to how Google created the CAPTCHA that uses mouse movement … but had to go back to (at least in some cases) the additional old school captcha.
I’m a programmer, yes it is. It’s not easy in the sense of easy to implement, it’s easy in the sense that everything else is impossible. Client-side anti-cheat is impossible, and by that I don’t mean hard, I mean perpetual-motion level of impossibility. If someone tells you they implemented a foolproof client-side anti-cheat you should be just as skeptical as if someone tells you they created a perpetual motion. It’s impossible, never going to happen, want an example? Robot using a camera to watch the screen and directly moving the mouse and keyboard, completely undetectable from the client side.
From the server perspective the person is cheating or is behaving like a human. If they’re behaving like a human their behavior is completely indistinguishable from a human, so who cares if they’re cheating?, whatever they’re doing has them still at human level so if the game has skill based matchmaking (which most of these games do) he’ll rise up until his cheating puts him in the same level of more skilled humans and everyone has fun. If he keeps rising forever he’s not on a human level, therefore a cheater. More importantly this also penalizes people who buy bot leveled accounts, because their matches will be all against people they can’t hope to win and the game will not be fun.
Server side can also trick clients into giving up that they’re cheating, e.g. sending ghosts behind walls to check for wall hacks or other similar things to gauge player responses.
But what do I know? I’m just a senior programmer who’s been working on servers for some years. l never worked on the client side anti-cheat though, also never tried to build a perpetual motion machine.
It’s not easy, but it’s really not worth the massive gaping security vulnerability you are giving your users. One disgruntled employee giving out the keys to the castle or one programmer plugging in an infected USB, and every user now has a persistent malicious rootkit. The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
This can’t be right.
Don’t throw your hard drive in the trash. Quarantine the infected computer, and then wipe that hoe and slap your choice of OS back on it and scan/monitor to see if any issues arise.
Edit: since folks may or may not read though the rest of the conversation: I am wrong, throw that SSD/HDD in the garbage like barbarian said.
I’m sorry to disappoint, but with rootkits, that is very real. With that level of permissions, it can rewrite HDD/SSD drivers to install malware on boot.
There’s even malware that can rewrite BIOS/UEFI, in which case the whole motherboard has to go in the bin. That’s much less likely due to the complexity though, but it does exist.
Outside of monitoring individual packets outside of your computer (as in, man in the middle yourself with a spare computer and hoping the malware phones home right when you’re looking) there’s no way of knowing.
Once ring 0 is compromised, nothing your computer says can be trusted. A compromised OS can lie to anti-malware scanners, hide things from the installed software list and process manager, and just generally not show you what it doesnt want to show you. “Just remediate” does not work with rootkits.
Dude… That’s fucked. They should really go a little more in depth on rootkits in the CompTIA A+ study material. I mean, I get that it’s supposed to be a foundational over view of most IT concepts, but it would have helped me not look dumb.
Please don’t walk away from this feeling dumb. Most IT professionals aren’t aware of the scale of the issue outside of sysadmin and cybersecurity. I’ve met programmers who shrug at the most egregious vulnerabilities, and vendors who want us to put dangerous stuff on our servers. Security just isn’t taken as seriously as it should be.
Unrelated, but I wish you the best of luck with your studies!
Could they harden their clients somehow or maybe randomize memory locations for things? Seems like their should be a better solution than installing malware to prevent cheating.
You’re asking good questions but factor this in: a development team at a game company will only want to spend as little time as possible on this process: it doesn’t make them more money - it costs it. Conversely a hacker / cheater is being paid (or gaining) directly from breaking this code. Which is more motivated? Now remember that the protection has to be in place first. Who has the advantage? Client side code will always be breakable. A rootkit doesn’t change the game - it just adds a new vector to attack for other hackers to exploit.
I’ve long believed that the main point of client-side anti-cheat is to serve as security theater.
If the player sees “PROTECTED BY ACME ANTI-CHEAT” on the boot screen of a game, they’re less likely to cry wolf when they get their ass kicked. At least, until they see a blatant example of hacking and lose all faith in the ability of the platform to protect them from it; from that point on, everyone better than them must be cheating from their perspective (speaking from firsthand experience here).
Given how infamously toxic and high-strung the LoL community is, I can only imagine that Riot’s basically at the end of their rope here. If you read the original forum post, they sure make this sound like a Hail Mary. “Sorry, it’s just what we have to do to make sure the game is fair.”
Hilariously, they even undercut their own points in the FAQ:
Q: If Vanguard is so good, why do I still see cheats on VALORANT?
For starters, we do not action every cheat or account instantly. Every ban is like broadcasting a signal to the developer that their cheat has been detected and that they need to “update” it. In order to slow the progression of our “cheat arms race,” we delay bans based on the sophistication and visibility of the cheat and cheater, respectively.
But also, cheaters gonna cheat. [Emphasis mine.] We’ve really driven our preventative layer as far as we can feasibly go without colliding with existing setups and hurting legitimate players. [Linux players aren’t legitimate I guess?]
Also, they’re apparently not bothering enabling Vanguard on OS X because apparently few people have actually developed cheats on it yet. Really tells you what’s the more developer friendly platform, Linux or OS X, doesn’t it? Or maybe the OS X market share is too small to care.
They do also mention using machine learning to detect cheating server-side but lament that it’s not always enough information, and that cheat developers have added “humanization” elements that play more like humans.
My thought is… if a cheat doesn’t make someone obviously better than a human player of a certain skill level, then what does it really matter? Congratulations, you made a bot that’s indistinguishable from a human, thanks for padding our player numbers.
The real problem is that botters don’t pay for microtransactions. And players who buy bot-leveled accounts probably don’t spend a ton either. Why would they? They got everything unlocked for them, they didn’t have to grind for it. That’s all Riot really gives a shit about.
In practice, client side anti cheat is essentially DOA because hardware cheats that analyse the player’s screen on a 2nd computer and proxy inputs to your mouse USB have made it so cheat clients are never actually executing code on the host machine.
At that point, even players cant tell someone is cheating because the cheats aren’t modifying the game state in a noticeable way- they’re still weak to effects that obscure your vision and have inputs that are difficult to differentiate from a “real” player.
IMO cheating is a social problem and one that is totally impossible to beat with rootkits by design.
It’s not easy. And league is free. So banning people won’t work well either. They can’t ban ip addresses either without banning college campuses, some apartment buildings, and Internet cafes.
There are solutions to this problem but they don’t want to permanently ban them. A ban = a new registration… maybe even two. Bonus! You get to pad your ban numbers and user registration numbers at the same time!
But that wastes their clockcycles to make sure you’re not cheating. So much easier to make everyone’s experience worse so they don’t have to upgrade and build out more servers.
Stop stealing our CPU cycles for high risk rootkits and start mitigating and detecting cheating on the server.
It’s that easy.
I stopped playing games that want this bullshit. Don’t need that shit in my life.
I’m guessing you’re not a programmer yourself? Because it’s really really not that east to /just/ detect in the server side, hacks can be super sofisticsted these days and there are often many client side exploits that you simply cannot detect serverside.
Actually, I am.
Using rootkit anti-cheat is a shortcut that reduces cost for both dev time and hosting time at the expense of your customers’ security and CPU. You also have to lay your cards on the table for those who are attacking you. It is not the right solution for this problem.
Authoritative servers. Never trust the client, especially with information the player shouldn’t have right now. Look at behaviors and group players based on if you think they cheat or not - let the cheaters play together, no need to spoil their fun and let them realize you know they cheat.
People do some or all of this on the server now, but root kitting all machines to try to solve this problem to play video games is one of the dumbest approaches ever and we will realize it one day when a state level actor pops their zero day against a big install base.
This. Having worked on some in-house anti-cheat solutions myself, it absolutely is just offsetting the processing and security cost to the players. The attack vector of having such a rootkit running on so many devices is just not even close to be worth the trade off of catching marginally (if really measurably at all?) more cheaters.
But… have you considered having control of 0-ring software that runs on hundreds of millions of computers, that can perform targetted updates to change behaviour on just a select few computers, even interact with the network adapters unbeknownst to the OS.
I’m not talking about zero days popping up for this. But rather, this being part of the design?
A less nefarious application: The root kit anti cheats already continuously monitor processes. Say it finds a crypto mining one. It can request the instructions needed to search for a wallet and snatch that off.
A more nefarious one: RK is known to be in the device owned by the kid of a military contractor. Etc.
Trusting the client is a fools errand. So we are in complete agreement. I never understood why the effort isn’t placed on server side. People are very good at knowing when others have cheated. They know this from information that exists on the server side, so with the correct classifier, the server should also be able to know this.
This is a big part of the problem, but it’s not the only problem. If you do all of that stuff right, you can’t build a responsive first person shooter. There’s some level of trust you need to put in the client.
Disclaimer: This is based on my experience playing shooters and as a programmer. I have not worked on anticheat systems hands on.
We see less and less of the “god mode” hacks where players can send the packet for a carpet bomb and the server just blindly trusts it. Or the ludicrous spinbots that spin at an extreme speed and headshot anyone that comes into line of sight.
What we’re seeing is increasingly sophisticated cheats that provide “buffs” to a player’s ability. An AI enhanced aimbot that when you click gently nudges your hand to “auto correct” the shot and then clicks is borderline impossible to detect server side. It looks just like a player moved the mouse and fired.
The “best” method to prevent these folks from cheating seems to be to detect the system or the game has been tampered with.
Maybe the way to deal with that is to just let it happen and deal with smurfs down ranking… So these “soft” cheaters just exist in the “pro tier” where the pros can possibly stand a chance.
One strategy I have seen that I wish more developers would do is sending “honeypot” information to the game client (like a player on the other side of the wall that isn’t really there but an aimbot or a wall hack might incorrectly expose).
Maybe the increasing presence of hardware cheats will result in new strategies that make these things unnecessary. I keep wondering if a TPM could be used to solve this problem someday… But I’m not sure exactly how/we may need faster TPMs.
I think by the end of your message you were starting to arc around a little bit to the right way you need to think about clients: as outside your security envelope. (TPM is a joke in my mind, just like client side anti-cheat.)
There are many ways to try to identify and stop cheating on the server side that have not been explored because executives have directed use of off-the-shelf anti-cheat because they do not understand why it is snake oil.
I thought this at first as well, but they have an interesting property.
They have a manufacturer signed private key. If you get the public key from the manufacturer of the TPM, you can actually verify that the TPM as it was designed by the manufacturer performed the work.
That’s a really interesting property because for the first time there’s a way to verify what hardware is doing over the network via cryptography.
Or, if I can extract that key from the hardware, I can pretend to be that hardware whenever I want, right?
Hmmm… I was going to say no because it’s asymmetric crypto, but you’re right if you are somehow able to extract the signed private key, you can still lie… Good point
Got some bad news. They already can do that. It’s a very low effort attack too. Current TPM spits its key out in clear text. Funny right?
You don’t necessarily need to detect the cheat itself, you can look at things like players having suddenly higher kill rates and put them into a queue for observation by either more advanced (more expensive) automation to look for cheating or eventually involve a human in the loop.
Even on consoles after a while it becomes obvious that you cannot control the hardware, let alone the software on the client side. Those are the very best argument for this kind of approach and they get cracked eventually.
That’s true, if the player suddenly has higher kill rates. However, that doesn’t work if they’ve been using the cheat from the start on that account. A sufficiently advanced AI powered aim bot would also be nearly indistinguishable from a professional player. Kind of similar to how Google created the CAPTCHA that uses mouse movement … but had to go back to (at least in some cases) the additional old school captcha.
I’m a programmer, yes it is. It’s not easy in the sense of easy to implement, it’s easy in the sense that everything else is impossible. Client-side anti-cheat is impossible, and by that I don’t mean hard, I mean perpetual-motion level of impossibility. If someone tells you they implemented a foolproof client-side anti-cheat you should be just as skeptical as if someone tells you they created a perpetual motion. It’s impossible, never going to happen, want an example? Robot using a camera to watch the screen and directly moving the mouse and keyboard, completely undetectable from the client side.
From the server perspective the person is cheating or is behaving like a human. If they’re behaving like a human their behavior is completely indistinguishable from a human, so who cares if they’re cheating?, whatever they’re doing has them still at human level so if the game has skill based matchmaking (which most of these games do) he’ll rise up until his cheating puts him in the same level of more skilled humans and everyone has fun. If he keeps rising forever he’s not on a human level, therefore a cheater. More importantly this also penalizes people who buy bot leveled accounts, because their matches will be all against people they can’t hope to win and the game will not be fun.
Server side can also trick clients into giving up that they’re cheating, e.g. sending ghosts behind walls to check for wall hacks or other similar things to gauge player responses.
But what do I know? I’m just a senior programmer who’s been working on servers for some years. l never worked on the client side anti-cheat though, also never tried to build a perpetual motion machine.
It’s not easy, but it’s really not worth the massive gaping security vulnerability you are giving your users. One disgruntled employee giving out the keys to the castle or one programmer plugging in an infected USB, and every user now has a persistent malicious rootkit. The only way to fix an issue that deep after it gets exploited is to literally throw away your hard drive.
This can’t be right.
Don’t throw your hard drive in the trash. Quarantine the infected computer, and then wipe that hoe and slap your choice of OS back on it and scan/monitor to see if any issues arise.
Edit: since folks may or may not read though the rest of the conversation: I am wrong, throw that SSD/HDD in the garbage like barbarian said.
I’m sorry to disappoint, but with rootkits, that is very real. With that level of permissions, it can rewrite HDD/SSD drivers to install malware on boot.
There’s even malware that can rewrite BIOS/UEFI, in which case the whole motherboard has to go in the bin. That’s much less likely due to the complexity though, but it does exist.
not all rootkits are made to do that. So yes in some cases, throw it in the trash. In others, remediate your machine and move on.
Outside of monitoring individual packets outside of your computer (as in, man in the middle yourself with a spare computer and hoping the malware phones home right when you’re looking) there’s no way of knowing.
Once ring 0 is compromised, nothing your computer says can be trusted. A compromised OS can lie to anti-malware scanners, hide things from the installed software list and process manager, and just generally not show you what it doesnt want to show you. “Just remediate” does not work with rootkits.
Dude… That’s fucked. They should really go a little more in depth on rootkits in the CompTIA A+ study material. I mean, I get that it’s supposed to be a foundational over view of most IT concepts, but it would have helped me not look dumb.
Please don’t walk away from this feeling dumb. Most IT professionals aren’t aware of the scale of the issue outside of sysadmin and cybersecurity. I’ve met programmers who shrug at the most egregious vulnerabilities, and vendors who want us to put dangerous stuff on our servers. Security just isn’t taken as seriously as it should be.
Unrelated, but I wish you the best of luck with your studies!
Could they harden their clients somehow or maybe randomize memory locations for things? Seems like their should be a better solution than installing malware to prevent cheating.
You’re asking good questions but factor this in: a development team at a game company will only want to spend as little time as possible on this process: it doesn’t make them more money - it costs it. Conversely a hacker / cheater is being paid (or gaining) directly from breaking this code. Which is more motivated? Now remember that the protection has to be in place first. Who has the advantage? Client side code will always be breakable. A rootkit doesn’t change the game - it just adds a new vector to attack for other hackers to exploit.
I’ve long believed that the main point of client-side anti-cheat is to serve as security theater.
If the player sees “PROTECTED BY ACME ANTI-CHEAT” on the boot screen of a game, they’re less likely to cry wolf when they get their ass kicked. At least, until they see a blatant example of hacking and lose all faith in the ability of the platform to protect them from it; from that point on, everyone better than them must be cheating from their perspective (speaking from firsthand experience here).
Given how infamously toxic and high-strung the LoL community is, I can only imagine that Riot’s basically at the end of their rope here. If you read the original forum post, they sure make this sound like a Hail Mary. “Sorry, it’s just what we have to do to make sure the game is fair.”
Hilariously, they even undercut their own points in the FAQ:
Also, they’re apparently not bothering enabling Vanguard on OS X because apparently few people have actually developed cheats on it yet. Really tells you what’s the more developer friendly platform, Linux or OS X, doesn’t it? Or maybe the OS X market share is too small to care.
They do also mention using machine learning to detect cheating server-side but lament that it’s not always enough information, and that cheat developers have added “humanization” elements that play more like humans.
My thought is… if a cheat doesn’t make someone obviously better than a human player of a certain skill level, then what does it really matter? Congratulations, you made a bot that’s indistinguishable from a human, thanks for padding our player numbers.
The real problem is that botters don’t pay for microtransactions. And players who buy bot-leveled accounts probably don’t spend a ton either. Why would they? They got everything unlocked for them, they didn’t have to grind for it. That’s all Riot really gives a shit about.
In practice, client side anti cheat is essentially DOA because hardware cheats that analyse the player’s screen on a 2nd computer and proxy inputs to your mouse USB have made it so cheat clients are never actually executing code on the host machine.
At that point, even players cant tell someone is cheating because the cheats aren’t modifying the game state in a noticeable way- they’re still weak to effects that obscure your vision and have inputs that are difficult to differentiate from a “real” player.
IMO cheating is a social problem and one that is totally impossible to beat with rootkits by design.
It’s not easy. And league is free. So banning people won’t work well either. They can’t ban ip addresses either without banning college campuses, some apartment buildings, and Internet cafes.
There are solutions to this problem but they don’t want to permanently ban them. A ban = a new registration… maybe even two. Bonus! You get to pad your ban numbers and user registration numbers at the same time!
But that wastes their clockcycles to make sure you’re not cheating. So much easier to make everyone’s experience worse so they don’t have to upgrade and build out more servers.
This. Server side anticheat is the only correct detection method. And it’s only part of the solution. Pure automation is pure garbage.