On “cheating” League’s Instant Feedback System

tl;dr I got sick of being punished for being toxic in League but also seeing everyone be toxic towards me with no repercussions. So I figured out how to cheat Lyte’s cute little filter system

Riot’s solution to toxicity

Update: This post is about the problem. I also posted my solution.

22 October 2017: Looks like some dude took this information and went. fucking. nuts. Some dudebro on the League forums claims he systemically found zero-tolerance patterns/rules for the IFS, and further found a way to force/coerce people into tripping them. As if that wasn’t funny enough, then he supposedly started some racket fooling kids into thinking they’d get paid by VMC to funnel them traffic. While I’m almost positive it’s a troll post, he does link here, and I wanted to take the opportunity to say…

I wrote this before ever knowing about Tantram or reading any of his posts. Information he’s been so kind to give out both confirms some things here, and dispells some other things here. I have not yet had an opportunity to go through and make these changes, because I also wanted to go through and cite my sources at the same time, and I’m very busy as of lately. 

For what it’s worth, here’s a post where Tantram admits people are doing this now and he’s gotta make some changes to compensate. Love it.

Tired of being the bigger person? I know the feeling. Every game it’s someone different (but usually yas) being an absolute cock to you, but if you dare say anything back, say goodbye to your account.

Riot seems to be very proud of their Instant Feedback System. The Instant Feedback System is the formal name of a collection of server-side processes in League of Legends that handles player reports and automatically issues punishments in cases ranging from toxic/negative in-game chat, feeding/”inting”, and AFK/”Leaver” behaviors. In the latter case, there is a line between IFS and LeaverBuster, but it’s pretty blurry.

Riot’s really secretive about this system, and when you take it apart, it’s easy to see why. The biggest thing Riot hides, because of the obvious implications, is that the Instant Feedback System is **FULLY AUTOMATED**. While that may seem obvious, I do mean fully. There’s literally zero human oversight during normal operation. It was specifically designed to remove humans from the equation as much as possible; whether that’s for reasons of avoiding bias or cost saving is up to the imagination. My cynical ass is leaning towards the latter. 

MY solution to toxicity!

Riot doesn’t say much about the system, but there are some major points that are frequently brought up, especially in press releases for it in various regions. In no particular order, they are: [Sources coming soon]

  • There is a “learning” component to the system. Whether this is done in stages/releases with bulk amounts of data, or done in real-time with every report is ambiguous, but Riot seems to, at a minimum, want to leave the illusion that the latter is the case. This ability to learn can be abused. Just ask Tay.
  • The system focuses on language and phrases. This is the dead giveaway as to what the system is actually doing.
  • Stats from the game, including build, KDA, etc. are factored into each “investigation”. Oddly enough, these totally-unrelated-to-attitude factors may be used in calculations for purely chat-related investigations and punishments
  • Riot, especially through interactions with support staff, stresses very heavily that it will take a lot of reports over a long period of time before anybody sees any punishment. They of course do not use this language, but this is the effect. The rationale is easy to guess; if you’re working with a 100% automated system, you want it to be 100% sure someone’s an asshole, and eliminate false positives.
  • The number of reports against somebody for a single game is not a factor in the decision made by the IFS; This includes decisions to issue a punishment and decisions to investigate in the first place. Remember, since the system is automated, there’s no reason not to keep track of every time anybody gets offended.
  • Reports are handled “individually”, which makes a lot of sense once you understand the system. This doesn’t seem like much, but it becomes important later.
  • If you report a player, and that game results in a punishment, you will get a pop-up notice within 15 minutes
  • The only reason Riot does anything or makes any change to this system is for the sake of “reform”

So what’s the big problem?

Riot’s really big on “reform”. There are very popular ideas for “fixing” toxicity in League (such as “prisoner’s island”, pooling toxic players together), and they’re always shot down with a huge post from Lyte Yagami about how that wouldn’t foster the correct environment for “reform”.

And the big problem is that this system isn’t really good for reform. And funnily enough, the best argument I’ve seen for this yet is the top comment on a huge post from Lyte about “reform”

That guy wrote a tome, but the overall message goes a little something like this: You get used to toxicity, you become toxic over time, and due to the IFS looking to throw out false positives, you’re only punished after months and months of progressively assimilating into your environment. If you’re one of the (as Riot loves to brag about) vast majority of players who “reform”, you clean up your act. But then, you notice that everyone around you is still toxic. Maybe you report them, but again, because of the IFS not wanting to falsely ban someone, your reports appear to mean nothing. Everyone’s just “allowed” to be toxic. And because you’ve already gotten a punishment, you’re on “thin ice”, and if anybody reports you, for any reason, you’re incredibly likely to get punished. Even if you did something totally benign, like say #noganks.

To summarize again, you get punished, then people around you continue to be toxic, and the whole system unravels before you. You’re held to nothing less than the highest standard of player behavior, but yas mains the world over still tell you to kill yourself, and despite Riot’s “zero tolerance” (lol) policy, they are never banned. Not for months to come.

It’s very hard to reform when this double standard is applied. If it doesn’t look like a double standard on it’s face, that’s okay; Remember, I still haven’t explained how the system works.

Who is “player behavior support”?

Sending in a support ticket for behaviors Riot supposedly has “zero tolerance” (lol) policies for? Good luck! Riot player behavior specialists are NOT employees of Riot Games. This role is contracted to a firm known as VMC Consulting. Just ask their former agent, Adam. And by the way, VMC knows this page exists. They’ve even been sharing it in the company Slack.

These reps likely don’t have the power to place or remove bans, which would explain why unjustified bans aren’t removed, and are blamed on literally any negative comment they can find, even if it means digging through chat logs for previous games, games having nothing to do with a punishment.

If you look at the job ad for this position, you can even see that candidates aren’t even required to have familiarity with League of Legends. It’s only a “huge plus”. That’s right, it’s entirely possible that the support rep helping give “insight” into your suspension, and has to understand your appeal, may not even know how the game works!

Additionally, it’s not hard to see why VMC’s employees aren’t very good at their jobs; they can’t even practice what they preach! The first comment on this article was made from the IP address 204.16.186.4. According to WHOIS, that netblock (a /21 to be precise) belongs to “OrgName: VMC Consulting”, who apparently employs people who aren’t even smart enough to do that kind of thing from their mobile data connection instead of VMC’s connection.

Player behavior support being contracted to an outside firm certainly explains why Riot, just this week, had to fess up to the right hand not being able to talk to the left and promise to keep everyone informed as to what the rules are. It would also explain the “smite support Nunu” fiasco; as you’ll learn later on (and is made clear by Adam’s job description), these contractors only exist to justify the output of the Instant Feedback System. If the computer says someone should be banned, it’s up to them to find a reason should the player ask.

By the way, this means that when they give you meaningless platitudes like “we will investigate your report and will take any actions we feel necessary”, they are lying to you. You can’t handle the 100+ interactions Adam mentions every day and actually do an “investigation” on each. They’ve even been tripped up by the clipboard when sending these form letters, occasionally not even bothering to swap out the last guy’s summoner name for my own.

So how does it work?

You want a tl;dr, don’t you? Okay…

The Instant Feedback System is a modified email SPAM filter.

Cypherpunk Ekko: Spam Assassin

The League of Legends reporting system is, at it’s most fundamental level, absolutely no different than the program your email provider uses to filter SPAM. Lyte would like you to think there’s a complicated set of mechanics at work, but the only things IFS does differently than a run-of-the-mill SPAM filter is that it only “scores” chat logs when specifically invoked by a player report, and that it only delivers a punishment after many, many reports.

Now I’ve used the word “score” a lot, and non-technical or younger folks might not know what I’m getting at. So I’ll quickly explain what a SPAM filter like SpamAssassin does; It looks over an email for evidence, for “signs” that it might be junk. These include words and phrases like “United States Dollars” or any mention of the FBI or some sort of state-sponsored lottery. It also includes things like an ALL-CAPS email or subject line. Finally, more technical details, such as an email originating from a part of the Internet commonly known for sending bullshit, is also factored in. All told, it checks each message against hundreds and hundreds of these clues, called “rules”.

But just one of these things, that of course doesn’t make a SPAM message. Just mentioning the FBI doesn’t mean something is junk. Just having an ALL-CAPS email doesn’t mean it’s junk. So how does SpamAssassin put it all together?

Every individual “sign” that something might be junk is assigned a “score”. Let’s say a mention of the FBI is assigned 3 points, a mention of a large sum of money is 5 points, and and ALL-CAPS subject is another 3 points. To mark something as junk, the total “score”, the sum total of the values of all the “signs” SpamAssassin found, must be higher than the pre-determined threshold. For our example, let’s say it’s 10. 10 or more points, and the message is marked as junk.

So an email that just mentions the FBI will get 3 points. That’s under 10, so it’s not likely to be junk. What about an email that mentions the FBI and has an ALL-CAPS subject? 8 points. But all three? That’s 11 points. Each one of these things, innocuous on their own, combine to make it really obvious, even to a computer, that this is total junk.

What’s all that got to do with League?

The Instant Feedback System, when asked to review a player’s chat log, “scores” that player’s chat log using a similar process. Instead of looking for scam emails, though, it’s looking for toxic behavior. It’s scouring the chat log for words and phrases that are commonly associated with toxicity. For each one it finds, it adds to that game’s “score”. Different toxic behaviors have different scores. For example, it would stand to reason that death threats would score a bit higher than calling someone trash.

While it’s important to remember that I have never seen the IFS source code and I have never directly observed its operation, if I had to design such a system today, these are some example words and phrases I’d tell it to look for…

  • garbage/trash
  • “fuck you”
  • uninstall
  • kys/”kill yourself”
  • “this $champname” on a line by itself
  • omg/omfg
  • wtf/”what the fuck”
  • “you fucking”
  • “is feeding”
  • boosted
  • ks/killsteal

It’s also not out of the question that more “meta” (NOT THAT META) things are considered and scored, such as frequently sending identical/similar lines, SENDING LINES IN ALL CAPS (especially when combined with an ‘omfg’ rule), using excessive punctuation (!!!?!???!!?), or mentioning a champion many times (which would look just like harassment).

And again, any one of these things, by themselves, doesn’t mean that a given chat log is an example of toxicity. That’s why a system like this would require more than one of them. Additionally, the IFS only scores chat logs that were the subject of player reports. That means context isn’t really that big of a problem for the IFS; if I jokingly called a good friend a boosted animal, they won’t report me, so the IFS will never get the opportunity to call it toxic.

Finally, it’s well known that the exact “list” of words/phrases/patterns being searched for is ever-changing; The IFS is continually updated using a machine learning algorithm that identifies patterns in “any” new harassing behavior that may arise in the future, provided large numbers of people report it. Again, learning capabilities like this can be abused.

Since in-game communications are primarily for relaying strategies, and not for socializing, it would also stand to reason that the IFS believes a longer chat log is more likely to be toxic. This effect would also essentially “stack”; A longer chat log that’s truly toxic will have far more examples of toxicity, and as such will have a much higher score. The message here is, even if more chat isn’t a red flag by itself, if you’re trying to fly under the radar, less is definitely more. Every line you send is not only another opportunity for the IFS to chalk you down for a few extra points, but is also another opportunity for a support agent to justify its output.

So now that we’ve gotten this basic understanding out of the way, let’s lay down some quick points

  • In SpamAssassin, there are rules that have a score higher than the default threshold. That means the rule, by itself, is enough to flag a message. This could just as easily be the case in the Instant Feedback System
  • In SpamAssassin, there are also rules that lower a score. These would include things like being from a known-good sender of messages (Like Amazon or Apple) or being from an address you’ve previously sent mail to.
  • Remember how I said earlier that only your chat logs are seen by the system? This is because a filter of this type isn’t equipped to take apart human conversation and figure out who’s talking to who, about what. All it knows is what you said, and how toxic each line was.
  • It’s very likely, but not certain, that most “neutral” profane words (fuck, shit, damn) are not considered inherently toxic, for obvious reasons
  • That said, profane words that are typically used in insults (bitch, cunt, homophobic/racial slurs) are probably scored by the IFS
  • This one’s super important… A SYSTEM LIKE THIS IS WHOLLY UNSUITED TO PICKING UP DRY HUMOR AND SARCASM.

So what does it do with the score?

You’ve gotten a report, the Instant Feedback System launched an “investigation”, scored your chat log, and determined that it was hella toxic, confirming the report. So what does it do now?

This is where the sauce is still a little secret. I can really only give you the high-level description, which is what Riot is already providing: The confirmed report (and, likely, its score) is stored, associated with your account, and is used during later “investigations” to determine if there is a long pattern of toxic behavior before issuing a ban.

Can we hypothesize what a “long pattern” looks like? The exact solution probably falls into one of the following categories, or a combination of them:

  1. x number of confirmed reports over y games
  2. x number of confirmed reports over y days
  3. x percentage of games over y period resulted in a confirmed report
  4. x games in a row resulted in a confirmed report
  5. Average confirmed report score with a minimum number of samples
  6. Total (sum of all confirmed reports, ever) score above a certain value
  7. Total (sum of past x games/days) score above a certain value

I think you get the picture. Even though that seems like a huge (and important!) source of ambiguity, the takeaway here is that in most cases, you can be as toxic as you’d like, as long as you aren’t toxic very often, and don’t take it too far by, for example, issuing death threats. Reducing false positives to an amazingly small amount (0.1%) was literally the number one design goal of the Instant Feedback System, and that’s not obtainable on a fully automated system without also having a ridiculous false negative rate.

Another nuance to keep in mind is that getting a punishment doesn’t change this calculation in any way; If you have four toxic games in a row, and suddenly get a 10-game chat restriction, it won’t take four games to jump to 30 or even a two-week ban; the next report you get, warranted or not, WILL net you the next punishment if your chat log scores high enough. Period.

It is almost certain that confirmed report data “decays”, and on the IFS FAQ, Riot explicitly says that “Players who show consistent reform or show positive behavior in their games will help heal their accounts reputation.” It is not mentioned how this is implemented; Some sources mention punishment levels decaying over a period of months.

Then what are the written comments for?

Human review.

“But you said humans never review reports!”

Humans never approve/deny punishments in this system. Except in very rare cases. If a person, for example, is receiving an extremely high number of reports, but none of them score high enough to pan out, the system would forward that to a human for manual review.

Humans routinely review random samples of punishments to make sure they were justified. If you find yourself asking a support agent to justify a punishment, the first place they’re gonna look is any written comments attached to the confirmed reports.

Un-official facts and observations

The fuck is an un-official fact? I’m using the term here to describe anything that is well-known (and frequently cited on the forums) about the Instant Feedback System, but isn’t in major publications about it, or has to be pried out of Riot staffers through carefully targeted questioning. I’m also taking this opportunity to share some of my own observations about the system…

  • The Instant Feedback System works, and issues punishments up to and including permanent bans, 24 hours a day, 365 days a year. My main account got its permaban (skipping the levels of 30 game chat restriction and 2-week ban) 10 minutes after a game that happened at 3AM on Christmas Eve. If you genuinely believe a human had anything to do with this decision, I just can’t help you.
  • The Riot staffer’s job is to justify the output of the IFS algorithm. That’s all. That’s their job description. That’s all a “behavior support specialist” does to earn their paycheck. Additionally, different staffers will look at the same chat log and give you vastly different justifications. This implies that no staffer can see the algorithm’s direct output and scoring. If they could see what scores contributed to the punishment, different reps would reliably cite the same reason: whatever reason the computer spoonfed them.
  • Social media accounts affiliated with Riot Games specifically recommend creating new accounts in response to a permanent suspension. Permanent suspensions are not meant to remove you from the game, they are meant to provide a “sting” by forcing you to start at summoner level 1, with no runes, masteries, or any of the good summoner spells. And let’s not forget, all the skins you bought on your old account? Gue$$ you’ll need to find $ome way to get tho$e $kins back!

Putting it all together

Let’s summarize before getting to the part you’ve been waiting for….

  • The Instant Feedback System is fully automated. A human being never observes, confirms, double-checks, or approves its output
  • When you contact a support specialist about a punishment, their job, first and foremost, is to cling to anything that justifies the IFS algorithm’s output
  • This bears repeating: Support staff aren’t interested in helping you “reform”. They exist only to protect the IFS. Remember, they aren’t even Riot employees!
  • The IFS, upon being triggered by a player report, scans YOUR (and only your) chat log for “clues” that indicate that it is, indeed, evidence of toxic behavior
  • If enough “clues” are found, the report is “accepted”, and the score becomes a permanent part of your account, used to issue a punishment later
  • After each “investigation”, if some larger account-based criteria are met (a lot of reports), a punishment is issued, and the those who reported you are informed that they won
  • It’s generally accepted that more reports doesn’t mean more likely to get punished. “1 is as good as 9” is a common mantra on the League forums. (I personally don’t believe it)

The Krabby Patty Secret Formula is…

One last little tidbit, because it didn’t fit anywhere else:

Only the in-game chat is logged and scored by the IFS. The pre- and post-game lobbies are a free-for-all. Shitposting in pre-game is your ultimate ability.

If you’re seriously still reading this you likely have one of two goals: 1) To just understand the Instant Feedback System enough so that you can help shield yourself against false reports 2) To be able to poke and prod players into getting themselves banned while keeping yourself safe and reducing the effectiveness of their seemingly-false reports. This section is for those in the latter group.

If you’re one of those people who needs to know the rationale for everything, here’s mine: I was just fine putting up with the toxicity, because I was just fine giving it right back. The situation became unacceptable the moment I stopped being able to do everything that everybody else was able to do. This system doesn’t work, it likely cannot be fixed, so let’s have some harmless fun!

The general idea is to be sarcastically positive about the game, so that when you do inevitably get reported, the IFS starts getting trained against word patterns that are associated with sarcasm. Because these are literally the same words used by actually-positive players, this will, if done by a large number of people, slowly make the IFS think good behavior is bad.

Know your goals

You basically have three goals. The first is to avoid being flagged by IFS. The second is to be able to convince a support agent that you really meant well. The third is to be indistinguishable from someone who is acting in a way that’s genuinely 100% benign and helpful. The biggest reason for this is when you’re indistinguishable from a “good” player, and you get reported, “good behavior” will slowly look worse and worse to IFS, increasing the number of false positives across the board. Seeing actual change in this manner will take a LOT of people poisoning the system this way. Remember, Riot’s #1 goal and priority with any player behavior program is to ensure innocent players aren’t unfairly affected. If false positives suddenly spike, this will essentially force Riot to take mitigating action.

Know when to quit

Even if you’ve mastered (or think you’ve mastered) shitposting in a “severably benign” way, as mentioned before, a long string of unsubstantiated reports, especially if they start coming from players known to be accurate, will get you popped off regardless of your chat log’s IFS score; you’ll either be booted by the number of known-accurate reporters or the sheer number of reports will flag your history for human review. Pick your battles. Shitposting in every game is not feasable. Take breaks.

Start in Pre-game

If your goal is to prod someone into losing their cool and tilting, while appearing innocent to the IFS, it’s important to make it perfectly clear in the score-free pre-game lobby. Remember, in-game you’ll be speaking in a way that could just barely be interpreted as negative. It’s going to be much easier to tilt them in the pre-game lobby and then set them off with a well placed “good job” or “lol” when they fail. The contrast alone between the absolute cock you were in the lobby and the goody-two-shoes you’re projecting in-game will add to the tilt factor.

Be severably benign

In short, this means speaking in a way that, if your chat log is viewed, by a human being, alone, without any context from the game (besides maybe your KDA and any written report comments) or anybody else’s chat, would look neutral at worst. The severable part comes in when remembering what a Riot support staff’s job is; Each individual line, by itself, should be engineered so that nothing about it could possibly be used to justify a punishment. The easiest way to leave this illusion is to pepper positive emoticons (winks, smileys, thumbs up) on your jabs.

Pick your mark

While you can be a general cockbag and tilt the entire team, if you’re trying to get someone else punished, it’s best to pick one person (who’s not doing well, was already tilted pre-game, or is playing yasuo), focus on them, and see if you can get others to pile on. Every other person attacking your mark will tilt him that much more, and with no additional risk of a successful report or punishment on your end. Additionally, while the consensus appears to be that “1 report is as good as 9”, it’s generally best to limit the amount of reports you are receiving, period. Picking just one person allows you to focus your efforts while minimizing risk.

Forget to ping

If you see your laner heading off to screw with your mark, keep farming, and “forget” to ping MIA. When you are called out for this, this is a perfect opportunity to admonish him for being toxic, in a friendly and “contructive” manner of course. To ensure the rest of your team sides against him, be sure to ping extra early and often for everyone else.

Just fucking killsteal

Realistically, you can’t get punished for killstealing. You have to make it really obvious, and it’s going to require a human review, anyway. In lower elos, this is going to be your bread and butter for tilting. Same as above; you’re less likely to run into an issue if it’s just one person you’re killstealing and just one person complaining about it. You may want to visibly hand off a kill to other teammates to ensure they view your mark as being in the wrong.

Avoid cliche sarcasm

IFS can’t detect sarcasm, but it can detect patterns of words typically associated with sarcasm. Examples include “I love it when I get ks’d,” and “wow nice gank”. Be original. Remember, your goal is to look indistinguishable from somebody who’s genuinely being kind and helpful, but for watching a full game and chat replay. A better approach to, for example, your mark taking a kill, would be something along the lines of “Thanks for letting me have that kill bro it’s really appreciated ;)”

Less, longer lines

A longer chat log naturally scores higher, and gives support agents excuses to justify punishments. At the same time, each individual line should be relatively verbose; if there’s any ambiguity as to what you mean (remember, IFS and a support rep will only care about your side of the chat, and won’t even view anybody else’s for context), clarify, preferably in the same line.

Conclusion

The League of Legends Instant Feedback System is inherently broken. Behavioral expectations, which are nebulously and invisibly defined by the mob at large, are applied unevenly based on factors sometimes as innocuous as account age. When punishments are issued, appeals and inquiries are handled by people who don’t directly work for Riot Games, and who aren’t even required to be familiar with the game when applying to work for Riot’s contractor. 

Especially given recent fiascos like a player receiving a 14-day ban because other players did not like his working-but-off-meta pick, or Riot being clear as mud regarding report solicitation (I was one of the people popped off for report solicitation, even though that’s, now, not bannable), the trust underlying the Instant Feedback System is eroding, day by day. Because trust is eroding in the system, people aren’t even bothering to file reports anymore. Because the IFS specifically requires a large number of reports before issuing a punishment, this leads to a vicious circle, where more people issue less (or no) reports, which allows a toxic player to continue longer, further reducing confidence in the system.

Riot is, and always has been, amazingly open with “how things work”, and their fascinating dev blog is, in my opinion, the best example of this. That’s why one should be suspicious of any systems they are secretive about. While Lyte did share quite a bit of information about how his Instant Feedback System works in a broad sense, he almost never touched upon how it recognizes toxicity on a per-game basis. The rationale is clear: If he did, asshats like me would work around it. But that only means the system is insecure. A secure system is one that remains secure even if everybody knows how it works.

Could I design something better? I really think I can, but it would certainly take a few more human resources than the current system uses. I’m considering a followup post about my proposed overhauls to the Instant Feedback System, if there’s any interest.

And for the record, I don’t like doing this. I’d much rather have a system that works so that I don’t have to play with toxic players. League of Legends is a game I play to relax and unwind after work. It shouldn’t be any more stressful than a game of baseball. I am, of course, willing to treat my teammates with nothing but respect, but I will not even pretend to show the slightest bit of decency to people who insist on treating me like shit.

The single biggest flaw of the IFS is that Riot went full robot. Never go full robot.

3 comments on “On “cheating” League’s Instant Feedback System

    • How much does VMC pay?

      Seriously bro, if you’re a Riot contractor who wants to leave me nasty comments, at least have the basic common sense to not do it from the company network

      If VMC or Riot Games feels I’m being… negative, they can feel free to report me to my host, OVH. Abuse form is right here: https://www.ovh.com/us/abuse/ Remember: Every report counts!

      By the way, OVH’s report system uses people, not machines.

  1. Thank you so much for this article. I read it a while back and soaked up all the information. Working in IT it just clicked when you said spam filter.

    In the past 30 days I have received over 100 of those nifty ban popups from reporting players. My record is 18 in a single day, and 4 from a single game – TWICE.

    Currently I average 1.4 players banned per game. This is my new hobby. It is like fishing, except the fish are always biting this bait.

Leave a Reply

Your email address will not be published. Required fields are marked *