Fixing League’s Instant Feedback System

Don’t worry guys, I’ll solve your problem for you, FOR FREE!

(No introduction, the previous post about what’s wrong with it in the first place should be used as one) 

0. Get rid of the contractors

When I started writing my last piece, I actually didn’t know Riot’s behavior support staff were all contractors. It was both a bit of a shock, and the answer to many questions that arose during my research. It explains why the platitudes differ between agents, and even present conflicts. For example, if you report toxic behavior, some agents will tell you that they’ll “take care of it”, but can’t tell you what they did, and others will outright admit they can’t do anything.

This man has absolutely no business interacting with your customers. And yet, he continues to do so, every day!

Riot’s brand is built on top of the relationship it has built with its playerbase. Outsourcing your interactions with players to people who you can’t seem to keep up to date with policy is not how you foster that relationship. The rest of my solution relies on human support agents, and they need to make the right calls, and they need to actually give a shit. Heed my advice: Phase out the contractors.

If you need a reason, please remember, one of your contractors left a nasty comment on my blog, from VMC Consulting’s address space. Your contractors, tasked with telling players to stop being toxic, can’t even do it themselves.

1. Implement Enhanced Human Review

Again, don’t worry, I did literally all of the work for you. The spec’s all here, if you just read.

Enhanced Human Review (EHR) is, in short, a Riot employee using specialized software (that’s NOT difficult to create OR use) to perform a rapid (less than 5 minutes for a skilled operator) but thorough check of all chat in a game where 1) somebody was issued a punishment 2) other players scored high on the IFS*. If fully implemented, Enhanced Human Review solves the following problems, while allowing you to keep Lyte’s IFS:

  1. Right now, there’s a line that must not be crossed, lest your account starts to get restrictions. But, you can cross that line, game after game and have no idea you crossed the line, until it’s too late! By issuing warnings to players who got heated, but didn’t necessarily go “too far”, there’s better communication with players who can be reformed, before they need it, and before they lose faith in the system!
  2. Because a human is about to manually confirm each person who wasn’t issued a punishment, the system can be more visible to 100% angelic players, and encourage them to both keep up the good behavior, and report negative behavior; This system would enable a rep to randomly issue a “reform card” (or just a toast notification, similar to LeaverBuster per-game notifications or the eSports notifications) thanking them for keeping it sportsmanlike.
  3. If there is already a program in-house to randomly verify IFS output, this program can perform that same function, and more at the same time, meaning you get more value out of any given unit of a support rep’s time. If implemented properly, this won’t even cost any more money in labor. In fact, you’ll save it, remember, by getting rid of VMC Consulting, because you’ll all but eliminate an entire class of tickets!
  4. The single biggest problem with the IFS is that it waits too long before issuing a punishment for minor toxicity, and on a per-game basis, unfairly punishes. To put this in perspective, somebody who’s “close to the edge” can receive a punishment for calling someone trash, but somebody in the same game with a slightly-newer account can outright threaten players with violence, and not receive a punishment. This is wrong, it absolutely DESTROYS faith in the system, and my system fixes this problem specifically.
  5. Because this system, by it’s nature, “forces” good players to be reminded that the system exists, they’re more likely to report negative behavior. More players reporting negative behavior means better data for the IFS to work with!

Enhanced Human Review works on the following theory:

  1. A game wherein a person was punished for negativity is downright likely to contain negativity on behalf of other players.
  2. Riot Games does NOT accept retaliation/”he started it” as an excuse for negative behavior
  3. The vast majority of players will reform after a nudge in the right direction (Riot believes this)
  4. A faster punishment reduces recidivism (Riot also believes this)
  5. Reminding players that the system exists (and works) keeps players on the right track, and keeps them reporting

Step one, take ALL games where a punishment was issued. In other words, if a player was punished for behavior that happened in a game (which may be up to three per punishment), it’s now a candidate for Enhanced Human Review. But that’s a LOT of games, we’ll need to find ones that are the most worthy of a rep’s time. So how do you find games with lots of problem players? You already have a system that can do this. Use it…

Step two, drill down the candidate games using the IFS scoring system. Ensuring it doesn’t use them to train itself, or edit a player’s account standing in ANY WAY, use the IFS to scan/score all ten players. You could either use a static score threshold or a bell curve, the punished player setting the curve. For a simple example, if there’s a game where one player was punished, but another scored higher on the IFS, or a few players were “almost as toxic”, that’s definitely a game we want a human to look at. Because a human’s about to oversee the process, we can make the machine a bit more sensitive for this purpose.

Step three, a human will be handed the game. The UX for the software an agent would use is what I’ve been working the hardest on, because this is where the system will be made or broken.

  1. The human agent will be given three chat logs; each team’s chat, and /ALL chat. These will be displayed side by side, synchronized in time; Gaps are inserted in each to make lines sent at similar times appear next to each other. By default, all three logs will scroll together, but the scroll bars can be unlocked.
  2. The game replay will load in the background, and the agent will be able to right click any line of chat, and be sent to the game, camera locked on that player, thirty seconds before the line was sent.
  3. All lines of chat will have a checkbox. If the checkbox for any given line of chat is ticked, it will be boldfaced in the reform card. This is one of the massive benefits to human review.
  4. THE AGENT WILL NOT KNOW WHICH PLAYER WAS PUNISHED. This outright forces Enhanced Human Review to be applicable as a double-check on the IFS, allowing those human resources to be moved over to this system without compromising… anything!

Step four, a human will be asked to select one of the following four options for all ten players. The agent will not be permitted to move on without selecting one of the following options for each of all ten players. If the agent neglects to select an option for more than zero of the players on the map, they will not be permitted to continue.

  1. Innocent – The player broke no rules
  2. Warning – The player got heated, but not to a level that would warrant a chat restriction
  3. Minor Infraction – The player will be issued a chat restriction following the below formula
  4. Major Infraction – The player will be issued a 7-day, 14-day, or permanent ban, in this order

A warning? But we’ve never done those before!

A warning is the single biggest thing you can do to increase visibility of the system, allows for reform without making a player feel targeted or personally slighted, and provides a gentle reminder to reformed players of where they need to be.

A warning looks just like a reform card. It lets the player know that, in a recent game they played, another player was punished, and it was flagged for Enhanced Human Review. As part of that review, their in-game comms were found to be a little heated, and it would be greatly appreciated if they instead chose to ignore and report problem players. If they don’t, they could be subject to actual punishments. LeaverBuster has a warning in place, and Instant Feedback needs one, too.

The following points have to stay, they aren’t optional! They’re important to the psychological component of this system!

  1. Another player was punished for negative behavior (You don’t have to say who)
  2. A Riot staffer reviewed EVERYONE’S chat, manually, with their human eyeballs
  3. This is a normal thing to happen and they can expect it to happen in any game where someone blows up

This reminds players that…

  1. Players do get punished
  2. It’s not 100% automated and potentially exploitable
  3. Ducking under the radar will only work for so long

Chat restriction formula:

Since this system will multiply the amount of punishments issued in aggregate, we can be way more flexible with chat restrictions, because they will be issued more liberally. Start with 10, add 5 for each subsequent restriction, up to a maximum of 30.

What do we do with innocent players?

This is where one can get really creative. At a minimum, you could roll for a ~20% chance to display a small notification (No more intrusive than “you have 4 games left in your LeaverBuster”) thanking them for keeping their cool. This could scale all the way to rewards; single-game rewards could include loot system bonuses, such as essence or keys. Larger, more visible rewards such as summoner icons or loading screen boarders can be given to players who are consistently judged innocent in Enhanced Human Review cases.

Additionally, if a player was judged to be innocent in an Enhanced Human Review, any reports against that player, for that game, can be safely marked as false reports. This also represents the random double-checking of the Instant Feedback System; if a player was issued a punishment by IFS, but was later judged to be innocent during Enhanced Human Review, this is a fantastic opportunity for Riot to proactively reach out and apologize to the inconvenienced player.


Right now, the Instant Feedback System focuses too heavily on negative chat, and not the things that create negative chat. And confusing as it may sound, this includes negative chat. The IFS relies on punishing things that are easy for a spam filter to pick up on, and things that won’t cause false positives. That means that while zero-tolerance behavior can be handled in as few as one game, it can take months for a passive aggressive troll to start seeing punishments. The “minor” negativity leads to the zero-tolerance behavior, and both ruin games.

With the heavy focus on the low-hanging-fruit, and the exclusive focus on chat, the Instant Feedback System also leaves itself vulnerable for abuse. There are no automated systems that detect trolling, only negative chat, intentional feeding, and AFK/leaving. While a human can look at game replays and report comments, this only happens for the most egregious cases, to people who have been reported, by many players, in every game, for months.

Players who troll the game by banning hovered champions, taking the wrong lane, or other “non-sportsmanlike” behavior cause the negative chat that the IFS targets. Because of this discrepancy, it’s entirely possible for somebody to troll games with the sole and exclusive purpose of goading others into saying things that would get them banned. As long as this sort of troll doesn’t use chat, he would be able to duck under the radar indefinitely, and would never be caught by a hotrodded email spam filter. But he’d be immediately caught by an Enhanced Human Review.

One comment on “Fixing League’s Instant Feedback System

Leave a Reply

Your email address will not be published. Required fields are marked *