How Does League Of Legends Deploy Machine Learning Models Into The Game?
How Does League Of Legends Deploy Machine Learning Models Into The Game?
Looking at how LOL deploys machine learning models with Ian Schweer - ML/MLOps Engineer @ Riot Games
I talk a lot about data infrastructure in this newsletter but I don’t often cover what people are actually doing with their data. So I was excited when I got the opportunity to work with Ian Schweer from Riot Games to discuss how their team deploys their machine learning models to League Of Legends.
Now let’s dig into how machine learning engineers deploy their models to League of Legends.
League of Legends and TFT(Teamfight Tactics) are worldwide games that serve millions of players. League of Legends is a 5v5 MOBA (multiplayer online battle arena) game where 10 players log into a game, each picking a champion for their particular role in the game (Top lane, mid lane, bot lane, support, and jungle) and attempting to take the enemy base through fights with the opponent teams, competing over neutral objectives, and leveling up their characters.
Both TFT and League of Legends leverage the League game engine which our team supports.
The data team is responsible for:
Collecting all the in and out of game telemetry to understand our players
Understanding the league player base, and what they like and dislike. Most importantly, data scientists work with game designers to help understand what the game is doing right for our players, and wrong.
Helping game developers build and ship data-driven features
In this post, I want to share with you some challenges in telemetry, player behavior, and shipping data in game.
The end of game file
One challenge often seen in enterprises is the need to join many datasets to get the picture of the “full customer journey”, or at least the customer journey of the happy path. League being a video game means we often want to know what a player did
in game to utilize in our analysis. Game events are a good proxy for player behavior (e.g. ITEM_BUY, CHAMPION_KILL) as they provide enough information to tell us something important happened, without cluttering the data.
However, there are millions of game clients, and thousands of game servers all under tight network and access restrictions. In addition, the game is very latency sensitive, and there is no baseline performance guarantee as many of our players play on heterogeneous hardware. This makes live game telemetry collection very difficult to accomplish, so we need to wait until the game has finished. This leads to one interesting solution: The game is server authoritative - so all game events come from the game server, and can be serialized at the end of the game (named the end of game file). This data maintains referential integrity and is generated by one process, meaning we don’t need to join anything to know about all the game events, we can just consume it. This is a core piece of data infrastructure that powers many features in League of Legends, though we’ll talk about two here.
Detect feeding
Feeding is when a teammate purposely plays poorly. They will purposely give kills to the opposing teams, intentionally fail map objectives, or other methods of helping the opposing team. It’s a terrible feeling when one person on the team is playing so badly, that there is little you can do to turn the game back in your team's favor. We wanted to be able to help players out of these situations by preventing feeders from playing more games.
We leverage Bayesian Inference for this (one of our Data Scientists even worked on mc-stan, a bayesian probabilistic programming language!). We first go through and model champion specific and agnostic representations of end of game feeding behaviors. Having both specific features per champion / scaled for champion, and general features allows us to have a generally good model, with some ability to tweak to any specific champion's needs. We have 160+ at the time of writing, so flexibility is really nice to have. The data is fetched through a custom library that parses the end of game files sitting in s3 with some extra filtering. This does not remove all data processing from training pipelines, but allows to some sharing between various others.
from oracle_lens.features.eog import
(
EVENTS_REQUIRED_SCHEMA_EOG_MAP,
INTERNAL_REQUIRED_SCHEMA_EOG_MAP,
SUMMARY_REQUIRED_SCHEMA_EOG_MAP,
Feature1,
Feature2,
GameEventCalculation,
get_difference_from_mean_rating,
get_participant_by_puuid,
)The data is then sent into one of several mc-stan applications that allow us to generate a posterior distribution (the goal of all bayesian methods!). We utilize this to create weights for a classifier. You can imagine this is similar to performing stochastic gradient descent, but instead our estimates include uncertainty (the great advantage of bayesian stats). In order to keep the features accurate, we leverage regular retraining efforts internally by watching replays of marked feeding games. This utilizes AWS SageMaker to distribute a replay and a survey. The developers of League all really care about making player games feel good, so the team is happy to keep this work up.
Once we have a model that is performing well on test data and validation, we can deploy the model. We leverage bentoML to give us a simple service with a `/predict` rest endpoint. We package the model and service in a particular way to ensure we can rollback
We package the model (with the model weights as a json) packaged and shipped to an internal PyPi instance.
In the Bento service, we then require the model package. This can then be put into a docker container
We then ship a Java Service that runs the bento service as a sidecar. The Java service allows us to be compatible with the standard League of Legends tech stack. If you’re interested in the gritty details of running services at riot, check out our article
This separation gives some nice benefits:
The data scientist can deploy any version of the model they want into Databricks to do more extensive testing, knowing that they cannot affect the model in production
We can rollback the model very quickly if we detect some regression in performance.
Now what about for in game models?
In game models
One of the most interesting challenges of being in a data / ML team on League is designing features that don’t take the fun out of the game. If we build something that tells people exactly what moves to make to win, then the game might not be as fun for players. An example of this is our recommendation systems. Item recommender is a system that helps players know which items to buy during their game. This could be simple recommendations at the start (buy boots!) or more specific recommendations curated by our game designers. In this case, we are utilizing similar techniques from popularity based recommendation systems.
One final design decision was allowing for game designer to override recommendations. Since we never want recommendations that would dramatically sway gameplay, we decided instead to give generally good recommendations, and allow game designers to add more specific recommendations. The override system comes for free by leveraging the existing tools developed for Riot game designers.
These recommendations run on the game client and game server. As described at the top, we need to be very careful. In order to prevent any mishaps, we pre-compute the recommendations and build them into the game. This is similar to unity’s concepts of prefabs. This largely breaks down into 4 steps, each described below in this table.
Because we hook into the build system, we need to write code on top of perforce( a version control system similar to git).
This can be slightly awkward as many git habits are ingrained in the engineering team, but there were some things we were able to do to reduce the complexity needed.
Separate the build jobs from the s3 parsers. We do this leveraging a decorate factory. This allows us to iterate on the transformation logic without needing to interact with the build system, and allows us to write custom drivers for integration testing.
.
Fail the pipeline early. We leverage asserts in our Databricks notebooks to kill jobs if we see any discrepancies. These include data quality checks. This prevents us from putting bad data into s3 altogether, reducing the risk of bad data going into the game.
Force partition date awareness in the build jobs. It is better for the build job to fail due to missing data (e.g. find data from the last 3 days, or fail). We are sitting on top of perforce, so we know the last known good recommendations are safe.
Using the end of game data generated for each game, we can easily ingest files to know which recommendations are used by our players and which ones aren’t.
We most recently shipped the Jungle Path Recommender using this exact system.
Conclusion
We covered a lot here! A lot more detail is left out and a lot of lessons. Keep your eyes on the Riot Games engineering blog for some of those lessons! In this post we covered:
How we handle in game telemetry
How we handle models in the service space with feeding detection
How we handle in game modeling and inferences
There are a lot of opportunities for anyone interested. LDC has lots of challenges for everyone across the data space.
Events
I don’t have any events planned for the next few weeks. But here are some you might want to check out
Data Practitioners Networking (in-person) - Interview Prep & DE Best practices.
Ethan’s Low-Key NYC Data Happy Hours
Jobs
Data Engineer - Oxfam
Senior Technology/Digital Transformation Consultant to Nonprofit Organizations & NGOs (Remote/Full-Time)
Manager Business Intelligence
Join My Data Engineering And Data Science Discord
Recently my Youtube channel went from 1.8k to 51k and my email newsletter has grown from 2k to well over 31k.
Hopefully we can see even more growth this year. But, until then, I have finally put together a discord server. Currently, this is mostly a soft opening.
I want to see what people end up using this server for. Based on how it is used will in turn play a role in what channels, categories and support are created in the future.
Articles Worth Reading
There are 20,000 new articles posted on Medium daily and that’s just Medium! I have spent a lot of time sifting through some of these articles as well as TechCrunch and companies tech blog and wanted to share some of my favorites!
Getting a job as an engineering executive.
I started my first executive job search when I was 25. Eventually, I got an offer to lead engineering at a startup with four engineers, which I turned down to join Uber. It wasn’t until a decade later that I joined Calm and started my first executive role. If you start researching executive career paths, you’ll find folks who nominally become engineering executives at 21 when they found a company, and others that are 30+ years into their career without taking an engineering executive role.
As these anecdotes suggest, there is no “one way” to get an engineering executive job. However, the more stories you hear about finding executive roles, the more they start to sound pretty similar. If you’re looking to find an executive role, I’ve condensed the many stories I’ve heard, along with my own experiences, into the repeatable process that prospective candidates typically follow.
The State Of Data Engineering
Over the past two months, I have been running a survey to get an understanding of what is going on in the data world.
During that time we had well over 400 respondents answer questions about what their data infrastructure looks like as well as ask about best practices and other logistical questions.
For this newsletter, we are only going to scratch the surface of this data because there are many different ways we can slice and dice the data we got.
For this part 1 of our initial breakdown we will be focusing on the following points.
Backgrounds On Who Filled Out The Survey
Vendor Selection( Data Analytics Platforms And Orchestration)
Best Practices
Frequently Dealt With Problems
End Of Day 68
Thanks for checking out our community. We put out 3-4 Newsletters a week discussing data, tech, and start-ups.