Kyle93
Registered User
- Mar 30, 2012
- 34,749
- 57
Can someone tl;dr this for FYC, thank you.
"Great post, ExcaliburZ. Allow me to share some of my personal thoughts on matchmaking...
We’ve been following all of the discussion around matchmaking. When topics get discussed in the community (and often among game developers) we tend to talk about things in very black/white or right/wrong terms. But most important decisions you make as a game developer are difficult trade-off decisions with no perfect answer.
The goal of the matchmaking is to make it so that you as a player do not have to find 11 other people to play with. You can click a Play button, and the system finds other players for you. That’s the basics. The reality is, the matchmaker is extremely complex in what it is trying to do. It does way more than I am going to mention in this post so while I am going to offer some information here, I am leaving some things out (not all intentionally – it’s just a really complex system).
At a most basic level, the matchmaker is trying to put you with 11 other people. But it doesn’t just randomly select 11 people. It takes into account a number of factors (more than I am going to list and not necessarily prioritized).
The first factor is time. The matchmaker will try to find you match quickly and not force you to wait too long. A very common thing that happens is that a player will become dissatisfied with a match and say “I don’t care how long you make me wait. I’d rather wait 20 minutes and have a good match than get matchmade into a match like you just put me into.” What we’ve seen is that when the time crosses a certain threshold, players begin to complain about it taking too long to find a match. It sounds good… waiting for that perfect match. But when the reality of waiting too long comes down on most people, they end up vocalizing their discontent on the forums. Also, there is an unrealistic expectation that if a player waits longer for a match, the “better” the match will be. The concept of “better” when it comes to matchmaking is a really hard one to define.
If I were to summarize match results into 5 broad buckets it would be these:
My team won. We beat the other team by a long shot.
My team barely won.
My team barely lost.
My team lost. We lost by a long shot. It wasn’t even close
It was a broken match somehow. Maybe someone disconnected, was screwing around or we played with fewer than 12 people.
(of course there are more cases than this – I am overly simplifying here)
Most players will say that they want a match to be either type 2 or type 3 as I described above. Those sound even. Barely win or barely lose. But I believe when psychology comes into play, most players actually expect type 1 or type 2 to be the result. Even an amazingly close type 3 match can turn into a highly negative experience for a lot of players. And if you keep “barely losing” it’s not a very fun night. Winning is fun and good. Losing is less fun than winning.
So waiting a really long time to lose by a long shot is obviously not good. But waiting a really long time to barely lose is also a negative experience. And if we assume that your chances of winning are 50%, that means that even waiting a really long time for a “better” match means that you’re going to wait a really long time to probably lose half the time… If your expectation was that you were going to wait a really long time for an awesome match where you either 1) Won by a long shot or 2) Barely won… but still won nonetheless, your expectations for what the system can or should do are in the wrong place. We do not generate bots to take losses so you can win more than 50% of the time. Those are real people losing on the other end of every loss you take.
A second factor we take into account is ping. We’re matchmaking people all over the world and we want to match people to the closest servers for the best play experience. In our second stress test, we had other things prioritized over ping-based matchmaking such as skill and time. For those of you who participated in that stress test, you’ll remember how terrible the game performance was on the first day as well as how “lit up” the forums were demanding that players be given an option of server choice. So now we prioritize ping for players. Some players live in challenging parts of the world when it comes to high speed data connections (I’m looking at you, Andes mountain range…) so it’s not perfect for everyone. But largely, most people get a really decent connection to our game servers. Matching players with wildly disparate pings also results in a higher frequency of undesirable side effects such as “getting shot behind walls”. Of course if you live in Houston, Texas and group with your buddy in Geneva, Switzerland, you’re now introducing uncertainty to our system that’s harder for us to deal with… but we allow it.
Which brings us to the next factor that we match on: grouping. The majority of our matches are comprised of either all solo players or solo players and players grouped with one other person. However, the system does try to match groups of equal sizes together first and foremost. As the time people wait grows, we expand the search to try to find others for them to play with. This means that occasionally we will match groups with players who are not grouped or in a group size that is smaller than their own. Like I mentioned, this is exceedingly rare but can happen. And that match is only made when players have crossed a waiting threshold that we deem too long. For most group matches a group of 6 is placed against another group of 6.
Groups are a big challenge in our matchmaking system. You can group with people of wildly varying skill and ping and we allow you to. It’s pretty unlikely that there is another group in the queue that exactly mirrors the unique circumstances that you have set up (pings, skills etc.). We want you to group. We feel that it’s the best way to play the game. So we try to avoid things that discourage grouping and we want to continually improve the social systems so that you’ll find it easier and easier to group with people you have chosen to play with. Playing with people you choose to play with is going to be more reliably fun than playing with people we choose for you. I once used the analogy of hanging out with people on a Saturday night. If you were to go out with five of your friends it would probably be a better time than if we tried to find 5 random people for you to go out with, no matter how smart we were in our selection process…
Anyway, this leads me to matchmaking rating. This rating is the most important thing that we try to match on. Basically this rating means “how good are you?” Commonly, you’ll hear this referred to as Matchmaking Rating or MMR. MMR is derived differently in different games. Overwatch borrows a lot of knowledge from other games but also does a lot of things unique to Overwatch. As each player plays games, their matchmaking rating goes up or down depending on if they win or lose. The system is extremely complicated and there is a lot more going on here than I am going to spell out. So please don’t take this as the comprehensive guide to how MMR is calculated in Overwatch. There is definitely a lot more going on under the hood.
In Overwatch, whether your MMR goes up or down is contingent on winning or losing. But there are a number of factors that determine how much that rating goes up or down. For example, what map you’re playing on and whether you were attacking or defending is factored in. We know the win rates on attack/defend on all of the maps and we normalize accordingly. Not all wins and losses are equal. We also look at your individual performance on each of the heroes you played during the match. Everyone has better and worse heroes and we have tons of data showing us what performance levels should be like on those heroes. We also look at your opponents and whether or not their matchmaking rating is higher or lower than yours. These are just a few of the things that are considered when determining how your skill should go up or down. At no point in MMR calculations do we look at your win/loss ratio and win/loss ratio is never used to determine who to match you with or against. We are not trying to drive your win/loss percentage toward a certain number (although the fact that so many people are at 50% win rates makes us extremely happy). All the system does when it comes to matching on skill is attempt to match you with people of a similar number.
The system is of course deeper than this. There are penalties and handicaps added for things like not playing for a while or playing in groups of varying sizes. We also do special things for brand new players to (hopefully) keep them away from the general population. Players will often mistakenly look at player level and accuse the matchmaker of making unfair matches. One thing that I have mentioned before is that we were evaluating your skill during closed beta, open beta and the second stress test weekend. If you played in any of these (over 10 million players did), we had already determined a skill rating for you (most likely). This means that it’s not uncommon to see a level 1 matchmade against much higher level players. In most of these cases, the Level 1 is a skilled player who played during the phases I mentioned but did not immediately play at launch.
There are many factors that are beyond our control that add noise to the matchmaking system.
• Leavers are extremely disruptive
• Players vary wildly in their skill with different heroes. We have no clue which of the 21 heroes you are going to play during a match
• Groups form with wild variance in skill levels and ping. Contrary to popular belief there is not a “perfect match” for your unique snowflake group
• Sometimes your little brother plays on your account
• Sometimes the cat walks in front of the screen
• Sometimes your wireless mouse runs out of batteries. (Why do you use a wireless mouse btw?)
• Sometimes a highly skilled player buys a new copy of OW to “start fresh” on a new account
• Sometimes you have internet problems
• Sometimes you play drunk or tired… or both
• That first game of the night…
• …that last game of the night
• “Life”
So this brings me to some thoughts I’ve been having about Overwatch. While this whole post has been mostly my personal thoughts – this next part is “especially” my thoughts and not reflective of the team or the company’s POV.
For better or for worse, we focused the design of the game on winning or losing as a team. OW is not a game where you ignore the map objectives and then look at your K/D ratio to determine how good you are. We want you to focus on winning or losing and as a result you do focus on winning or losing. We tried to make it so that losing isn’t the end of the world, but to a lot of people they expect to win far much more than they lose. I sometimes wonder if we were able to clone you 11 times and then put you in a match with and against yourself, would you be happy with the outcome? Even if you lost? Out of the 5 types of matches I described above, it is my belief that you would still experience types 1-4. Are those “stomps” still not acceptable? Because they will happen…
And I believe OW is strange game in that regard. I spend a lot of time studying the matches that I am in because I am very focused on matchmaking. I’ve been in so many Control Point maps where my team got destroyed on the first point, the enemy team got destroyed on the second point and then we play the third point to a 99%/99% overtime. If you judged any of those single points on their own merits you could say you have two stomps (one in your favor, one against you) and one close match. Same players...no change in matchmaking. Or take a match that I was just in on Route 66, for example. My team was on attack and could barely push out past the train cars. Two members of our team swapped heroes and we proceeded to march the payload all the way to the end of the map practically uncontested. The match went from a stomp in one direction to a stomp in the other direction.
So while it is possible for a mismatch to result in a stomp, not every stomp is a mismatch. If every time a team dominates another team it is viewed as “the matchmaker is broken”, the problem we have is with perception and expectations. Look across all pro sports. Even matches happen every night. Stomps happen every night. It’s a reality of any competitive game. Does that make being on the receiving end sting any less – probably not.
We are constantly improving the matchmaker. We learn more each day. We have one of our best engineers and best designers full time dedicated to the system. Many of those “silent” patches that go out during the week are adjustments to the system. For example, we recently realized that “Avoid this player” was wreaking havoc on matchmaking. One of the best Widowmaker players in the world complained to us about long queue times. We looked into it and found that hundreds of other players had avoided him (he’s a nice guy – they avoided him because they did not want to play against him, not because of misbehavior). The end result was that it took him an extremely long time to find a match. The worst part was, by the time he finally got a match, he had been waiting so long that the system had “opened up” to lower skill players. Now one of the best Widowmaker players was facing off against players at a lower skill level. As a result, we’ve disabled the Avoid system (the UI will go away in an upcoming patch). The system was designed with the best intent. But the results were pretty disastrous.
We will always be working on our matchmaking system. We’re listening to feedback, we’re playing the game a ton ourselves and we’re looking at hard data to inform our decisions. This post wasn’t my way of saying everything is fine. I just wanted to share some of my thoughts as someone who has been evaluating the system itself very closely as well as monitoring the feedback. I want to put it out there that there is a lot of room for improvement but also suggest that there are forces in play that cause some fair matches to sway lopsided due to forces out of our control. The game is as much (if not more) art than it is science. We’ll keep working to make it better!"
"Great post, ExcaliburZ. Allow me to share some of my personal thoughts on matchmaking...
We’ve been following all of the discussion around matchmaking. When topics get discussed in the community (and often among game developers) we tend to talk about things in very black/white or right/wrong terms. But most important decisions you make as a game developer are difficult trade-off decisions with no perfect answer.
The goal of the matchmaking is to make it so that you as a player do not have to find 11 other people to play with. You can click a Play button, and the system finds other players for you. That’s the basics. The reality is, the matchmaker is extremely complex in what it is trying to do. It does way more than I am going to mention in this post so while I am going to offer some information here, I am leaving some things out (not all intentionally – it’s just a really complex system).
At a most basic level, the matchmaker is trying to put you with 11 other people. But it doesn’t just randomly select 11 people. It takes into account a number of factors (more than I am going to list and not necessarily prioritized).
The first factor is time. The matchmaker will try to find you match quickly and not force you to wait too long. A very common thing that happens is that a player will become dissatisfied with a match and say “I don’t care how long you make me wait. I’d rather wait 20 minutes and have a good match than get matchmade into a match like you just put me into.” What we’ve seen is that when the time crosses a certain threshold, players begin to complain about it taking too long to find a match. It sounds good… waiting for that perfect match. But when the reality of waiting too long comes down on most people, they end up vocalizing their discontent on the forums. Also, there is an unrealistic expectation that if a player waits longer for a match, the “better” the match will be. The concept of “better” when it comes to matchmaking is a really hard one to define.
If I were to summarize match results into 5 broad buckets it would be these:
My team won. We beat the other team by a long shot.
My team barely won.
My team barely lost.
My team lost. We lost by a long shot. It wasn’t even close
It was a broken match somehow. Maybe someone disconnected, was screwing around or we played with fewer than 12 people.
(of course there are more cases than this – I am overly simplifying here)
Most players will say that they want a match to be either type 2 or type 3 as I described above. Those sound even. Barely win or barely lose. But I believe when psychology comes into play, most players actually expect type 1 or type 2 to be the result. Even an amazingly close type 3 match can turn into a highly negative experience for a lot of players. And if you keep “barely losing” it’s not a very fun night. Winning is fun and good. Losing is less fun than winning.
So waiting a really long time to lose by a long shot is obviously not good. But waiting a really long time to barely lose is also a negative experience. And if we assume that your chances of winning are 50%, that means that even waiting a really long time for a “better” match means that you’re going to wait a really long time to probably lose half the time… If your expectation was that you were going to wait a really long time for an awesome match where you either 1) Won by a long shot or 2) Barely won… but still won nonetheless, your expectations for what the system can or should do are in the wrong place. We do not generate bots to take losses so you can win more than 50% of the time. Those are real people losing on the other end of every loss you take.
A second factor we take into account is ping. We’re matchmaking people all over the world and we want to match people to the closest servers for the best play experience. In our second stress test, we had other things prioritized over ping-based matchmaking such as skill and time. For those of you who participated in that stress test, you’ll remember how terrible the game performance was on the first day as well as how “lit up” the forums were demanding that players be given an option of server choice. So now we prioritize ping for players. Some players live in challenging parts of the world when it comes to high speed data connections (I’m looking at you, Andes mountain range…) so it’s not perfect for everyone. But largely, most people get a really decent connection to our game servers. Matching players with wildly disparate pings also results in a higher frequency of undesirable side effects such as “getting shot behind walls”. Of course if you live in Houston, Texas and group with your buddy in Geneva, Switzerland, you’re now introducing uncertainty to our system that’s harder for us to deal with… but we allow it.
Which brings us to the next factor that we match on: grouping. The majority of our matches are comprised of either all solo players or solo players and players grouped with one other person. However, the system does try to match groups of equal sizes together first and foremost. As the time people wait grows, we expand the search to try to find others for them to play with. This means that occasionally we will match groups with players who are not grouped or in a group size that is smaller than their own. Like I mentioned, this is exceedingly rare but can happen. And that match is only made when players have crossed a waiting threshold that we deem too long. For most group matches a group of 6 is placed against another group of 6.
Groups are a big challenge in our matchmaking system. You can group with people of wildly varying skill and ping and we allow you to. It’s pretty unlikely that there is another group in the queue that exactly mirrors the unique circumstances that you have set up (pings, skills etc.). We want you to group. We feel that it’s the best way to play the game. So we try to avoid things that discourage grouping and we want to continually improve the social systems so that you’ll find it easier and easier to group with people you have chosen to play with. Playing with people you choose to play with is going to be more reliably fun than playing with people we choose for you. I once used the analogy of hanging out with people on a Saturday night. If you were to go out with five of your friends it would probably be a better time than if we tried to find 5 random people for you to go out with, no matter how smart we were in our selection process…
Anyway, this leads me to matchmaking rating. This rating is the most important thing that we try to match on. Basically this rating means “how good are you?” Commonly, you’ll hear this referred to as Matchmaking Rating or MMR. MMR is derived differently in different games. Overwatch borrows a lot of knowledge from other games but also does a lot of things unique to Overwatch. As each player plays games, their matchmaking rating goes up or down depending on if they win or lose. The system is extremely complicated and there is a lot more going on here than I am going to spell out. So please don’t take this as the comprehensive guide to how MMR is calculated in Overwatch. There is definitely a lot more going on under the hood.
In Overwatch, whether your MMR goes up or down is contingent on winning or losing. But there are a number of factors that determine how much that rating goes up or down. For example, what map you’re playing on and whether you were attacking or defending is factored in. We know the win rates on attack/defend on all of the maps and we normalize accordingly. Not all wins and losses are equal. We also look at your individual performance on each of the heroes you played during the match. Everyone has better and worse heroes and we have tons of data showing us what performance levels should be like on those heroes. We also look at your opponents and whether or not their matchmaking rating is higher or lower than yours. These are just a few of the things that are considered when determining how your skill should go up or down. At no point in MMR calculations do we look at your win/loss ratio and win/loss ratio is never used to determine who to match you with or against. We are not trying to drive your win/loss percentage toward a certain number (although the fact that so many people are at 50% win rates makes us extremely happy). All the system does when it comes to matching on skill is attempt to match you with people of a similar number.
The system is of course deeper than this. There are penalties and handicaps added for things like not playing for a while or playing in groups of varying sizes. We also do special things for brand new players to (hopefully) keep them away from the general population. Players will often mistakenly look at player level and accuse the matchmaker of making unfair matches. One thing that I have mentioned before is that we were evaluating your skill during closed beta, open beta and the second stress test weekend. If you played in any of these (over 10 million players did), we had already determined a skill rating for you (most likely). This means that it’s not uncommon to see a level 1 matchmade against much higher level players. In most of these cases, the Level 1 is a skilled player who played during the phases I mentioned but did not immediately play at launch.
There are many factors that are beyond our control that add noise to the matchmaking system.
• Leavers are extremely disruptive
• Players vary wildly in their skill with different heroes. We have no clue which of the 21 heroes you are going to play during a match
• Groups form with wild variance in skill levels and ping. Contrary to popular belief there is not a “perfect match” for your unique snowflake group
• Sometimes your little brother plays on your account
• Sometimes the cat walks in front of the screen
• Sometimes your wireless mouse runs out of batteries. (Why do you use a wireless mouse btw?)
• Sometimes a highly skilled player buys a new copy of OW to “start fresh” on a new account
• Sometimes you have internet problems
• Sometimes you play drunk or tired… or both
• That first game of the night…
• …that last game of the night
• “Life”
So this brings me to some thoughts I’ve been having about Overwatch. While this whole post has been mostly my personal thoughts – this next part is “especially” my thoughts and not reflective of the team or the company’s POV.
For better or for worse, we focused the design of the game on winning or losing as a team. OW is not a game where you ignore the map objectives and then look at your K/D ratio to determine how good you are. We want you to focus on winning or losing and as a result you do focus on winning or losing. We tried to make it so that losing isn’t the end of the world, but to a lot of people they expect to win far much more than they lose. I sometimes wonder if we were able to clone you 11 times and then put you in a match with and against yourself, would you be happy with the outcome? Even if you lost? Out of the 5 types of matches I described above, it is my belief that you would still experience types 1-4. Are those “stomps” still not acceptable? Because they will happen…
And I believe OW is strange game in that regard. I spend a lot of time studying the matches that I am in because I am very focused on matchmaking. I’ve been in so many Control Point maps where my team got destroyed on the first point, the enemy team got destroyed on the second point and then we play the third point to a 99%/99% overtime. If you judged any of those single points on their own merits you could say you have two stomps (one in your favor, one against you) and one close match. Same players...no change in matchmaking. Or take a match that I was just in on Route 66, for example. My team was on attack and could barely push out past the train cars. Two members of our team swapped heroes and we proceeded to march the payload all the way to the end of the map practically uncontested. The match went from a stomp in one direction to a stomp in the other direction.
So while it is possible for a mismatch to result in a stomp, not every stomp is a mismatch. If every time a team dominates another team it is viewed as “the matchmaker is broken”, the problem we have is with perception and expectations. Look across all pro sports. Even matches happen every night. Stomps happen every night. It’s a reality of any competitive game. Does that make being on the receiving end sting any less – probably not.
We are constantly improving the matchmaker. We learn more each day. We have one of our best engineers and best designers full time dedicated to the system. Many of those “silent” patches that go out during the week are adjustments to the system. For example, we recently realized that “Avoid this player” was wreaking havoc on matchmaking. One of the best Widowmaker players in the world complained to us about long queue times. We looked into it and found that hundreds of other players had avoided him (he’s a nice guy – they avoided him because they did not want to play against him, not because of misbehavior). The end result was that it took him an extremely long time to find a match. The worst part was, by the time he finally got a match, he had been waiting so long that the system had “opened up” to lower skill players. Now one of the best Widowmaker players was facing off against players at a lower skill level. As a result, we’ve disabled the Avoid system (the UI will go away in an upcoming patch). The system was designed with the best intent. But the results were pretty disastrous.
We will always be working on our matchmaking system. We’re listening to feedback, we’re playing the game a ton ourselves and we’re looking at hard data to inform our decisions. This post wasn’t my way of saying everything is fine. I just wanted to share some of my thoughts as someone who has been evaluating the system itself very closely as well as monitoring the feedback. I want to put it out there that there is a lot of room for improvement but also suggest that there are forces in play that cause some fair matches to sway lopsided due to forces out of our control. The game is as much (if not more) art than it is science. We’ll keep working to make it better!"
