1/28/2018 · originally published on uselessanalysis.com

Stupid, Stupid Idea: A Smash Bros. Spectacular!

Two years ago we redefined the scale of useless analysis when we watched several hours of the demo mode in Street Fighter II in order to see which character is the best when controlled by the computer. We followed this up with a Street Fighter IV version as well.

It's time to bring this concept back, this time for Super Smash Bros. for Wii U!

Super Smash Bros. for Wii U is crazy. It has 40 fighters included in the game out of the box, you can unlock and purchase more, and the game even includes an 8-player competitive mode which is just ridiculous mayhem. It's the perfect match for some useless analysis!

I decided that I wanted to test two different elements in the game: 1) who are the best fighters when they're controlled by the computer, and 2) how much better is the computer when it's set to level 9 vs. when it's set to lower levels? This second element is easily testable because Smash allows you to set the CPU's skill level between 1 - total newb - and 9 - expert.

So given several dozen fighters and multiple CPU levels to test, here's how I set things up:

Set up a series of 8-player CPU battles
Each battle will be a stock battle with 3 lives: 1st place goes to the last fighter standing, 2nd place goes to the 2nd-to-last fighter standing, and so on
In each match, there will be one CPU set to level 9, one to level 8, and so on down to level 2 (I decided not to test level 1)
Fighters and stages will be completely random
Only the original 40 unlocked fighters are included in the simulations
I watched 90 rounds of these 8-player battles, which took around 10 hours to do across a few weeks (I originally was targeting 200 rounds but decided that would take way too long)

All in all, I think this was the most research put into an analysis in the history of this site! (Or perhaps as my wife would characterize it, the most time wasted?)

So what did we find out? Let's start with CPU skill level. Here's a breakdown of the 90 matches into winners and top-3-finishes:

As we'd expect, the higher CPU levels perform better than the lower CPU levels with randomized fighters. Some of our takeaways:

72 of the 90 battles (80%) finished with CPU 7, 8, or 9 winning
There's a big dropoff from CPU level 9 down to 8, and a bigger one from CPU level 7 down to 6
CPU level 6 is the only one with fewer wins and top 3 finishes than a lower level fighter
Every CPU level won at least once - although CPU level 2 got its win in round #89 out of 90!

It's unclear to me how the game processes different levels. Do higher level CPUs try to target their biggest potential threats? That might explain why CPU 6 struggled while CPU 5 slipped under the radar.

It's also possible that CPU 6 simply had some bad luck - either with the random fighters it was assigned or just plain random variation - and extending the simulation would show that CPU 6 is truly better than CPU 5.

From here on out, we'll calculate each CPU's "average finish", which is simply the average of the rank placement across the 90 rounds:

Average Finish (lower is better)
CPU 9: 3.02
CPU 8: 3.73
CPU 7: 3.67 (better than CPU 8!)
CPU 6: 4.58
CPU 5: 4.42 (better than CPU 6!)
CPU 4: 5.02
CPU 3: 5.58
CPU 2: 5.63

This is basically the same story as the wins / top-3 graph, but we need these figures in order to move onto the next phase of the analysis, identifying the best characters!

So we have have a few challenges when evaluating the fighters. Since characters were randomly assigned, not every character was chosen an equal amount of times, and there definitely is not guarantee that every character had the same number of attempts as CPU 9 as they did as CPU 2.

Ultimately, I decided to look at the data several ways:

Win %
Average finish
A +/- rating that compares to their expected finish based on the CPU levels they were assigned

#1 and #2 were easy to compile since they're basically the same as metrics above.

For #3, I set up a simple weighting function where I took the distribution of CPU levels for each fighter and multiplied it by the average finishes for each of the CPU levels above. So if a fighter was always assigned to CPU 9 we'd expect them to average a finish of 3.02. If that fighter's actual finish was a 2.0, then they finished 1.02 points better than expected and therefore would have a +/- rating of 1.02.

On to the results table!

Some highlights that I found interesting:

I certainly never expected Kirby to be #1! And he's way out in front of #2, Mega Man!
I love that Toon Link (#3) beat regular Link (#5), although he has a much lower win %
I always made fun of people who chose Mario (#4), but it seems like that's actually a pretty justified choice
Shulk (#11) ends up below Palutena (#10) despite a win % that is substantially higher - this is because Shulk had stronger CPU levels on average than Palutena did (2nd-highest for Shulk, 35th for Palutena)
- I have never played a game that had either of these characters in it, so Smash is my introduction to them!
Little Mac (#36) was top 5 based on his win %, but was ultimately very boom-or-bust - much like his trademark Star Punch
I'm sad that Zero Suit Samus finished in last place, but at least normal Samus (#24) semi-redeemed the franchise

Calling back to the original question, what's a bigger factor - CPU skill or the character selected? Well, the difference between Kirby and Zero Suit Samus is (1.45 + 1.09) = 2.54. That's nearly the difference in average finish between CPU 2 and CPU 9 of (5.63 - 3.02) = 2.61. So it looks like they're basically equal factors: Kirby at CPU level 2 and Zero Suit Samus at CPU level 9 would be expected to be comparable to each other.

This exceptionally useless exercise was fun, and it demonstrates to me that watching video games play themselves can continue to be productive. Now, let's close it out on a Kirby dance. Great job little guy!!

-rj