Balance on ANB rewards

[Ozymandius]

I can't make sense of this... How can you be in a state where you haven't seen the data?

Scientists make their predictions and present their hypothesis in a research proposal before doing the study and collecting the data. Making a conclusion after looking at the data is considered to be "data mining" because it takes advantage of random variations in the existing data set. You could look at the results from this event, and then make predictions for the next event; that is called "bootstrapping" and is considered to be acceptable under most situations because you are testing your hypothesis on a different set of data.

I don't understand why you use approximate percentage numbers while the calculator allows for the exact population counts.

A Z-test can only be used to test the significance of the difference between sample means or proportions; in this case proportions made the most sense. The only statistical test where you can use the actual numbers is the Chi-square test (or Fisher's Exact test when sample sizes are small).

Are you trying to impress the readers by using Latin?

No, ceteris paribus is a term that scientists use to remind themselves that a single test of a hypothesis rarely "proves" anything because you do not have the experimental controls in place to test alternative hypotheses. Since you are using statistics to support your position, I assumed that you would recognize the term.

After further thinking and discussions, I no longer advocate this. Same for all is simple, doesn't need to be adjusted as players discover new strategies using more and more unlocked glory recipes, and also if there is an adjustment it should be done across all milestone rewards, at least the high end ones.

Plus, each class have it's pros and cons, this is just another factor that players must take into account when choosing a class, and surely not the most important one. (Which is, have fun!)

Anyway, I thank you for this conversation, and the link, I learned a lot about these statistical tests. Have fun in the sixth ANB!

I agree with your conclusions, class differences in achieving ANB milestones is another factor that players should consider when choosing a class, but in the end, none of these factors should interfere with having fun!

[LodWig]

I used a two-tailed test for the null hypothesis that there is no difference between classes because I did not make a claim that one class would come out higher than another BEFORE seeing the data [making a more specific hypothesis AFTER seeing the data would be considered unscientific and my goal is to objectively analyze the data].

I can't make sense of this... How can you be in a state where you haven't seen the data? Or maybe you mean the results of some tests?

I am not sure where you got the numbers for your Z-test.

As is hinted by the hypothesis tested, "Warriors are more likely to get to the All Class [sic] ANB Top 100 Leaderboard", the numbers come from the All Classes ANB Top 100 Leaderboard (sorry for the improper grammar). I had given them at the end of this comment, you must have missed this bit I added in an edit.

If you combine BH and Mages you have 155 out of 6198 = .0250, and for Warriors you have 100 out of 3184 = .0314.

It's 155 out of 6918 = 0.0224, and 100 out of 3184 = .0318. Besides these two errors, I don't understand why you use approximate percentage numbers while the calculator allows for the exact population counts. With the correct data, I get:

Keep in mind that this is assuming the worst possible case for warriors, that is only 100 got to TL 130. One player who reached it told me he was ranked 169.

The hypothesis that Warrior has an advantage over other classes is again not supported by this "post hoc" analysis.

It would if you had entered correct numbers, as shown by the Fisher test in my previous post.

Your three column test did come out significant, but

This test was the first suggested by the site wizard. It came out significant, meaning there is a correlation between the class and achieving the milestone. Why discard this result? On what grounds? Also, it came out significant and was contradicting your results, you should have investigated, it would have show you the many errors you did.

Like the Z-test of the same hypothesis, the Chi Square also supports this claim:

Even better with the correct numbers:

In summary, the results suggest that, all other things being equal (ceteris paribus), Warriors did appear to have had an advantage over Mages in achieving the Trial 130 milestone in this event.

There! (Are you trying to impress the readers by using Latin?)

However, all other things are rarely equal, and we still do not know if there were significant differences between Warriors and Mages on CL points or damage/toughness stats.

Of course there are differences between classes! If not, why having three classes? The "all things being equal" not being true pertains to the out-of-the-game realm, e.g. hardware used, natural skills, money, gaming uprightness, etc. And all these factors may very well bend the results, I am aware of that.

It could simply be that the Warriors in this event who passed the TL130 milestone had more CL points or damage/toughness than those Mages who tried but failed to reach that same milestone.

And again, it is a fact that Warriors may reach higher Armor than Mages (the attribute range is higher for warrior items), so you can't say "it's not a correlation to class but to armor" because precisely the two are factually correlated. The same goes for the CL, it is common belief, based on many observations, that classes are not equal in this respect, especially when the time allocated is so scarce.

My own thinking is that the major difference between the classes is that Warriors have higher maximum Armor stats than Mages, who have the lowest maximum armor of all.

The question was not to pinpoint the reason why some class have it easier than others, but to see whether it's true or not. I do think another big factor is the speed at which CLs are gained, especially in a 6 hours event, given that less XP is needed to increase a low CL, and gives better relative benefits than a high CL increase.

Instead of arguing for lowering the milestone for Mages in ANBs, it makes more sense to push for more balancing of maximum armor stats across the three classes or for adding some other gear buff that would make up the difference.

Sorry that makes no sense, as it would end up having all three classes equal in all respect, and also the classes are not directly competing to each other, so balancing is not required. Also, imagine if hunters were as tanky as warrior?

About the adjustment of milestones to class: After further thinking and discussions, I no longer advocate this. Same for all is simple, doesn't need to be adjusted as players discover new strategies using more and more unlocked glory recipes, and also if there is an adjustment it should be done across all milestone rewards, at least the high end ones. Plus, each class have it's pros and cons, this is just another factor that players must take into account when choosing a class, and surely not the most important one. (Which is, have fun!)

Anyway, I thank you for this conversation, and the link, I learned a lot about these statistical tests. Have fun in the sixth ANB!

[Ozymandius]

I ran my Z tests using the numbers that you provided in your original post:

TL for #100 is:
- hunter, 129 out of 4154 = .0310
- mage, 120 out of 2764 = .0434
- warrior, 136 out of 3184 = .0427

Number of players reaching TL 130:
- hunter, 99 out of 4154 =.0238
- mage, 56 out of 2764 = .0202
- warrior, ?? out of 3184 = ?

None of these comparisons (e.g., BH vs. Mage, BH vs. Warrior, Mage vs. Warrior) had a statistically significant Z test result. However, if you replace the ?? with 100 for Warriors completing TL130 vs. Mages completing TL130, then that test would be significant (z = 2.697, p < .01).

I used a two-tailed test for the null hypothesis that there is no difference between classes because I did not make a claim that one class would come out higher than another BEFORE seeing the data [making a more specific hypothesis AFTER seeing the data would be considered unscientific and my goal is to objectively analyze the data]. I am not sure where you got the numbers for your Z-test. If you combine BH and Mages you have 155 out of 6198 = .0250, and for Warriors you have 100 out of 3184 = .0314. The results of this analysis are shown below:



If you are worried about the Z test not being robust to violations of the normal distribution, you could use a nonparametric test, like a Chi Square, which should be relatively close to the Z-test results (see below):


The hypothesis that Warrior has an advantage over other classes is again not supported by this "post hoc" analysis.

Your three column test did come out significant, but I think that the test that you really wanted to do was to compare Mage vs. Warrior based on your hypothesis that Mage is at a disadvantage for completing Trial 130 compared to Warrior. Like the Z-test of the same hypothesis, the Chi Square also supports this claim:



In summary, the results suggest that, all other things being equal (ceteris paribus), Warriors did appear to have had an advantage over Mages in achieving the Trial 130 milestone in this event. However, all other things are rarely equal, and we still do not know if there were significant differences between Warriors and Mages on CL points or damage/toughness stats. It could simply be that the Warriors in this event who passed the TL130 milestone had more CL points or damage/toughness than those Mages who tried but failed to reach that same milestone. My own thinking is that the major difference between the classes is that Warriors have higher maximum Armor stats than Mages, who have the lowest maximum armor of all. This difference appears to be exaggerated following the new War Gear update, which factors in Armor more than the previous version of the game. Instead of arguing for lowering the milestone for Mages in ANBs, it makes more sense to push for more balancing of maximum armor stats across the three classes or for adding some other gear buff that would make up the difference.

[Argon]

Warriors get milestones easier than mages, but mages get medals easier than warriors. BHs somewhere in the middle?
But best option would be a buff for mages, plus fixing frostbeam on mobiles and vortex in general.

[orion_134]

Would I ever say such a thing?!

[LodWig]

Could you elaborate on the test you made on the website? You did three comparison, "warriors at 130+" vs "non-warriors at 130+" etc? Also, what are the conditions on the observed statistic for this Z-test to be adequate?

PS. I just used the calculator, for the hypothesis "Warriors are more likely to get to the All Class ANB Top 100 Leaderboard", entering 65 / 3184 / 35 / 6918 in the four fields, is that the way to do the test? I get this result: The value of z is 7.2426. The value of p is < .00001.

PPS. Still learning, I used a Chi-Square test for correlation between class and reaching TL 130, using the low estimate of 100 for warriors, got this:

Results
	Hunters	Mages	Warriors	Row Totals
Pass	99 (104.86) [0.33]	56 (69.77) [2.72]	100 (80.37) [4.79]	255
Fail	4055 (4049.14) [0.01]	2708 (2694.23) [0.07]	3084 (3103.63) [0.12]	9847
Column Totals	4154	2764	3184	10102 (Grand Total)

The chi-square statistic is 8.0413. The p-value is .017941. The result is significant at p < .05.

And also a Fisher Exact Test:

Results
	Warriors	Others	Marginal Row Totals
Pass	100	155	255
Fail	3084	6763	9847
Marginal Column Totals	3184	6918	10102 (Grand Total)

The Fisher exact test statistic value is 0.0093. The result is significant at p < .05.

Am I doing it wrong?

[LodWig]

the proportion of Warriors in the top 100 is not significantly higher than the number of Mages or BHs in the top 100

Huh? I wrote

The ratios to the respective populations are, in the same order, 2.04%, 0.48% and 0.54%.

and still you say not much more warriors? Again, because you do a difference instead of a division?

most of the variance in the Season leaderboard rankings would come from CL points

In retrospect I shouldn't have mentioned the Season LBs. It's useless, because many accounts are dead, because many cheaters, because a wide variety of account creation dates (and at one point in time there was only one class). Also, not everyone have pushed yet. So, forget about it. Anyway, saying that the correlation is more to the CL than to the class is completely ignoring the correlation between ease of XP grinding and class. Which is even stronger on a 6h long limited event.

just analyzed Damage, Toughness, and Recovery stats as predictors of final ranking.

"Damage" is a very poor indicator of the real in-combat DPS

[strngtn]

Seems you are giving a wrong(leading) result based on your data. i.e
There 4% Warriors passing TL130 and 2% for Mages. So the difference is only 2%, that is quite a small and even ignorable difference. The balance is great!

Actually the difficulty of passing TL130 for Mage is doubled while comparing to Warriors.

BTW, the long tail data are usually useless . e.g I just create an account and pay no efforts on TL in last Silver ANB.

Here are the final leader board:

Warriors Mages Bounty Hunters
Rank Name Trial Time Name Trial Time Name Trial Time
1 Chinahwang 157 9m49s Koreaabin 152 8m39s Illiililil 159 8m43s
100 Harok 135 8m35s Odnol 118 6m38s Move 129 9m43s

[Ozymandius]

I used a two tailed Z-test for comparing the difference between the proportions of heroes in each class using the data that you provided with the null hypothesis being that the proportions are equal. The Z-test somewhat corrects for the fact that there are different numbers of players in each class. You would naturally expect that more BHs than Mages would meet the criterion, since there are 1,390 more players in that class compared to Mage. I was not able to calculate a Z-test for Warrior because the number of Warriors completing TL130 is missing, but the proportion of Warriors in the top 100 is not significantly higher than the number of Mages or BHs in the top 100. Therefore, I concluded based on your data that there is no statistically significant difference between the success rate of the three classes of heroes. You can run the tests yourself at the following website:

https://www.socscistatistics.com/tests/ztest/

Just remember that 2.38% is equal to a proportion of .0238; you just enter the proportions and sample sizes, and the program will do the calculations and provide the result. If the percentage of Warriors who completed TL130 does turn out to be 4% as you hypothesized, then that one Z-test would be statistically significant, however, the other five tests would not be. It is difficult to argue in favor of an imbalance between classes based on only 1 out of 6 statistical tests. Your season leaderboard data perhaps would provide a better test, but it is currently a moving target as the numbers have already changed since your post, so that analysis would have to wait until the end of the Season when the results are finalized. It would be interesting to look at a multivariate ANOVA of the Season results to see if hero class turns out to be a significant influence, but my initial scan of the data leads me to believe that most of the variance in the Season leaderboard rankings would come from CL points (i.e., the top leaders regardless of class trend toward having more CL points than heroes lower down on the leaderboard). It is not a perfect correlation, but I would venture that the Spearmann's r for the correlation between final ranking on the leaderboard and CL points would be very high. You would probably get even better predictive results if you ignored CL points altogether and just analyzed Damage, Toughness, and Recovery stats as predictors of final ranking. Once you account for these three stats, any variance left over would have to come from either player skill, luck, or some combination skill X luck.

I would still argue that changing the ANB milestones is premature.based on the existing analyses, though I would be willing to crunch the data again once the ANB final leaderboards have been scrubbed of players who failed to play by the rules. If you broaden your research to include Damage, Toughness, & Recovery data, you might conclude that although the average weapon damage stats are fairly equal across the three classes, the Warrior class has a distinct advantage in having larger maximum Armor stats which gives them an instant advantage in Toughness. The best way to balance classes may be a simple as boosting the Armor stats for BH and Mage, especially now that Armor plays a bigger role following the War Gear Update.

[LodWig]

My point is that if you think that another class can reach the goal more easily, then you are free to play that class.

Which, again, is not an argument for or against the fact that there is imbalance.

The classes seem reasonably balanced right now

This is precisely the question we are debating. Saying upfront there is no imbalance is not a convincing argument.

but that could change once people are able to craft more of the new gear sets

In one way or another: maybe more balance, maybe more imbalance. We are discussing the now, more precisely next league.

I can run the Z-test again once we have the actual number of Warriors who reached TL130, but based on the current estimate, it is not a statistically significant difference.

What is the statistic on all the population you are looking at? The trial level? You suppose a normal distribution for this? And seriously you don't think the trial level is influenced by the class? Have you looked at leaderboards, where you can see the best players in each class? 34 hunters, 57 warriors, 9 mages in top 100. We can't know the active population, total or per class, but may suppose the percentages follows the one for last event. Surely you can reasonably suppose the class is significant in best trial reached, no? Or do you think best players choose to play warrior, worst to play mages, and less than average choose to play hunter?
Please show the statistical test you did, I will try to understand it, this is not my area of expertise at all (to say the least) but I will seek help.

Finally, if there is a significant imbalance between classes, then the Developers should work on buffing up the attack options within that class instead of just changing the ANB milestone goals.

Yes, this could also be a solution. I think the milestone changing could be easier to implement. The developers may use all the data they have in order to tune the milestones to their liking, but imho the bar should be set equally for all classes. The previous important milestones (80 and 100 for the Celestial Transform) were more balanced, even the present milestones for CTs (90 and 110) are good, given the War Gear update, in the sense that the difference between classes in the ease to attain them was/is not much. But of course, the higher the TL, the higher the imbalance shows. And as you are sure aware, TL 130 is way harder than TL 110 (foes hits 2.7 times more, and have 6.7 more health).

I'm sure the developers wanted the last reward to be challenging. I'm not advocating for lower bar. A reward at 140 for warrior, 135 for hunters and 130 for mages would be good by me. But a flat 130 for everyone doesn't seem fair to me. (Full disclosure: I reached TL 139 with a hunter in last gold.)

PS. Given the time limit on events, the comparison with Season leaderboards is not totally relevant. Still, only looking at the ANB all class leaderboard shows a huge majority (65) of warriors in top 100, then about the same number of hunters (20) and mages (15), with the first mage being rank 25. The ratios to the respective populations are, in the same order, 2.04%, 0.48% and 0.54%. But, again, the top mage is #25, above him you find 15 warriors and 9 hunters. (With only hunters in top 5, yeah! At least for now...)

[Ozymandius]

My point is that if you think that another class can reach the goal more easily, then you are free to play that class. The classes seem reasonably balanced right now, but that could change once people are able to craft more of the new gear sets. I expect that we will see more of these newer builds in the next ANB now that players have Glory to open up a full set.

I can run the Z-test again once we have the actual number of Warriors who reached TL130, but based on the current estimate, it is not a statistically significant difference.

Finally, if there is a significant imbalance between classes, then the Developers should work on buffing up the attack options within that class instead of just changing the ANB milestone goals.

[LodWig]

Seeing a ratio of 3.14/2.02, or more likely 4/2.02, in percentages, I conclude that it's easier for warrior. Doing a difference is not wise, as the percentages are small. If the percentages were 99 and 97, then I agree, the class would not be a significant factor, but then the ratio would be 1.02, not 2.

Also, yes, nothing stops me for playing warrior, so what? How is that related to imbalance?

[Ozymandius]

I am confident that a percentage difference of 1.22 [3.14-2.02] would not be statistically significant. As a researcher, I would have to conclude that there is no difference between groups that cannot be accounted for by chance alone.

But hey, there is nothing stopping you from playing Warrior next ANB. Based on the number of players in each class, it looks like a lot of former Arcanist Mages probably changed classes this past ANB after the upgrade nerfed their attacks. People switch classes a lot. I am thinking about playing a Fury Warrior in Bronze myself, mostly because it is a fun build, and I am low on gems after Gold. I will be lucky to get even one CT using the Epic box gear.

[LodWig]

I don't have the data for TL 120, not for TL130 for warriors. But even with 100 warriors going to 130 or more, you get 3.14%, which is 1.55 times more than 2.02%. Even the ratio of percentages for hunters and mages is just shy of 1.18.

And two times more warrior than mage (in percentage) needs only 128 warriors reaching 130+...

[Ozymandius]

It is more balanced than you think. Since the number of players in each class is different, you need to look at the percentage of players who hit a certain milestone instead of the raw numbers.

Using your data, here are the % pf players who completed these milestones:

Warrior 4.27% [??]
Mage 4.34% [2.02%]
Hunter 3.10% [2.38%]

It looks like just over 2% of players completed TL130. You did not post a number of Warriors who finished that trial, but these % look fairly balanced. We don't really know which class had more experienced players versus players who were trying out a new class or a new gear set for maybe the first time. Also, the leaderboard has not been purged of any players who may have broken rules to hit those milestones.

If you have the data, I would like to see how many completed the TL120 milestone.