Dev Diary - Game Stats

Defiant

Project Leader
Movie Battles II Team
Code Leader
Posts
790
Reaction score
1,065

Developer Diary - Game Stats

In the next release we will be adding the capacity for game servers to send back statistics for analysis by the development team. This is one of the most important features that we have ever added to MBII. Up until now all design decisions and feedback we have had has come from anecdotal evidence and different people often disagree wildly. Gathering statistics will allow these discussions to be data informed rather than relying on different people's often irreconcilable points of view.

To that end, with the next release of MBII libcurl will be a required component for server hosts. For windows users this library will be supplied, pre-packaged in the usual way either via launcher or the manual installations. For Linux server hosts the 32bit libcurl library will need be installed or the game will not load.

What data is being gathered?

Whilst this will be an ever evolving component of MBII we are starting off by gathering the readily accessible but most meaningful data. This is the data which was quick to implement but will give us the biggest capability when it comes assisting decision making in the short term. As time goes on extra pieces of information will be gathered, which could be more of the generic data as the first release is, or highly targeted data to assist with answering specific questions.

The first release will gather the following information:

For each Game:
  • Server Name/IP Address
  • Map
  • Game Mode
  • Game Duration
  • Winning Team
  • Are Teams Swapped?
  • MBII Version
  • GameStats Version
  • Game Date and Time
  • Progressive Siege Area
  • List of Kills (The specific data for this is specified in the "For each Kill section")
  • List of Players (The specific data for this is specified in the "For each Player section")
For each Kill:
  • Client Slot Killed
  • Client Slot Killed By
  • Client Slot Assisted By
  • Time between Assist Damage Dealt and Death
  • Location of dead Player
  • Location of killer Player
  • Location of Assisting Player
  • Means of Death
  • Means of Assist
For each Player:
  • Client Slot
  • Player Abilities
  • Player Class

What data is not being gathered?

Whilst it is impossible to list every piece of information that is not being gathered, hopefully the following list will show the types of data that are not being gathered and will never be added to the game stats we do gather. These are:
  • Any information from the client. All data is gathered from the servers. Only dedicated servers will send data (Where g_dedicated is not 0) - local games will not send data back.
  • Any personal information that the server sees - including but not limited to: Player Names, Player IP addresses, Player Scores, any information that could be used to identify a given player.

Analyzing the Data

For the first release we will be looking to analyse the data in two ways. Firstly for game play design and secondly to provide information back to level designers.

Level Designers

From the information gathered level designers will be able to access heat maps for each map built into MBII. A separate heat map will be available for kills, deaths and assists and level designers will be able to access these either at the level for individual games, or for all games played on that individual map. The sort of conclusions that might be drawn from this are:
  • Are players generally being killed before they get too close to the objective?
  • Are players being killed in an unanticipated place?
  • Are there places which are not intended to be reachable which can be exploited for kills?
Here is a heat map for a few sessions played by the beta team on Lunar Base:
HeatMap.png


Gameplay

Gameplay statistics are useful in a huge variety of ways, and how we can analyse the data effectively will be an on going learning activity for us. As a starting point we have reports written which will answer a few basic questions.
  • What is the relative popularity of the classes?
  • Which abilities are most commonly chosen for each class? At what level are they taken?
  • Which builds are very commonly chosen?
  • Which weapons are getting the most kills?
  • Which weapons are getting the most assists?
Importantly this give us a base point to analyse the changes we make and determine how effective they have been. For example, if Elite Trooper wasn't often selected and didn't often get kills when it was played, we might decide to look at if it was under powered. If it was, and we made changes to buff it, we would be able to look after the release and see if the amount of people choosing the class has increased and the class was being more effective in combat. A sustained improvement over time would indicate that what we had changed was successful, where as a short term peak followed by a return to the old values might show people who have tried the changes we've made but it not being enough to make the class competitive.

Here is what our statistics from a few beta sessions look like - try not to pay attention to the numbers too much as this is a highly skewed group who would have been testing specific other things working their way through development. As an example of how useful the statistics are, you could probably work out which classes they have been doing extra testing on...

ClassStats.png


FcfStats.png


MODStats.png


Server Owners
All this new functionality requires use of a third party library - in this case we are using libcurl to perform the network functionality. If you are a Windows server owner or a player self hosting on Windows then the required library will be provided by launcher in the flavor of libcurl.dll. For Linux server owners (And anyone self hosting on linux via OpenJK) you will need to install the i386 version of curl in order to start a game.

For Ubuntu/Debian systems:
Code:
sudo apt-get install curl:i386
For Older Ubuntu (At least 16.04 and Prior)/Debian
Code:
sudo apt-get install libcurl3:i386
For CentOS systems:
Code:
sudo yum install libcurl.i686
If this library is not available the following errors will appear in the console output:
Code:
Sys_LoadDll(<homepath>/GameData/MBIITest/jampgamei386.so) failed: "libcurl.so.4: cannot open shared object file: No such file or directory"
Sys_LoadDll(<executable>/MBII/jampgamei386.so) failed: "<executable>/MBII/jampgamei386.so: cannot open shared object file: Not a directory"
Sys_Error: Sys_LoadDll (jampgame) failed dlopen() completely!
 
Posts
10
Reaction score
3
Wow this is awesome! Thanks. It'll be cool to get some objective data instead of relying on peoples' personal anecdotes.
 
Posts
15
Reaction score
32
Can't wait to see the analytics that say that 80 percent of the playerbase is playing jedi/sith and most of the deaths are assisted by force push followed by suicide. I also I just realized there is a "Mod_Slime" in that screenshot above...what?
 
Posts
57
Reaction score
45
Mmm... analytics. I like it, it helps to look for bugs. And you can throw the numbers in some MS PowerBI :D
 
Posts
23
Reaction score
34
Hard numbers and facts are delicious so this gets a thumbs up from me.

If you see a lot of Disruptor rifle usage, it'll be because of me just so you know.
 
Posts
25
Reaction score
28
Now my data won't be safe even when playing MBII! Is there a single company out there which isn't trying to steal my data?! smh my head
 

Defiant

Project Leader
Movie Battles II Team
Code Leader
Posts
790
Reaction score
1,065
Ok, now we are moving from a team-based game to an Orwellian dystopia
Disagreeing wildly about core mechanics and gameplay is a basic human right and you will not take it from us
Yes. This is the latest step in my plot to form the first galactic empire. But no, we wont be talking away your ability to argue. Just we will bring statistics to the fight. No one has ever been able to argue about statistics...

Can't wait to see the analytics that say that 80 percent of the playerbase is playing jedi/sith and most of the deaths are assisted by force push followed by suicide. I also I just realized there is a "Mod_Slime" in that screenshot above...what?
That is what we will find out. Are 80% of people playing Jedi? Do most deaths come from being pushed? If for arguments sake we reduced the range of push, in the next version does it affect the number of people dieing from push assists in the way we expected or not?

As mace said. MOD_SLIME is acid.

Now my data won't be safe even when playing MBII! Is there a single company out there which isn't trying to steal my data?! smh my head
Nothing from the client is being sent. The only information being gathered is from the MBII server - infact the only piece of information even indirectly being sent from your client is the force string which determines class and abilities. All other data being sent is generated and owned by the server.
 
Posts
19
Reaction score
37
This is very cool. I think it'll be fun to see how the data shakes out. My only concern is that while data is useful, ultimately what matters is the balance and gameplay philosophy the dev team is aiming for, since this will inform how the data is interpreted.

I'm thinking of something like my friend's propensity to fire rockets at a group of Jedi and the shocking number of times said rockets don't get pushed back, killing two or more players.

On paper, it's going to look like the rocket is highly effective, whereas in reality, it's the enemy players who are not being diligent about countering the rocket with push.

I don't EXPECT that the data will be interpreted that way by the dev team, just vocalizing that it's one of the risks of using data-based feedback generically.
 

MaceMadunusus

Level Designer
Movie Battles II Team Retired
Posts
1,621
Reaction score
2,163
Are the analytics going to be publicly accessible at any point?
Likely not, the processing requirements are pretty high for even just us to view them. For example it can take over a minute to generate the heatmap for just a few days of usage. We don't have particularly high powered servers to process all the information for viewing by a large number of people. We would have to bump up donation requirements significantly if we wanted to do that. However, after it has run for a while, maybe if you want something specific you can ask us about it. It will be a while before we actively start using this as it needs time to gather information.

My only concern is that while data is useful, ultimately what matters is the balance and gameplay philosophy the dev team is aiming for, since this will inform how the data is interpreted.generically.
That is why Defiant mentioned us being "data informed". We are merely just adding another input to our design process, not making it the only part of our process. The data cannot give a full picture but can help us unblur the picture a bit. It would be really bad to use the data on its own to balance/design things, especially in a Star Wars game like we have, where certain classes will be picked more simply because of the theme, and not because of any kind of balance related thing as an example.

If you followed DOTFV2 at all, I specifically mentioned that I just didn't have enough beta testers, players, etc to be fully happy with the layout and things on release, and noted that it wasn't perfect just needing more data. There is only so much I can do when joining servers and flying around in spectator watching people for example. Things like the heatmap are not a replacement for anything like that for reasons you and I both mentioned, but will help refine further just allowing me to see more.
 
Posts
19
Reaction score
37
If you followed DOTFV2 at all, I specifically mentioned that I just didn't have enough beta testers, players, etc to be fully happy with the layout and things on release, and noted that it wasn't perfect just needing more data. There is only so much I can do when joining servers and flying around in spectator watching people for example. Things like the heatmap are not a replacement for anything like that for reasons you and I both mentioned, but will help refine further just allowing me to see more.
I was on hiatus through all that, but I can respect that even with the best of intentions, the resources are not always at hand to do the job you'd like to do. It's reassuring to see a measured approach to the inclusion of data in feedback--thanks for your response.
 
Posts
44
Reaction score
23
Great,i hope with that update you guys will be able to balance the game better.(Would be also cool to ban all timewasters,trolls,glitchers, and aimbot/wh/macros/scripts/etc users)
 
Last edited:
Posts
25
Reaction score
28
Nothing from the client is being sent. The only information being gathered is from the MBII server - infact the only piece of information even indirectly being sent from your client is the force string which determines class and abilities. All other data being sent is generated and owned by the server.
Sorry, the comment I posted above was just an attempt at a joke! God forbid someone agrees with what I wrote!

I had read the main post and I understood that no private data would be gathered.

For the record, I think it's a great idea and I hope it will make it easier for you to see what needs to be balanced further. :D
 
Top