Dev Diary - Hosting Migration

iwCSA26.png


A Brief History
About three and a half years ago, after a rather disastrous loss of hosting, the active team and many retirees pulled together to rebuild from the ashes. For those around at the time, it's fairly well known that LoU did all of the backend work setting up the website, forums, and development infrastructure, among other things. Less well known is that our main hosting and beta testing servers were generously provided by Hexodious. Alas, all good things come to an end...

The team was given notice at the end of last year that this arrangement would need to come to an end soon, so we have been assessing our options. While the timeline for completing the move is undefined (no need to panic over an encore of 2015), the consensus of the team is to try and get it done ASAP to relieve Hexodious of the ongoing responsibility.

And of course we would like to formally, and publicly send our enormous gratitude to Hexodious for what has been his large personal financial contribution over the past few years. We as developers and players are forever in your debt.

The Future
While I am retired from development (and not planning to change that status), I've still kept tabs on goings on with the team and done a few things like recreating the web serverlist. Since the start of the year I've been sizing up hosting options and mentally spitballing the problem. Last month I volunteered to take on the task of migrating everything.

As already mentioned, this time around we have a bit of time to consider infrastructure and making improvements. While the current hosting has served us well over the years, its main weakness has been the lack of network capacity at release time. Any large updates (> 1 GB of files) usually result in very slow download speeds on release day up until enough of the new assets are cached with the CDN. While Defiant's refactoring of launcher downloading has greatly helped with asset corruption associated with that congestion, the update speed is still a problem. This means we want the new host to have both better upload speed and enough data transfer to avoid overage fees or throttling.

Another consideration is security. Historically it's not uncommon for official and community servers and services to come under DDoS attacks. As disappointing as that is, it's still a reality that demands planning for. This constrains us to only picking among seasoned hosting providers (won't fold the moment an attack starts) and carefully planning infrastructure to help compartmentalize the effects. All of this ultimately increases hosting costs.

Lastly, since we are essentially refactoring most of the infrastructure anyway, I am aiming to document as much as possible so it may be more easily maintained and modified in the future. This is definitely a non-trivial task, but in my experience it almost always pays dividends (such as making any future migrations like this much easier).

As for timing goes, the first major part of the migration (moving these forums) is planned to take place this Friday morning (GMT). There will be at least some brief downtime that day.

tl;dr: The new hosting infrastructure I've planned will hopefully bring some improvements for the trouble, but the end result will still be a doubling of annual project expenses relative to last year.

Step 3
????

Non-Profit
So what are those expenses? That's a question I had to research myself when I started looking at this project. Here is my projection for the next three years:
  • Hosting: $19/month - $228/annual
    • websites, development, master, regional officials
  • Fees: $95/annual
    • domain renewal, forum license maintenance
  • Other: $300 (budgeted at $100/annual for expected minimum lifetime)
    • hardware dedicated to offsite backup storage
  • Total: $35/month - $423/annual
Everything together really adds up. The month to month hosting cost will probably climb a little over time as storage needs increase (our subversion repository for beta testing is 60 GB right now - we're going to recreate it on the new server to save a little since the history isn't critical, but it will still balloon over time).

While we've had many donations over the years, the majority of hosting expenses have been covered by current and former developers. It's a great practical example of how passionate we are for the mod, but I don't see it as a sustainable model. Even if development stopped forever tomorrow and we cut things down to the bare minimum, it would still cost around $200 a year to keep the remaining servers running.

So as you might have seen coming a mile away, here's the pitch. Even if you don't have the time or knowledge to work on the mod directly, you can help with keeping the lights on. By our current estimates the mod's active population is in the neighborhood of 1000 unique players. That means if even only half the community contributed $1 we'd be set for the year!


We're considering some options like setting up a donation meter to make funding levels easily visible, but that won't be coming until after the migration. For now I've written this post to highlight the funding problem and try to make things more transparent. Feel free to let us know if you have any suggestions. And a big thanks to anyone contributing past, present, and future!
 
Last edited:
Posts
299
Likes
184
Wow thanks alot Hexo! Thanks to Spag and all the Dev.Team! I hope the community stand up as an whole to support the game and thank you guys
 

jacklul

Movie Battles II Team Retired
Posts
14
Likes
15
You didn't mention I broke everything up the first time (was it like 5+ years ago?) we were setting everything up

PS. Hi
 
Last edited:

MaceMadunusus

Level Designer
Donator
Movie Battles II Team
Posts
1,888
Likes
2,570
One of the more historically right-leaning developers is alt-left now. Love it. Thanks for the laugh k4far.

69e8276660baa069f3072fc2d2d415cf.png


MB community still goin strong.
 

Spaghetti

Floating in the void
R2D2
Movie Battles II Team Retired
Posts
1,637
Likes
1,633
Just gave you a 25€ allowance, be good boys and don't spend it all at once.
*whistles innocently*

PSA: Both of the displayed donation options go to the same place, just one has forum integration that automatically hands out the gold titles. I've extended those periods for a few people who I think accidentally went with the non-integrated option before the other. Thank you all the same though! We're almost at 60% of the budget for the year, which is a lot more than I was anticipating this quickly.
 
Posts
821
Likes
928
Not making any assumptions but curious as to who was receiving donation money in the past/how it was being spent if, as this thread suggests, Hexodious was the one paying for most of Mb2's assets out of his own pocket?

Again, not pointing any fingers just want to be reassured before myself and many others, I'm sure, start to donate.
 

MaceMadunusus

Level Designer
Donator
Movie Battles II Team
Posts
1,888
Likes
2,570
Not making any assumptions but curious as to who was receiving donation money in the past/how it was being spent if, as this thread suggests, Hexodious was the one paying for most of Mb2's assets out of his own pocket?

Again, not pointing any fingers just want to be reassured before myself and many others, I'm sure, start to donate.

Full details are not entirely ready for the new setup, but we can provide some more details when everything is fully migrated if needed. However I believe, and Spaghetti can correct me if I am wrong on any of this, prior to the last few months of last year the donations went to LoU who was hosting the master server and EU official servers to cover those costs. They went into an account created specifically for MB servers set up to auto-pay & if there were shortfalls any months LoU would cover it while extras rolled over. To my knowledge donations did not even cover the costs of those servers, which is partially why they went down 3 or so times in the Sept.-Nov. period of last year. So yeah, Hex has been paying the majority, if not all of the costs of those particular servers as the donations were not keeping up with even one set of servers. Some other servers at the time were also run by Spaghetti (US Official servers, build servers) and I believe kikili (additional build servers) who covered the costs of those themselves. So things were spread across 3 or 4 people.

After the master server went down several times we cut the EU official servers, and I believe moved the master onto the same box as the US Officials. The donation link was temporarily updated to handle these servers. This was also pretty much the start of us realizing we needed to set up things better than we currently have, as they were set up rapidly as mentioned, so we can be more sustainable in the future. For example, the backup server costing nothing after they are paid for as it a lump sum investment that will only cost like $5/year in electricity (Which isn't included in donation amount listed and is coming out of my pocket because its no big deal) so long as we don't exceed the HDD space and need to expand. The system itself could easily last us 10 years or more.

When we started the whole process of updating our infrastructure the donation account was updated again and turned into a full business account. When done I believe everything should be fed from that account (though some of it done manually, as some of it is up front costs not monthly auto pay). Spaghetti is currently doing the accounting for that as he is the one managing the migration process.
 

Spaghetti

Floating in the void
R2D2
Movie Battles II Team Retired
Posts
1,637
Likes
1,633
This is a good summary. As I mentioned in the OP, the full cost of everything being hosted had never been organized and accounted for before. It's still a little hard to assess the value of donated systems like build machines. Speaking of...

Kikili has committed to donating a server dedicated to development use, which looks like it will get our annual budget down from $534 to about $423 for this year and onward. I've updated everything accordingly. Thank you very much Kikili!
 

Defiant

Nerd
Project Leader
Movie Battles II Team
Code Leader
Posts
1,007
Likes
1,451
Mace's post is entirely correct. In fact, since I've been project lead to the best of my knowledge there wasn't a single month that donations covered Lou's costs alone. A thank you to Lou from the team is missing from Spag's initial post.
 

Caelum

Donator
Posts
24
Likes
53
You're probably not interested in this - I've talked about the possibility with various past MBII teams over the years - but if circumstances have changed, I'd be happy to offer either free web hosting or JKA server hosting through JKA.io if it helps. The only caveat is we can't give you hundreds of gigabytes of disk space for SVN, or more than a few terabytes of monthly bandwidth, so you'd still need a solution for SVN & CDN. Might be able to help take the load off in terms of website/game servers, or eliminate the DDoS issue for you though?

I'm not just saying this to prop myself up, if it's a thing that'd genuinely help at some point (or right now), yell. The resource usage wouldn't really be noticeable for us, we've got JKHub and some 50 JKA servers hosted on the same platform.
 
Last edited:
Top