Dev Diary - Hosting Migration

Discussion in 'Developer Diaries' started by Spaghetti, Mar 13, 2019.

  1. Spaghetti

    Spaghetti cable wrangler Donator Movie Battles II Team Retired

    Posts:
    801
    Likes Received:
    772
    [​IMG]

    A Brief History
    About three and a half years ago, after a rather disastrous loss of hosting, the active team and many retirees pulled together to rebuild from the ashes. For those around at the time, it's fairly well known that LoU did all of the backend work setting up the website, forums, and development infrastructure, among other things. Less well known is that our main hosting and beta testing servers were generously provided by Hexodious. Alas, all good things come to an end...

    The team was given notice at the end of last year that this arrangement would need to come to an end soon, so we have been assessing our options. While the timeline for completing the move is undefined (no need to panic over an encore of 2015), the consensus of the team is to try and get it done ASAP to relieve Hexodious of the ongoing responsibility.

    And of course we would like to formally, and publicly send our enormous gratitude to Hexodious for what has been his large personal financial contribution over the past few years. We as developers and players are forever in your debt.

    The Future
    While I am retired from development (and not planning to change that status), I've still kept tabs on goings on with the team and done a few things like recreating the web serverlist. Since the start of the year I've been sizing up hosting options and mentally spitballing the problem. Last month I volunteered to take on the task of migrating everything.

    As already mentioned, this time around we have a bit of time to consider infrastructure and making improvements. While the current hosting has served us well over the years, its main weakness has been the lack of network capacity at release time. Any large updates (> 1 GB of files) usually result in very slow download speeds on release day up until enough of the new assets are cached with the CDN. While Defiant's refactoring of launcher downloading has greatly helped with asset corruption associated with that congestion, the update speed is still a problem. This means we want the new host to have both better upload speed and enough data transfer to avoid overage fees or throttling.

    Another consideration is security. Historically it's not uncommon for official and community servers and services to come under DDoS attacks. As disappointing as that is, it's still a reality that demands planning for. This constrains us to only picking among seasoned hosting providers (won't fold the moment an attack starts) and carefully planning infrastructure to help compartmentalize the effects. All of this ultimately increases hosting costs.

    Lastly, since we are essentially refactoring most of the infrastructure anyway, I am aiming to document as much as possible so it may be more easily maintained and modified in the future. This is definitely a non-trivial task, but in my experience it almost always pays dividends (such as making any future migrations like this much easier).

    As for timing goes, the first major part of the migration (moving these forums) is planned to take place this Friday morning (GMT). There will be at least some brief downtime that day.

    tl;dr: The new hosting infrastructure I've planned will hopefully bring some improvements for the trouble, but the end result will still be a doubling of annual project expenses relative to last year.

    Step 3
    ????

    Non-Profit
    So what are those expenses? That's a question I had to research myself when I started looking at this project. Here is my projection for the next three years:
    • Hosting: $19/month - $228/annual
      • websites, development, master, regional officials
    • Fees: $95/annual
      • domain renewal, forum license maintenance
    • Other: $300 (budgeted at $100/annual for expected minimum lifetime)
      • hardware dedicated to offsite backup storage
    • Total: $35/month - $423/annual
    Everything together really adds up. The month to month hosting cost will probably climb a little over time as storage needs increase (our subversion repository for beta testing is 60 GB right now - we're going to recreate it on the new server to save a little since the history isn't critical, but it will still balloon over time).

    While we've had many donations over the years, the majority of hosting expenses have been covered by current and former developers. It's a great practical example of how passionate we are for the mod, but I don't see it as a sustainable model. Even if development stopped forever tomorrow and we cut things down to the bare minimum, it would still cost around $200 a year to keep the remaining servers running.

    So as you might have seen coming a mile away, here's the pitch. Even if you don't have the time or knowledge to work on the mod directly, you can help with keeping the lights on. By our current estimates the mod's active population is in the neighborhood of 1000 unique players. That means if even only half the community contributed $1 we'd be set for the year!


    We're considering some options like setting up a donation meter to make funding levels easily visible, but that won't be coming until after the migration. For now I've written this post to highlight the funding problem and try to make things more transparent. Feel free to let us know if you have any suggestions. And a big thanks to anyone contributing past, present, and future!
     
    Last edited: Mar 16, 2019
    W!zard, Gargos, Garblargh and 9 others like this.
  2. GistBattles2

    GistBattles2

    Posts:
    155
    Likes Received:
    40
    first (u mad bro?)
     
  3. Spaghetti

    Spaghetti cable wrangler Donator Movie Battles II Team Retired

    Posts:
    801
    Likes Received:
    772
     
    Townsend likes this.
  4. GistBattles2

    GistBattles2

    Posts:
    155
    Likes Received:
    40
    Also have u started a gofundme?
     
  5. |CB|Lyrion

    |CB|Lyrion Internal Beta Team

    Posts:
    287
    Likes Received:
    173
    Wow thanks alot Hexo! Thanks to Spag and all the Dev.Team! I hope the community stand up as an whole to support the game and thank you guys
     
    Tylenol likes this.
  6. 2cwldys

    2cwldys Internal Beta Team

    Posts:
    70
    Likes Received:
    45
    Thank you for everything you have done MBII Dev Team, Spag and Hexodious! You all have provided a unique experience and a wonderful modification for such a legendary game.
     
  7. Puppytine

    Puppytine Slayed dreamer

    Posts:
    2,050
    Likes Received:
    1,310
  8. WaceMindu

    WaceMindu

    Posts:
    48
    Likes Received:
    64
    Thanks Hexodious, Spaghetti and everyone involved. Gonna pitch in ;)
     
    Spaghetti likes this.
  9. jacklul

    jacklul Movie Battles II Team Retired

    Posts:
    13
    Likes Received:
    9
    You didn't mention I broke everything up the first time (was it like 5+ years ago?) we were setting everything up

    PS. Hi
     
    Last edited: Mar 14, 2019
  10. Spaghetti

    Spaghetti cable wrangler Donator Movie Battles II Team Retired

    Posts:
    801
    Likes Received:
    772
    *waves*

    Forums are now running on the new host. I've also added a funding meter on the donation page which I'll update periodically until a better solution is in place. Thank you very much to everyone who's chipped in so far!
     
    Helix and MaceMadunusus like this.
  11. SK55555555555555555

    SK55555555555555555 Donator Moderator Internal Beta Team

    Posts:
    360
    Likes Received:
    478
    Just gave you a 25€ allowance, be good boys and don't spend it all at once.
     
  12. MaceMadunusus

    MaceMadunusus Level Designer Movie Battles II Team

    Posts:
    1,311
    Likes Received:
    1,637
    One of the more historically right-leaning developers is alt-left now. Love it. Thanks for the laugh k4far.

    [​IMG]

    MB community still goin strong.
     
  13. Spaghetti

    Spaghetti cable wrangler Donator Movie Battles II Team Retired

    Posts:
    801
    Likes Received:
    772
    *whistles innocently*

    PSA: Both of the displayed donation options go to the same place, just one has forum integration that automatically hands out the gold titles. I've extended those periods for a few people who I think accidentally went with the non-integrated option before the other. Thank you all the same though! We're almost at 60% of the budget for the year, which is a lot more than I was anticipating this quickly.
     
  14. Townsend

    Townsend

    Posts:
    120
    Likes Received:
    98
    saying it like it's the end of the mod
     
  15. noel

    noel Internal Beta Team

    Posts:
    609
    Likes Received:
    601
    Not making any assumptions but curious as to who was receiving donation money in the past/how it was being spent if, as this thread suggests, Hexodious was the one paying for most of Mb2's assets out of his own pocket?

    Again, not pointing any fingers just want to be reassured before myself and many others, I'm sure, start to donate.
     
  16. MaceMadunusus

    MaceMadunusus Level Designer Movie Battles II Team

    Posts:
    1,311
    Likes Received:
    1,637
    Full details are not entirely ready for the new setup, but we can provide some more details when everything is fully migrated if needed. However I believe, and Spaghetti can correct me if I am wrong on any of this, prior to the last few months of last year the donations went to LoU who was hosting the master server and EU official servers to cover those costs. They went into an account created specifically for MB servers set up to auto-pay & if there were shortfalls any months LoU would cover it while extras rolled over. To my knowledge donations did not even cover the costs of those servers, which is partially why they went down 3 or so times in the Sept.-Nov. period of last year. So yeah, Hex has been paying the majority, if not all of the costs of those particular servers as the donations were not keeping up with even one set of servers. Some other servers at the time were also run by Spaghetti (US Official servers, build servers) and I believe kikili (additional build servers) who covered the costs of those themselves. So things were spread across 3 or 4 people.

    After the master server went down several times we cut the EU official servers, and I believe moved the master onto the same box as the US Officials. The donation link was temporarily updated to handle these servers. This was also pretty much the start of us realizing we needed to set up things better than we currently have, as they were set up rapidly as mentioned, so we can be more sustainable in the future. For example, the backup server costing nothing after they are paid for as it a lump sum investment that will only cost like $5/year in electricity (Which isn't included in donation amount listed and is coming out of my pocket because its no big deal) so long as we don't exceed the HDD space and need to expand. The system itself could easily last us 10 years or more.

    When we started the whole process of updating our infrastructure the donation account was updated again and turned into a full business account. When done I believe everything should be fed from that account (though some of it done manually, as some of it is up front costs not monthly auto pay). Spaghetti is currently doing the accounting for that as he is the one managing the migration process.
     
  17. Spaghetti

    Spaghetti cable wrangler Donator Movie Battles II Team Retired

    Posts:
    801
    Likes Received:
    772
    This is a good summary. As I mentioned in the OP, the full cost of everything being hosted had never been organized and accounted for before. It's still a little hard to assess the value of donated systems like build machines. Speaking of...

    Kikili has committed to donating a server dedicated to development use, which looks like it will get our annual budget down from $534 to about $423 for this year and onward. I've updated everything accordingly. Thank you very much Kikili!
     
    Black Dynamite and Townsend like this.
  18. Defiant

    Defiant Project Leader Movie Battles II Team Code Leader

    Posts:
    604
    Likes Received:
    726
    Mace's post is entirely correct. In fact, since I've been project lead to the best of my knowledge there wasn't a single month that donations covered Lou's costs alone. A thank you to Lou from the team is missing from Spag's initial post.
     
    Townsend and noel like this.
  19. Jaaros

    Jaaros

    Posts:
    22
    Likes Received:
    6
    :) Nfo webhosting is $10 a month so you know.
     
    Last edited: Mar 17, 2019
  20. Caelum

    Caelum Donator

    Posts:
    19
    Likes Received:
    39
    You're probably not interested in this - I've talked about the possibility with various past MBII teams over the years - but if circumstances have changed, I'd be happy to offer either free web hosting or JKA server hosting through JKA.io if it helps. The only caveat is we can't give you hundreds of gigabytes of disk space for SVN, or more than a few terabytes of monthly bandwidth, so you'd still need a solution for SVN & CDN. Might be able to help take the load off in terms of website/game servers, or eliminate the DDoS issue for you though?

    I'm not just saying this to prop myself up, if it's a thing that'd genuinely help at some point (or right now), yell. The resource usage wouldn't really be noticeable for us, we've got JKHub and some 50 JKA servers hosted on the same platform.
     
    Last edited: Mar 17, 2019
    W!zard, jacklul, Gargos and 8 others like this.

Share This Page