A Decade Of Camelot

Ten years ago, I began my career in the gaming industry by signing on to help with the imminent launch of Dark Age of Camelot. That game and its team still holds a special place in my heart, though most of the founders from that day have gone their separate ways. Matt Firor, the original producer, asked me to post his recollections of that launch.

(Note that he does make me look like a bit of a doof at one point. I have three things to say in response: 1 – I’m still pretty happy with how we managed to get a working CS front-end and back-end system up and running in about six weeks, 2 – it wasn’t *quite* as doof-tastic as Firor makes out, as I explain in annotations to his tale, 3 – I am in fact a bit of a doof.)

And with that, I give you Matt Firor.

We went into October 9 relatively calm and serene. Vivendi, our distribution partner, had forecast 100,000 sales of the game lifetime, with about 50,000 coming in the first couple of days after launch, and as such, they only “sold in” a very limited number of boxes into the retail channel. We were very comfortable that we could handle those numbers, as we had just had a very successful beta program.

Then, just before launch day, Vivendi got in touch with us and said because customer/retail demand was so high, they were going to release all 100,000 boxes into retail. Fortunately, with delivery times, these extra copies would be delivered to stores a couple of days after the initial 50,000 boxes. This was very helpful to us, as all the boxes were not available on the same day which spread out the “opening day” crush of users over five days. Camelot would go on to sell more than a million boxes in the next couple of years.

Camelot’s official launch day (as in boxes were in the stores) would be October 9 2001, but everyone that had a free account (lots of media, some friends/family, and of course all of us) were able to play starting October 8, as soon as we put our first seven servers online starting around 5:00pm that day.

At the time, Mythic’s offices were located in a townhouse office community near the middle of Fairfax, VA. We were about 30 developers in October 2001, and about 40 customer support. We didn’t have enough space for CS in our original office, so we had to lease space in another building in the same complex, about 50 yards from the developer wing of the company, in a basement space.

Everything was done on a shoestring at that time, so to get internet access over to the CS “center” 50 yards away, Rob Denton, the Development Head of Mythic at the time, and an electrical engineer by training, set up a pair of IR “guns” to make a link between the two spaces. We put one “gun” in a window in the development office, and another in a basement window, pointing up, in the CS center. The link worked very well, and allowed us to share our one Internet line with both spaces. However, because the CS Center was below grade, we had one problem: if a vehicle over a certain height (about 5 feet) parked in a particular parking spot, the link would be broken. We lived in fear those couple of weeks that a delivery truck would park in that spot and cut off Internet access to the CS center. We arranged a quasi-official parking schedule to ensure that an employee car (a short one) be parked in that spot 24/7. The link, fortunately, was never broken.

It's on.

All day on the 9th, we watched as the server numbers grew and grew. I ran the login utility on my laptop all day, just so I could see the population numbers of each server, real time. The population numbers started small around 10:00am on the 9th, and grew slowly but steadily until around 5:00pm, when they exploded. All servers in the space of about an hour after 5:00pm were jammed full – and we had a very large server population setting (about 3,000 players). Even with full servers, everything ran smoothly.

By about 8:00 we were jubilant. Everything was smooth and easy. CS was functioning, and had already responded to many trouble tickets and issues. People were playing, the servers were up.

A group of us formed in Rob’s office, talking and generally basking in the glory of the moment. Each of those 20,000 (max peak players that night) was a paying customer, and each represented significant revenue to us (remember we were very small at the time). It appeared that we finally were going to make money on one of our products. We were giddy with excitement – everything was going awesomely.

Brian Axelson, the 21 year old whiz-kid programmer/designer who had been working for us since he was 16 – responsible for inventing, implementing, and designing Camelot’s combat system, including Combat Styles – was so happy he slammed his fist down on Rob’s desk and said, “Ain’t nothing going to bring this house down!”.

At that moment, all the servers crashed, simultaneously.

We all looked at one another in dread, and sprinted back to our offices, each checking on the part of the game we were responsible for. Everything checked out – nothing seemed wrong. But the servers were down and wouldn’t reboot.

All the programmers were summoned to Rob’s office – I was a fly on the wall – and he walked them through the problem. It wasn’t a code problem, although that wasn’t immediately obvious. It was something keeping the servers from booting and authenticating properly.

After about an hour, the problem was traced to the Customer Support tool – that very day programmer Scott Jennings had made a small modification to the CS tool to take advantage of a database feature buried in MySQL to make database queries work faster. That change did in fact significantly increase the speed at which the tool made queries to the database – but at full load, the index that he built quickly became overloaded, and started to time out and lock out other queries.  (Editor’s note: This change was actually fully tested… with one low-population test server running. Guess what changed on launch day!)  Because the game servers relied on access to the database as well (for player authentication, etc.), they couldn’t keep up with the crush of players logging on and off – and they crashed like the proverbial house of cards. And, because the database was locked up, when they rebooted, they immediately ran into the same problem and crashed again.

The helpful message I added to greet CS reps on launch day.

Once the problem was found, it was very easy to fix, for the moment. The CS tool was modified to not make any of those specific types of queries, the database server was rebooted, the index rebuilt, and everything came up again – this time smoothly and without error – and ran flawlessly until the next afternoon, when we had our first bug-fix patch. Scott had fixed the DB/authentication problem in the meantime (Editor’s note: and said programmer slept three days later), and that functioned properly as well.

In the end, a very smooth launch, but a lesson was definitely learned that when you’re dealing with something as complex as a MMOG launch, you never know what is going to take you down.

There are many more stories to tell about the early days of the service – like how we had to expand servers quickly because of demand, but couldn’t get them delivered from Dell because we had no credit rating. All our purchases up until that point had been made on the spot with no leasing. We had no leasing history, so Dell wouldn’t ship us servers quickly. We were forced to drive to MicroCenter (in Fairfax) and buy a dozen or so desktops, quickly installed Linux, and then drove them (in a pickup truck) to our colocation facility, and stacked them up like firewood in a cage. Those two servers clusters (lovingly called the “gimp servers”) ran for at least a year with no problems, at which point they were swapped out with standard Dell rack-mounted models.

I remember walking into the office one morning towards the end of October. By that time it was obvious we had a smash hit on our hands. Our marketing/sales consultant, Eugene Evans (now the GM of the studio) had a whiteboard near his desk (right by the front door) where he jotted down sales numbers. By October 27 or so, it showed that we were not only the #1 selling PC game for October, but also the #1 selling PC product for that month. Since this was the first boxed retail product Mythic had, I asked him if this success was normal. Eugene, and old industry veteran, looked at me like I was insane and replied, “No, this isn’t typical.” He then broke out in laughter. It seemed so easy at the time: you make a game, put it in a box, and it sells like hotcakes.

Most everyone knows the rest of the story from here – Mythic quickly outgrew its space and in 2002 relocated a few miles away to new mid-rise building, where it grew to take over three floors. A buyout by EA followed in 2006, and the studio is now known as Bioware Mythic. Dark Age of Camelot’s numbers have dwindled down to a fraction of what they were in those heady days of 2001-2003, but it is still up and running, ten years later.


47 thoughts on “A Decade Of Camelot

  1. Gasbandit says:

    To this day I actually still remember DAOC’s launch as being the smoothest I’ve experienced.  That it was done on a shoestring across a parking lot to a basement makes it all the more awesome, and makes me even more grumpy with what passes for quality of service/uptime among “big” companies today. 3 years in DAOC, and I wouldn’t give a one of them back.

    GB, aka “Vortac Verloren,” Guinevere server, Vae Victis

  2. red dot mist says:

    Approximately how large of a team would you need to remake classic DAoC in a more modern engine, with new player and monster art assets?  It seems like a better use of resources than the flop that Warhammer: Wrath of Heroes is destined to be.

  3. Ishmael Scarratt says:

    That was REALLY interesting, I’d say I’d rather read more of this than play most 2 realm MMO’s :).

    Will there be more DAoC posting, I hope so?

  4. Sinij says:

    Very interesting story. With such strong release, what do you think single most significant cause of a decline of DAoC “empire” ? What I am trying to understand is why WoW seems to defy release-grow-decline pattern _every other_ mmorpg experiences? Is “decline” stage preventable?

  5. Polynices says:

    DAoC released the Trials of Atlantis expansion 2 years later which was a fundamentally misconceived product, far worse than anything Blizzard has done to WoW. Decline may have eventually been inevitable for DAoC, but there was nothing inevitable about ToA and how unhappy it made many players.

  6. Marty Brown says:

    Ah Scott. Not a week goes by that I don’t both curse your name and then immediately praise it for what you were able to accomplish. Suffice it to say you were instrumental in the success of DAoC and the Herald was the progenitor of all web-based game stats databases. Kudos sir, and thanks!

  7. Walt Yarbrough says:

    Thanks for the write up, Matt and Scott..  I remember with the official launch, how all of the developers jumped into their ‘play’ accounts on a live server. . .   But none of us had to buy boxes, or patch the game, so we could jump right in and play, in advance of the general public . . .

    Leading to a very suspicious population chart:

    Server all the devs were on: 35 players
    Other Server:  5 players
    Another Server:  5 players
    Final Server:  5 players

    And, of course, as we all were in the same guild, it was just a TINY hint that we were the devs.

    We ran around frantically getting everyone to log out and create alts for the first few hours.

  8. Chilibreath (Former TL) says:

    Well the Trials of Atlantis was certainly the tipping point but the PvE aspect of the game was (and still is?) utterly broken. ToA just exposed it in much greater detail and it just wasn’t any fun for a lot players. (Unless you could artificially inflate your attacking numbers.)

  9. Expfighter says:

    I wasn’t there at launch but started playing in june 2002, and well i lust logged off DAoC a few minutes ago.  Strong after 9 years, Love it still!

    March on DAoC

  10. “why WoW seems to defy release-grow-decline
    pattern _every other_ mmorpg experiences?”

    WoW doesn’t defy that pattern. If you look at the numbers, the game is just entering the decline phase. It’s just that it took six years to attain the plateau, and subsequent decline, when the influx of new players isn’t enough to compensate the old ones giving up.

    EQ had a similar long run growth, close to four years, followed by a long plateau of stable subscription numbers, then a decline leaving only a base of rabid fans. DAoC had a slightly shorter cycle, but only slightly.

    It’s only the recent games who have compressed the expansion-decline-die hard fan stable natural cycle to less than a year…

  11. Great story man, and sounds like you have some very fond memories from the olden-golden days. Congrats on launching something that lasts for 10 years and impacts the whole of PC gaming!

  12. CJ_Grebb says:

    A short memory . . . .

    For launch day itself, what I really remember was how much
    of a breeze the whole thing was for us in the art department (I was the Co-Art
    Director alongside Lance Robertson).

    Like pretty much everyone else in the building, our real “crunch
    time” had happened a month earlier, when we were getting the DAoC Gold Master
    together. Rob Denton was determined to have a miniscule patch on opening day.
    My memory is that he wanted it to be no larger than 50 MB. Others can correct
    me if I’m not remembering that number right. The point is, it was small, and
    very little of it was allocated to new art, so if it was to  be seen in the game, it damn sure needed to be
    on the disc already.

    The upshot is that we had already killed ourselves, and had
    been doing so for months up to Gold Master. I modeled the majority of Jordheim in
    a single all-nighter. All of the artists could tell similar stories, I’d
    wager.  They really put their hearts into it. Considering the tech we had and
    the restrictions we were under, I continue to be incredibly proud of what we
    pulled off.

    So while everyone else was running around like mad people
    making sure that the barn didn’t catch on fire, my memory is that the artists
    were kind of in an amused state, watching all the activity and hoping we’d all
    have jobs in a few months.

    Edit in response to Michael Hartman. Glad you liked Muspelheim but unfortunately I mixed up my “heims” when I originally posted. I modeled the city of Jordheim not the zone of Muspelheim. I don’t want to take any credit that isn’t due to me. I’m not sure who created Musphelheim, but it might have been Walt Yarbrough.

    • Muspelheim was a beautiful zone.

      I remember beating the snot out of those spiders early on – delighting that my Thane’s hammer did bonus damage to them.

      Then we moved in deeper over time to take on the rest of the baddies. Great zone.

  13. slyde says:

    ah, and i was ONE of those first hundred thousdand who purchased the game on launch day.

    I ADORED Daoc back in those early days.  Played it without end (much to the chagrin of my wife). 

    DAOC was the game that finally took me away from UO, which at the time hadnt thought was possible..

  14. Slapshot1189 says:

    To this day all I search for in a game is DAoC with upgrades graphics (low priority) and improved coding/networking.  Instead I get games that only last for a few months and then burn out.  DAoC holds such a cherished memory in the hearts of many gamers… a sequel (prequel ala Dawn of Camelot) seems like a no brainer.  I cannot understand why 10 years later we don’t have one…

  15. Jack Casey says:

    Ruined by buff bots more than anything else IMO. 🙁
    Even not having ToA arties wasn’t as bad as not having access to a buffbot.
    I guess when 50% of the player base is dual boxing it’s a hard call to fix the ‘problem’ of 50% more revenue… Sad.

  16. Reggie says:

    daoc was my first mmo ever. I had been drooling on beta pictures for months. Man did i have a great time in that game. Ah the memories. I’ve been in love with daoc ever since and even though i’ve played many other mmo’s after , ive played mmo’s 10 years straight now, i’ve always returned to daoc. In total i must have returned to daoc about 6 -8 times. No other mmo made me miss it so much as daoc did. I still do miss it alot and last year i returned again for 6 months or so.
    Best game ever really. Nothing still comes close to it. No other game gave me the sence of belonging like daoc does. Other games’ community sux. I cant care less about anything. In daoc i was part of a whole, the realm and it made you feel proud !
    Thanx alot mythic !

  17. Spyke Alexander says:

    If Firor thought the heart stopping aspect of all of the servers going down was bad, he should have been in Darrin and my shoes… 

    Since I was specifically hired to make sure the network/server infrastructure was stable… Let’s just say the pucker factor was cranked up REALLY high at that point.  =)

    Chickens.  Heads.  Cut off.

    That IR link was the bane of our existence, too.  It was the best option that Darrin and I could come up with, at the time, for what little wiggle room we had access to.

  18. icopartners says:

    Matt said I should share that story over here, so here it goes.

    I have my own story about the DAoC launch day – except it is not about the launch in North America, but the European launch. At the time, I was running the team in France that ran DAoC. We had one another MMO under our belt but there was a lot of first for us with this game. This was the first game we launched in a box, the first game we charged a subscription for and the first game we serviced in English and German on top of French.

    That was also a game significantly more sophisticated than our previous title and the server set-up was a big project  in itself…

    To give you some notion on how crazy it was, when DAoC launched in the US, we were in the Beta. We had no one in the team that would speak German yet, and I was the German Community manager (on top of managing the whole project and setting up the teams) – and if you have heard me speak German, I was either saying “Genau” or “hubschrauber” as that’s basically the only 2 words I know – and we had the mother of one of our guy who was a German teacher help us by translating the announcements for the website. Hopefully, by launch we had 2 very good guys who had joined us.

    The game was huge content-wise and we received the localised files from our providers TWO days before the launch.
    So we had very little time to test it, but it worked. We were ready.

    The day comes, players rush to buy the game and log-in.
    And BAM.

    French and German servers keep crashing.
    In loop…

    Soooo… We were in a panic – we didnt have any devs in-house to support us and so we called in Yvette. Yvette is my favourite person from Mythic over any other one. She was the one person in charge of supporting us and fixing anything that would go wrong (on top of doing loads of other things there).
    I get hold of Yvette and let her know about the problem.

    She gets access to server, check the error log and asks me:
    – Thomas, did ANYONE changed the localisation files since 2 days ago?
    – Well, yes. I did. There was a couple of very offensive typos that needed to fix. So I changed them. But I only changed a few characters in each file, I promised.
    – Thomas [with a tone full of patience and understanding], you saved the file with the wrong format. The servers don’t like that. I am goign to fix this for you.
    – …

    In my book, you haven’t really run an MMO game until you have personally crashed the servers… Just avoid to do that with the one in production.

  19. Dragoncroft says:

    Great article and what a fun read. I had just posted yesterday on my guild forums that DAOC-Shrouded Isles remains to this day the best gaming and fun I ever had.

  20. @twitter-46346835:disqus I completely forgot all about that! If I could go back in time, one thing I would have done was try to coordinate better with you all. 🙂

  21. Jörg Koonen says:

    My most memorable DaoC story?

    One day after DAoC had launched in Europe I went to the store to get me my copy. One day after that I logged in (after downloading megatons of patches 😉 ). And one day after that, I was looking for a source of information.

    Sufficient to say that the former daoc-guide.de’s admin made me Mod within a week and that after four years of hard work – working my way up from moderator to page admin, here I am… working in the industry and I wouldn’t want to miss a single second.

    The best moment with DAoC was on that one day, when Paris went dark when Goa was running a game and website maintenance and neither was available for hours. Our forums were flooded with people asking for an update and while my co-admin was on the phone with the team in Paris, he got me the updates which I then posted on forums and website. I was thankful that my boss at that time excused me (he allowed me to take half a day off while I worked on the forums from the office).

    That was the day that I discovered that this was what I wanted to do for a living. 🙂

    Some time later I met Matt Firor at the Games Convention and had a great chat with him. He got me in touch with Sanya and then I started copying her style. (/wave I love you!!! 😉 )

    I’m still around under a different nick. 😀

  22. Walt Yarbrough says:

    [i]In my book, you haven’t really run an MMO game until you have personally crashed the servers..[/i]

    Fortunately, my own crash of the servers – with a monster generator force-spawning guards, was in late Beta

  23. Wulfrique says:

    I have played some great games since DAoC but none have given me that feeling of “This is soooo awesome” like Camelot. I’m sure it helped that it was my first MMO but I tried Everquest and didnt like it so there was something about DAoC that other games didnt have. Happy tenth anniversary everyone. Good to see some familiar names here.

  24. My fondest memories of DAOC were the times when it was so few of us against whole zergs of the oppoents. There were times where we’d hit zergs of 80+ people and wipe them out with just 8 people. TOA and buffbots really gave us what was needed to overcome massive amounts of disorganized people. Even LOTM was great for bringing back that feeling of invincibility vs huge numbers.

    I think my best single memory was the night they released LOTM and my group was playing our mid 8 man team. When we hit the first stairwell camped by a bainshee we realized bainshee cones were doing damage up the stairs and through walls. We immediately decided to log over to our old hibernians and enjoy the evening. Our spiritmaster had never played hib, but snagged a bainshee who we ran down to the stairwell in laby. The rest of the group was simply 5 more bainshees a bard and a druid.  We couldn’t stop raking in RPs all night, we killed hundreds of players dozens of times. By the end of the night the “new” bainshee was level 46 and realm rank 6. He had a collection of hate posts on the IGN forums, but hey, at least we didn’t waste the night in mid on another server getting owned at the stairwell 🙂

  25. Triforcer says:

    I’m glad so many enjoyed DAOC, but I could never get into it.

    I logged in launch week dressed in brown, and next to me there was a brown hill with frogs on it.  Next to me was a stone mushroom with a merchant under it dressed in brown and a sign saying “Dear Hibernian- IOU one realm.  XOXO, Mythic.”  I didn’t even last a week…

    • Cadiva says:

      If you’d changed the word brown for green then you might have worked the irony, Hibernia wasn’t known as the golf course for nothing.

      DAoC was, and always will be, my first love in the world of graphic MMOs, although a text MUD stole my heart in the mid 1990s. I’d be an EverQuest girlie but it wasn’t until DAoC’s RvR that I really found what made the experience of an online graphic game epic. There’s something hugely different in fighting against a computer’s AI (all the EQ raid content) versus fighting against tens of other human brains.

  26. ToeJob says:

    I missed all the fun being hooked on Dragon Realms at the time.  To this day I regret not giving it a try in its prime because of the stories my friends tell.  To this day they still talk about DAoC so the team did something right for sure!

  27. Iain Compton says:

    My favourite DAoC story to this day is one of the roleplay events and its follow up that we ran in Hib. Back in the day we used to produce events for the realms on a roughly two week rotation – so each realm across all the servers would see an event every six weeks or so. These events were scripted one-evening things with custom NPCs, monsters and so forth. Some of the NPCs were just scripted like regular quest-givers while others, where we needed a bit more flexibility or for the NPC to react were reskinned player characters played by the GM team or our small core of volunteers.

    So we had an event in Hibernia, the basic idea was that there was an eeEEeeEvil wizard living in an ice cave who was flooding Hibernia with beasts that didn’t belong there and disrupting the balance. The players were asked to go and sort it out so the Head Druid handed out flying mounts to take the players to the dungeon – which was Tuscaran Glacier (the Mid SI dungeon). We’d closed it to Midgard for the evening and spawned a custom set of monsters inside it instead of the usual generators.

    The players fought their way through the dungeon until they come to the final boss, a massive, animated suit of armour that took a huge beating before going down. As it dies, a Lurikeen (played by me on the Prydwen server) leaps out of it. The players (not unreasonably) ask who I am and why I was hiding inside the armour so I tell them that the wizard has gone and I am his assistant. I had been making some money for myself by selling off the results of his experiments. The players asked me a whole bunch of questions and I lied, contradicted myself and generally showed I was not to be trusted. This was commented on extensively and I was quickly debunked as an incorrigible liar. At one point I claimed that I had baby dragons that I could sell. One player asked if he could buy one to which I said sure, but they hadn’t hatched yet and wouldn’t be ready for about two weeks, however I could take money now and deliver them when they did. The players immediately *formed a line* to give me their money. I charged 2 plat a time for them (and back in the SI era, 2 plat was a big chunk of change).

    The next day we ran a news item with my character on a wanted poster, the story of what had happened went around the forums and the next week we ran a follow-up event on that server where the players had to hunt my Lurikeen down by following clues and defeating some scripted encounters and then placed her on trial. Again there were NPCs who showed up as witnesses for the defence while a volunteer playing a Head Druid formed the players into a jury. All the players I’d taken money from were witnesses for the prosecution. The biggest PvP guild on the server at the time (a guild called Celtic Fist) who generally had no time for roleplay, came down en masse and inducted me into their guild so that I’d benefit from their protection for the proceedings. Witnesses included my mum (who implored the jury to execute me as I’d never been anything but trouble), a Kobold who was my business partner in a weapons trade with Midgard (treason!) and various others designed to destroy my character as much as possible. Inevitably I was fined the amount of money I’d stolen and then banished from Hibernia for ever (Celtic Fist wouldn’t let the jury execute me).

    We ran many other events but that particular event on that particular server was by far the best one and most of it just happened in reaction to the players.

  28. Auta says:

    Wow, so interesting to see the stories
    from behind the glass. (Being an IT guy I feel the pains of trying to
    keep servers up! hehe) I for one started just shortly after launch,
    having came from Everquest DaoC was my second mmo, and by far the one
    I played the longest. I remember the anticipation, I had always been
    enthralled by Arthurian legends, and I had the chance to play in a
    world setting that was truly amazing. There was no doubt in my mind
    what I was playing, I installed the game, plate mail, sword, shield,
    the quintessential knight, I rolled a Briton armsman, and what a trip
    it was.

    I remember prydwen, my first block
    (shields in EQ didn’t function like that, it was a big pet peeve of
    mine), strutting around in my chain mail, I remember fighting with
    paladins over who was the better tank (damn heal chant!), bringing
    the pain in RvR. I remember the sieges, the battles, the way the
    style system worked, catapults, and the zerg fights we would have in
    emain, and bragging about it later on the VNboards.

    My first guild, lasted awhile, my
    second, lasted much longer, I still have friends from that DaoC
    guild, we moved into other MMO’s after catacombs and as the game
    became muddied with expansions, but even to this day we reminisce
    about the old days. There was something about the player base in DaoC
    that was just more… quality… than the new fish in the mmo genre.

    One event that sticks out in my mind
    was when a respected Hibernian player past away, I can’t remember his
    name or what he died from (I think it was cancer or maybe an auto
    accident). But he always wore red, and so a huge group of
    albs/hibs/mids all wore red, everyone, and had a moment of silence at
    the alb mile-gate in emain, no pwnzoring, just honor.

    For all the lumps I laid on that game
    at the time, I wouldn’t trade those years, you don’t find those kinds
    of players anymore, you don’t find that kind of game anymore.

    Thanks for all the adventures and
    memories 🙂

    Auta, 50th Season Briton
    Armsman, Guinevere Shard

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.