Perl productivity has suffered this month with JT Smith's announcement
of The Lacuna Expanse, a web-based space empire strategy game. As with
most of Smith's projects, TLE uses Perl extensively. Perl.com recently
conducted an email interview to explore the world behind the game
world.
You're a successful serial entrepreneur. How'd you get into
Perl?
I started my professional career as an Engineer at a TV station. As the web
started to get popular in the early 90's I started picking up web development
for the TV station, and then eventually went to work at an ISP as a web
developer and system administrator. That's when I first picked up Perl, as it
was already installed on the DEC Unix boxes they were running. I realized how
easy it was to use it to automate a lot of my job (deploying sites, running
backups, collecting statistics, munging logs), and a little web stuff here and
there too (processing forms, writing message boards and polls). Since then I've
used several other languages (PHP, Java, and Ruby mostly), but I always come
back to Perl because it solves the most problems for me with the littlest
amount of fuss.
What's your business background?
I have no formal business training, but I've worked at lots of companies
big and small, and either started or helped start about a dozen companies
now, four of which I still own. So I've really picked up a lot of my
business expertise through trial and error, and through watching the
successes and failures of other businesses.
Some people follow sports and can quote you the scores and statistics of
their favorite teams. For me, I prefer to watch businesses and business
leaders. And when I read for entertainment, it's almost never fiction.
Instead I like to read about things that can give me ideas to apply. For
instance, I just finished "The Viral Loop," which covers viral marketing
history from Tupperware through Facebook. I know all this sounds pretty
nerdy/geeky/dorky, but so be it.
How did you decide to do a browser-based game?
Actually long before I built WebGUI, in the
CGI era, I built one of the very first web-based RPG systems. It was called
Survival of the Fittest. And back about that time I had the idea for The Lacuna
Expanse (it was called Star Games back then), but the technology wasn't there
to pull off what I really wanted to do.
Then last year (released July 14, 2009) I built a new business called The Game Crafter. It is a web to
print company, where people design board games and card games using their
web browser (plus some offline image editing) and when they're done, they
can order a copy for themselves, or put it up in our online store to sell
to potential customers. It's sort of like Lulu or CafePress, but for traditional board and
card games. Here we are just over a year later and that business has really
taken off, with over 1,500 people making custom games, and 70% of customers
returning for more than one order. I should mention that TGC is built with
100% pure Perl as well.
About the time that The Game Crafter launched, another business that I
had created four years earlier started actually making some good money.
That business is CMS Matrix, and yes
it's 100% pure Perl as well.
After about 6 months of seeing how well The Game Crafter and CMS Matrix
were doing, and knowing that I had a solid team in place to keep WebGUI
marching forward, my business partners and I decided we should take a
chance with yet another business. But this time we decided we wanted to
tackle something much more ambitious and risky.
One of my business partners reminded me of the Star Games idea. And
there's hardly anything more risky than making a video game. It has a large
up front cost of both time and money, and video games pretty much either
make a lot of money, or none at all. There's not much of a middle of the
road. With Star Games as our foundation, we started designing game
mechanics. We didn't want to build yet another war game (too many of those)
so we settled on espionage as our conflict mechanism. And until WoW and The
Sims came around, there was one game that dominated the landscape as far as
revenue goes, SimCity. So we knew the game had to have a city building
element. And everything else was stuff we either made up, or ideas we
borrowed from our favorite games.
What did you have to invent and what did you reuse?
Luckily CPAN came to the rescue as it has on basically every Perl
project I've ever tackled. So I was able to not have to reinvent the wheel
on basically any foundational level. Here's the list of Perl modules I
used:
When I first started development I was convinced that to be massively
parallel I was going to have to go with an async server like Coro or POE,
and a NoSQL database.
I quickly realized that writing this system to be completely async was
going to be a nightmare that would take more than double the time. Part of
the problem was that while I was familiar with developing async
applications, I had only done it on a small scale in the past. The other
problem was that I kept running into modules I wanted to use that weren't
async compatible. Ultimately I ditched the idea of going async within the
first month.
Unfortunately I wasn't so quick to ditch the idea of NoSQL. I started with
MongoDB and CouchDB, but had trouble compiling them with the Perl bindings. I
planned on hosting on Amazon at that
point, so I decided to give SimpleDB a go. The downside there was that there
were no decent Perl bindings for SimpleDB, that weren't entirely bare bones. So
with that I created SimpleDB::Class
(based loosely on DBIx::Class). The
module works great. Unfortunately SimpleDB doesn't. It's super slow. So four
months into development, with a whimper, I had to ditch my beloved SimpleDB::Class
module. I'm glad I did. Development has been much faster and easier since then,
and a good amount of thanks goes to DBIx::Class for
that.
WWW::Facebook::API
has been largely abandoned by its author. He told me he doesn't have time to
maintain it anymore. And I was having a hard time getting it to work anyway. As
luck would have it Facebook just announced their Graph API, so I decided to
take on that project, and build a Perl wrapper around it. And Facebook::Graph
was born. This enabled me to allow Facebook users to Single Sign On into the
web site, the game, and also interact with their accounts.
About the only other non-game piece that I had to invent of any consequence
was JSON::RPC::Dispatcher,
which is a Plack enabled web service generator. There are some other JSON-RPC
modules on CPAN, but for one reason or another I found them all completely
insufficient. Mostly because of one or more of four reasons:
- It didn't support JSON-RPC 2.0.
- Its documentation was so poor that I couldn't make it work.
- It made me write a ton of code to simply expose a web service.
- It wasn't PSGI/Plack compatible.
With JSON::RPC::Dispatcher,
I can expose object-oriented code as web services with a single line of
code.
I'm not very happy with the Perl modules that are out there for S3 right
now. Right now we're using a combination of SOAP::Amazon::S3,
and Net::Amazon::S3,
and neither are particularly good, at least for our purposes. They both work,
but only for fairly basic purposes. Sometime in the near future I'll either
take on a massive overhaul of one of those modules, or write my own from
scratch. Which remains to be seen depending on how open the authors of those
modules are to patches.
What did you need from SimpleDB besides more speed?
What I was hoping I'd get out of SimpleDB was three things: massive
parallelism, hierarchical data structure storage, and schema-less storage
to make my upgrade path easier. It provided all of those things.
What I hadn't anticipated was all the limitations it would place on me.
Speed was just the nail in the coffin. It also puts pretty harsh limits on
the amount of data per record, the amount of data returned per result set,
and the complexity of queries. In addition, like most NoSQL databases it's
eventually consistent, which provides its own host of problems. I had
worked my way around pretty much all that, and then finally hit the
performance bottleneck.
At that point I knew I had to make a change, because I wouldn't be able
to make up the difference in parallelism. For example, in order to process
functions on a building, I would need planet data, and empire data in
addition to the building data. But I wouldn't know what empire or planet to
fetch until I fetched the building, which meant I'd have to do serial
processing. And I couldn't cache all the data for the empire and the planet
in the building (or vice versa) because of the limits on the amount of data
allowed to be stored per record. My two options were: 1) Bring everything
forward into memcached, which has its own problems because I'd have to
create an indexing system; 2) Move to a relational database.
When did you start using Plack?
I first heard about Plack late last year when one of the contributors to
WebGUI did an initial test implementation (PlebGUI)
to see if we could use it in WebGUI. After seeing how cool it was I knew that
WebGUI 8 had to have it, and all my future projects would also use it.
What's been the biggest benefit for you?
The benefits are so huge that they are hard to enumerate, so I'll pick
the top 3 or 4. For the Lacuna Expanse, the main benefit was ease of
development. There were no longer any hoops to jump through (the mod_perl
landscape can be tricky to navigate). Sometime later Starman came out and
that gave me an instant boost to performance, which was also nice. For
WebGUI the middleware components were instrumental. They allowed us to
eliminate a lot of code we previously had to write ourselves rather than
using shared libraries. In addition, just switching from mod_perl 2 to
Starman (no other changes), gave us a 300% performance boost.
You seem to be comfortable using a lot of new technologies with a
reasonable amount of traffic. How do you see the risks and the
rewards?
First and foremost Miyagawa, who wrote Starman and much of Plack, is my
personal Perl hero. It seems like everything he writes is absolute gold. So
the fact that he wrote it adds some confidence.
But you're right, in general I'm not averse to using new technologies.
The problem with "tried and true" is that it's often "old and stale". So
from my perspective, there are just as many risks choosing proven
technologies as there are new ones. That doesn't mean you can blindly adopt
new technologies, but you should be on the lookout for them. The rest of
the risk/reward decision comes from my business experience: Change is
inevitable. If you try something and it doesn't work out, so what? Sure
it's going to cost you some time/money, but maintaining antiquated systems
costs a lot of money too. These days the pace of technology moves too
quickly to rest on tried and true alone.
Are you comfortable with the risk that you'll run into maturity
problems and can patch them or work around them, or do you think that
despite their relative youth, they're very capable for your
purposes?
Here's the thing. When you're running a technology based business, the
only thing you can plan for is that things will change.
Let's use scalability as an example. If you try to build a system that
will infinitely scale before you have any users then you'll spend a lot of
time and money on something that may never get any users. At the same time,
if you put no time into planning for some amount of scaling, then you won't
have enough breathing room to keep your users happy while you refactor.
Likewise you can't anticipate all the features you'll ever need, because
user's desires are hard to predict. And because of this, at some point
you'll likely make a fundamental flaw in your architecture that will
require at least a partial rewrite of your software. This is very much a
business decision. Most developers I know cry when I say that, because most
believe that it's both possible and desirable to reach
design/implementation nirvana. The fact is that users don't care if your
APIs are perfect. They care if your software does what they need it to do.
From a business perspective it's often more profitable to build something
quickly and then continually refactor or even rewrite it to match
demand.
I say all of this to make the point that if a particular new technology
doesn't work out like we expected it to, then we'll simply replace it in
the next iteration. If you go into the project with that mentality you'll
likely be more successful.
What's the basic architecture of The Lacuna Expanse?
The basic software architecture looks like:
Basically per server configurable game rules go into various Config::JSON config
files. DBIx::Class and
MySQL handle all of the game data storage and querying. Memcached sits off to
the side and handles lock contentions, limit contentions, session management,
and other server coordination communication. Unfortunately, not much can
actually be cached due to the dynamic nature of the game, unless I was willing
to cache basically everything, which I'm not. And all the static stuff, like
images, JavaScript, and CSS files get served up from CloudFront. We also push
our RSS feeds and other semi-static game content out to S3.
The game engine itself is basically an MVC setup built with Moose. DBIx::Class and Config::JSON act as
the model. Some custom Moose objects tied to Memcached::libmemcached
act as the controller handling session management, privileges, etc. And JSON::RPC::Dispatcher
acts as the view.
The basic server architecture looks like this:
Any of the server nodes can be set up in either a clustered or load
balanced formation to handle traffic growth.
And finally we use Github as our
deployment system. We use its service hooks feature to trigger pushing new
content to S3/Cloudfront and code to the game servers.
How many people are working on this?
Six, plus a bunch of play testers. One artist named Ryan Knope; plus a part time helper,
Keegan Runde, who is the son of one of the other developers. One on iPhone
development, named Kevin Runde. Two on Web Client development John Rozeske
and Graham Knop. Myself on server development. And myself and my business
partner Jamie Vrbsky on game mechanics development.
We started development in January 2010, and officially launched the game
on October 4, 2010. Now that we've launched, I've brought in one of my
other business partners, Tavis Parker, to help out with marketing the game.
And we're still pushing forward on new releases. We hope to have our first
expansion for the game, called "Enemy Within", out sometime in Q1 2011.
How do you manage your development process?
We're very loose on management.
We basically have a strategy meeting every 2 weeks at a local pub, where
we discuss whatever needs to be discussed in person. Beyond that we have a
play testers mailing list, a developers mailing list, and a defect tracking
system that we use internally. And other than that communicate through
Skype and email.
We manage all of our code and content through various public and private
github repositories. We share documents and art mockups using Dropbox.
I publish all the JSON-RPC APIs out using POD (nicely formatted using Pod::Simple::HTML)
to our play testers server, which is what the client guys develop against. And
then ultimately once vetted and implemented by our client guys, the APIs are
pushed out to the public server here at http://us1.lacunaexpanse.com/api/.
What little project management and coordination we need is handled by me
emailing back and forth.
How often are your releases? What's the breakdown between bugfixes
and new development?
For the Lacuna Expanse we're doing releases about 4 or 5 times a week. 1
or 2 of them contain some new features, and the rest are bug fixes.
However, TLE is very new. In the beginning it's very important to react
quickly to your users needs because they often find bugs you didn't, or
have feature ideas that are almost fundamental after you hear them, but you
never thought of them during the development process. By the end of the
year our development cycle will slow down quite a bit, probably to once per
week.
For WebGUI we release approximately once per week, and those releases
are primarily bug fixes. We generally do about 2 major releases per year
that are primarily new features.
For The Game Crafter we've stopped doing releases, except for the
occasional bug fix because we're coming into the holiday season. Starting
in January we're going to get going on a complete rewrite (about a six
month process), which will quadruple our feature set, give us about a 700%
performance gain, and allow us to scale with the growing demands our
customers are placing on us.
Recruit existing Perl developers in your area, work with people
you've worked with before, or hire good people and train them?
All of the above. When you're looking to hire someone you should hire
the best person you can afford to hire. In our case this means we've
decided to design our businesses around telecommuting. We still maintain a
small office, and still hire locally when we can, and we even provide
incentives for our employees to move to Madison if they so desire, but we
never throw out a resume based upon location, what schools they attended,
or whether or not they've happened to work with the particular modules and
technologies we're working with.
I keep an eye on one of the alliances in the game populated by a lot
of well- known Perl developers, and they seem to be pushing the limits of
the public API. I know you made this API public for a reason (and increased
the call limit)--but do you foresee an endgame where the best client
automation wins, or do you expect that the game strategy will be malleable
such that clever players have an edge over automation?
Automation has its advantages certainly. It's great for getting the
mundane crap out of the way.
Most games spend a lot of time and effort doing everything they can to
prevent people from automating their game. The trouble is that you end up
wasting a lot of effort trying to stop smart people from being smart. If
they really want to automate something they will find a way around your
restrictions. It's a never ending arms race.
In our case we decided to embrace these people. Better and better tools
will come along and ultimately that means these people are adding features
to our game that we didn't have to write. Because eventually the tools will
get simple enough that your average Joe can run them.
As far as the game is concerned it doesn't make a bit of difference that
you can use a tool to push a button in the game, rather than pushing the
button yourself. You still have to follow the same rules. It takes a
certain amount of time to happen, you have to spend a certain amount of
resources, etc. When it comes right down to it, someone still has to make
all of the important decisions, and that's not likely going to be a tool
anytime soon. You have to decide what buildings to upgrade in what order,
what ships to build, who to attack, how to defend, etc. And once the next
expansion comes out, you'll have to work with your team mates to build a
space station, enact laws, and defend your federation of planets. It will
be very much a social endeavor.
