Author Topic: Organization and Workflows  (Read 13933 times)

Fenrir

  • Rogueliker
  • ***
  • Posts: 473
  • Karma: +1/-2
  • The Monstrous Wolf
    • View Profile
Organization and Workflows
« on: October 21, 2010, 06:23:44 PM »
This is a topic that I don't see much talk about in game development circles, perhaps because I don't look in the right places.

What I want to know is: what organizational steps do you use to take a game from concept to release? How much planning goes into what you do? To you get everything written out on paper? or do you just sit down and start writing code?

I'm not necessarily talking about the structure of your code; I'm as much curious about when you program and what you program in what order. How do you decide what to work on next? How do you decide when to release?

Describe the way your average "programming day" progresses.

linux_junkie

  • Newcomer
  • Posts: 28
  • Karma: +0/-0
    • View Profile
    • Email
Re: Organization and Workflows
« Reply #1 on: October 21, 2010, 06:54:01 PM »
Well, I know that a lot of programmers hate planning, but I'm a bit of an excessive planner myself.  I tend to sketch out the general concept of my game in a notebook (the old pen and paper method).  Then I figure out which systems will need to be coded, and I try to arrange them in the most logical order of implementation, with the goal of being able to have testable code at all times.

For my recent 4DRL (which only took 2 days), I first laid out my exact vision of what I wanted to accomplish.  Then I started implementing it, in this general order:

* Basic game loop + rendering + updating
* Map generator
* Player movement
* Sound effects and music
* Basic GUI + status bars + message buffer
* Primitive AI for zombies
* Guns and ammo
* Powerups
* Barricading system
* Basic AI for allies
* Zombie spawning in waves
* Scoring system
* Splash screen + game over screen
* Help menu

And that's basically it.  As for the decision of when to release, I did it right away, since it was an nDRL challenge.  The game was slow, and kinda buggy, but that can't be helped in such a tight timeframe.

For my main game, I'm waiting until it's relatively stable and bug-free before releasing an alpha.  Unless it's an nDRL type of game, there's no good reason to release an early, unfinished, buggy project.  It'll just turn off potential players, who may not be willing to give it another shot when the game is finished.

Bear

  • Rogueliker
  • ***
  • Posts: 308
  • Karma: +0/-0
    • View Profile
Re: Organization and Workflows
« Reply #2 on: October 22, 2010, 04:28:48 AM »
I start by brainstorming ideas for the game content and functionality. These go in text files labeled "items", "monsters", "dungeon/maps", "timing", "save/restore", "skills", "abilities", "magic", "interface", etc. depending on general category.  To the extent I reject ideas at this point, I reject them for being bad game design, not for failure to match a preconceived code design.

Then I brainstorm ideas for implementation/code design, and "weed out" design ideas that don't allow the kinds of content and functionality I want.  There's some back-and-forth where I re-imagine game ideas developed in different ways to allow their implementation under the design I'm considering, but if an implementation idea would require several content ideas to have different observable effects as opposed to different underlying structure, I generally scrap that implementation idea.  Anyway, I keep track of various approaches and ideas using another text file, initially named "design." 

Implementation ideas are categorized according to "completion" in this file.  Any design element that allows all the stuff I want to do, and gives me ways to implement any support I can imagine some other part of the design requiring, becomes "Final" or accepted.  At that point I delete any other ideas I've been considering for it.  Any design element that I'm confident I can do in some standard way when I get to it, because the requirements for dozens of standard ways to do it are satisfied, and which doesn't itself need to provide anything peculiar or hard to support other parts of the code, becomes "don't care," and I don't expend any immediate brainpower on it because planning it can wait for the day I do it.  It just goes into the design file with a one-liner that says something like "Implement Random Number Generator." The overall design for the software becomes viable in my head when all the subproblems I can think of are in one of these two categories.  The third category can best be termed "no solution yet" and, as such, is the main focus of the planning effort.  The design entry for such elements will usually outline a half-dozen or so different proposals, characterizing each for efficiency and what support it requires from other design elements, and mention some number of rejected ideas and the grounds on which each was rejected.

Once the "design" file is a viable software design, I rename it to "changelog", write the word "TODO" in front of the description of each major subsystem, pull in a small amount of stuff from the content files (monsters, magic, items, etc) as the line in the sand for content for the first release, and write "TODO" in front of each of those.  Then I put a row of asterisks under the TODO list, make a C file with a main() routine and a version number 0, and start.

At that point, it's just the long slow march or the "programmer's twelve-step therapy."  This is essentially:

1) increment the internal version number
2) pick something from the TODO list.
3) If I can implement it in one or a few sittings, do so. 
4) Otherwise, split it up into TODO subtasks and implement one testable subtask.
5) Comment the hell out of the code, because it can be weeks for me between coding sessions.
6) Write testing framework for the new feature if necessary. 
7) Sprinkle lots of asserts around to make sure it's doing what I think it's doing.
8) Test and Debug, focusing on new stuff.  Repeat tests from previous testing frameworks for old stuff.
9) Make sure comments explaining it are still accurate after debugging.
10) Add to the changelog a description of what's new in this version number, right below
the line of asterisks (so TODO lines are above and version descriptions are below).
11) Save the code on a backup system by its internal version number.
12) Goto 1.

A lot of the TODO's expand into subtasks, which recursively expand into sub-subtasks, etc.  A fair number of new TODO's will become evident when you're actually doing the work or trying to test and debug it. I discovered a need for a facility for emitting debug messages to a file, for example, while debugging the curses display.  A lot of TODO's, once complete, trigger other TODO's.  For example, completing the implementation of monsters meant adding another TODO for "Modder's guide needs a description of how to add a new monster."  This can be discouraging because the TODO list never seems to get much shorter, but is actually okay.  If you ever get discouraged about the length of the TODO list not changing, you can look at your versions list and see how much change has happened.  Some things provide a natural testing framework for others, and should be implemented first.  For example, implementing TODO's about a map display naturally came before implementing TODO's about character movement commands.

The first release is ready when I have no more TODO's.  At that point I'll pull in more "content" TODO's from the idea files and start on the next release.  Sometime later, more TODO's should be apparent from bug reports from the first release.  So, I'll start with a new TODO list, repeat the programmer's twelve-step therapy program until the TODO list is empty again, and then I'll have the next release.

Bear

Krice

  • (Banned)
  • Rogueliker
  • ***
  • Posts: 2316
  • Karma: +0/-2
    • View Profile
    • Email
Re: Organization and Workflows
« Reply #3 on: October 22, 2010, 08:37:53 AM »
How much planning goes into what you do?

I try to plan more these days. I think it's hard to plan technical details (especially if you don't yet know how to implement them) but the game content can be always planned. With my recent project Teemu I'm still using micro-planning by inventing a feature I want and then implementing it, sometimes splitting the tasks involved in that feature. I don't have a big list of features for the next version, because I have learned that planning too much can be frustrating. So, in a way Teemu is progressing like a true roguelike:)

For the next roguelike I try to create a complete gameplay content plan, but I guess I don't need to do that much technical plans if I'm going to use Teemu's engine where stuff is more or less ready. Maybe some plans for town level theme generator.

linux_junkie

  • Newcomer
  • Posts: 28
  • Karma: +0/-0
    • View Profile
    • Email
Re: Organization and Workflows
« Reply #4 on: October 22, 2010, 02:40:20 PM »
...
Bear

Seconding this.  I forgot to mention the extreme values of TODO and CHANGELOG files.  They singlehandedly push me forward.  I find something very pleasing in watching both lists grow and grow, until finally the TODO list starts to shrink.  Charting your own progress is essential to motivation.

Fenrir

  • Rogueliker
  • ***
  • Posts: 473
  • Karma: +1/-2
  • The Monstrous Wolf
    • View Profile
Re: Organization and Workflows
« Reply #5 on: October 22, 2010, 05:16:00 PM »
Thanks for the detailed replies!

Bear and linux_junkie, I may well adopt this means of doing things, as I fear I'm dreadful at improvising, and, if I don't know what the "next step" is, the odds of my doing anything drops dramatically. is this a "standard" way of going about things that you learned somewhere? or did you develop this way of doing things on your own?

Krice, I figured you were the sort to "shoot from the hip." Just out of curiosity, why do you find planning frustrating?

linux_junkie

  • Newcomer
  • Posts: 28
  • Karma: +0/-0
    • View Profile
    • Email
Re: Organization and Workflows
« Reply #6 on: October 22, 2010, 05:31:08 PM »
Thanks for the detailed replies!

Bear and linux_junkie, I may well adopt this means of doing things, as I fear I'm dreadful at improvising, and, if I don't know what the "next step" is, the odds of my doing anything drops dramatically. is this a "standard" way of going about things that you learned somewhere? or did you develop this way of doing things on your own?

Krice, I figured you were the sort to "shoot from the hip." Just out of curiosity, why do you find planning frustrating?

I don't think there's any set way of doing things, but my general method is similar to Bear's.  I draft up all of my main game modules (map generator, player input, AI, items, etc.), and put them in a TODO list.  Then I break things down further, until I reach the most basic primitives.  An example for breaking down the map generator might be:

* Map Generator
- Build Room
- Build Tunnel
- Connect Room/Tunnel
- Place Doors
- Place Stairs
- Populate with monsters/items

Then I start implementing them, starting with a blank core.  For example, my map generator might implement the basic dungeon building algorithm, calling empty versions of the various primitives.  Then I start filling in the primitives, and test the results *constantly*.  This iterative process really helps me get actual results.  My old habits of just improvising are now long gone, as this methodology really gets things done, as opposed to my old way of doing things, which often resulted in vaporware.

And as Bear said, you'll find your TODO list growing constantly larger, as you break down things into more specific forms, but this is a good thing.  And your CHANGELOG will give you the motivation to keep going, as you'll be able to see exactly how productive you've been, since sometimes it seems as if you aren't moving forward at all.

I'll also second Bear's comment about comments.  Use them liberally, as they'll make debugging much easier.  And since the debugging phase usually takes up 500% more time than the actual coding, this is vital.  Comments also make altering your code for new features much easier, since even the best programmer's can get lost in their own algorithms at times.

Looking back on my early days of coding, I often wondered why anyone would bother doing things like commenting, creating TODO and CHANGELOG files, or even using makefiles.  As you gain more experience, you gradually come to the same conclusions every other programmer reaches.  So, in other words, I can't recommend TODO and CHANGELOG files enough.  I wonder how I ever got anything done before I started using them.


Fenrir

  • Rogueliker
  • ***
  • Posts: 473
  • Karma: +1/-2
  • The Monstrous Wolf
    • View Profile
Re: Organization and Workflows
« Reply #7 on: October 22, 2010, 09:42:34 PM »
Then I brainstorm ideas for implementation/code design, and "weed out" design ideas that don't allow the kinds of content and functionality I want.  There's some back-and-forth where I re-imagine game ideas developed in different ways to allow their implementation under the design I'm considering, but if an implementation idea would require several content ideas to have different observable effects as opposed to different underlying structure, I generally scrap that implementation idea.
Could you give one or two examples of this "content/functionality idea --> implementation idea" part?

Maybe I learn better when I see things applied, or maybe I'm just a pain, or both. Apologies either way. Fenrir smirks.

Krice

  • (Banned)
  • Rogueliker
  • ***
  • Posts: 2316
  • Karma: +0/-2
    • View Profile
    • Email
Re: Organization and Workflows
« Reply #8 on: October 23, 2010, 07:58:15 AM »
Just out of curiosity, why do you find planning frustrating?

It's not frustrating, but I don't plan every detail before starting programming. From technical side it has been difficult, because most of the things I did for Teemu were the first time, like before Teemu I didn't exactly know how to implement save/load game. Also, I've been experimenting with events for greater flexibility and I had no idea how that would work before I tried.

Pretty much the same goes with the game content. I like to play the game and see what could be improved and what is missing. I now know that it's important to avoid long feature list and carefully select only some features that are going to be implemented in the next version. That's also a kind of plan that keeps me from getting tired of endless todo list.

Bear

  • Rogueliker
  • ***
  • Posts: 308
  • Karma: +0/-0
    • View Profile
Re: Organization and Workflows
« Reply #9 on: October 23, 2010, 05:54:11 PM »
Fenrir wrote:

Could you give one or two examples of this
"content/functionality idea --> implementation
idea" part?

F'r example, here are some gameplay ideas. 

One is that some items buff with use, to some extent the same way that characters buff with experience.   So, when the character takes experience, the items he or she is using may also take some experience. 

Another gameplay idea is that the bonuses, exact range and damage, etc, of particular items should be variable, so that it's possible to find a fire wand that's better (or better for particular uses) than the fire wand you have now, or organize your tactics around the peculiarities of equipment you happen to have now. 

Another idea is that "staves" should affect spellcasting, buffing your spells, or maybe particular classes of spells, by making them some combination of easier, faster, longer-range, higher damage, lower mana cost, etc.  So your staff isn't a charged item, it's a weapon that you wield while spellcasting in the same way that a sword is a weapon you wield while slicing.  And it affects what your spells do, in a way comparable to the way your sword affects what your physical attacks do.

Another idea, is that generally items will have many different uses, and may have disparate power levels in buffs for those uses. 

Another idea, this time for a particular item, is that there's a Rarity, a sword named Famine, which is more powerful the more you are hungry (but this is self-limiting, because killing things with it - even things you couldn't normally eat --  nourishes you). 

Another idea is that there's a type of object which is enchanted specially against particular species or types of creatures.  Examples include a crooked arrow of wumpus slaying, a sword of dragon slaying, etc. 

And a final gameplay idea - though it's really more an interface idea - is cross-system, mod-safe, and version-safe save and restore.  To the extent possible, installations of the game on different operating systems, installations of locally modded games, and installations of later versions of the game, should understand the savefiles left behind by versions found on all systems, by earlier (or all) versions, and/or by unmodded versions of the game.



Okay, now holding all these ideas in your mind, consider the question:  how to implement and represent magical (or normal) items and their properties.

When I'm brainstorming ideas for how to store item bonuses and properties, therefore, I reject out of hand the "rogue" idea that item bonuses and properties are fixed by type and that the appropriate representation is just a reference to the item type, because it doesn't account for individual objects buffing, nor for variability within type. 

I consider the pval business from Angband, where an item is buffed by incrementing its Pval number - and the meaning of the buff is a function of item type and pval.  At first this doesn't seem like it would work, because it only stores one number.  If I want *this* fire wand to have longer range but lower damage than some other fire wand, but the fire wand always interprets pval the same way, then one number for range and one number for damage can't both be the pval.  On the other hand, it seems to work okay if I make "foo of range" fireball wands separate from normal fireball wands.  And with large multiplication of item types to cover each combination of disparate power-levels in buffs, this covers most cases.  But due to the diversity among staves (diverse staves X diverse affected spells = lots of cases) in particular and other weapons somewhat, it requires an astounding multiplication in item types.  And it doesn't cover all the cases, because Famine in particular and foo-slaying weapons in general need to refer to their wielder or target, respectively, to figure out just how awesome they are on a particular swing.  So, I rejected it.

Angband modified this idea for the benefit of foo-slaying weapons by devoting bits in each creature representation to say which sort of foo it is (of the very limited number of general groups that foo-slaying weapons in Angband target), but I don't think that's general enough, and it still doesn't account for the specific wielder-hunger query of Famine.  So, I reject that too.

I consider the idea of having a separate pval'ish enchant value for each of several attributes - to-hit, to-damage, to-defend, etc, recorded on each item - along with a number for antagonist-type to determine the species or class that this is a bane against.  But then I realize that with possibly several different enchants on a given item, this too is a non-starter.  While it abstracts the reference to the target and returns enough info for the foo-banes, it still can't resolve that the user is hungry enough to unlock Famine's totally-awesome-hits-of-over-the-topness.   And it's ridiculous to have code in your hit routine to check the hunger level of the wielder when 99.999% of all items won't use it and Famine is a Rarity anyway so it won't even appear in most games.

Hmmm, code.  Here comes another implementation approach.  The code that checks the hunger level of the wielder ....  It has to "belong" to Famine, doesn't it?  Nothing else uses a check like that.  It is what OO aficionados call a "member function" of the object, right?  And the code that checks the species, class, type, etc of the target has to belong to (but can be shared by all) foo-banes. So, if there's code associated with particular items, when and how does it get called? 

And then I'm back to the idea of "Events."  Each event has actor roles, including four corresponding to subject/attacker, agent/weapon, agency/ammo, object/target.   What if there's a moment when the event-processing code looks at each actor for "methods" corresponding to the current event-type and the current role of the actor within that event, and calls them if they exist?  The argument given that code, under the assumption that it needs to key on some combination of properties of the actors in that event, would be a reference to the event itself, where it can see references to all the actors.  There's a practical problem with this, which is the sheer number of "methods" involved.  Every combination of event and role within that event would be many thousands of function prototypes, most never overridden, so it would be really ugly in C++ or one of the obligate-OO languages where you have to have a prototype for each overridable method.  But I'm writing in C not C++, so I'm free to use a Data-Directed Design instead of an Object-Oriented Design. if I take a data-directed approach, I can query a hash table for a code pointer, returning NULL if there's no such method, and then check for and just not call NULL, so I never have to write a prototype at all - only the specializations.

So....  each item has a hash-table of attributes.  If there aren't very many, it won't waste the space for very many.  NULL return is always an option, so there's no need to store values for attributes that don't exist.  Some attributes have code-pointer values, and the keys of those attributes are "Triggers" that the event processing code will compute from item type and role, then query for, and if non-null, call.  Famine can check the hunger level of the wielder, Foo-slaying swords can check to see if the target is a foo, and no event whose actors are uninterested in the results of those checks will ever cause such code to be called.  The code can express its results by writing back into particular fields of the "event", changing, for example, hit probabilities, range, damage levels, costs, etc before the event gets executed.  Number of possible attributes can increase without limit, and each item uses space proportional to the attributes it actually has.  And items can have an "experience" attribute that keeps track of their level of buffing.  In fact, it can be the same "experience" attribute that monsters have, and an add-experience event can call the same incrementing/buffing/leveling  code if the items, like creatures, have the takes-experience attribute set to that code pointer.  Meanwhile, this leaves room for both "ordinary" items and creatures that take no experience (undead, golems, etc, just don't have a takes-experience attrbute).

I check this last approach against the ideas I have for items, and find that it works.  It covers everything I want to do, several things I hadn't thought of under the heading of "items" before, and sometimes with mild abuse, everything I can imagine wanting to do with representing the special properties of items and monsters in my game.

But when I check it against other ideas, I reject it because it violates save-and-restore.  The same code pointer value won't mean the same thing in the restored game that it meant in the saved game, because there's little chance of the same routines being stored at the same addresses.  But it's close.  Can I make it pass save-and-restore by adding an indirection to it, where instead of code pointers I have an index number that I use to access one from an array of code pointers?  The index numbers would refer to the same code pointer, across different operating systems, installations, etc.... but this still violates version-proof save-and-restore because later versions of the game will add some code pointers and rearrange the numbers that refer to them.

So, this new approach still doesn't pass version-proof save-and-restore either, but it's very close.  Okay, I say, make the numbers into named constants, and save the symbolic names in the savefile rather than the number.  Does it pass version-proof save-and-restore now?  At first I thought so.  But I was wrong.  I realized I was wrong when I added another actor reference to the structure of the event. I had to change the code that figures out what key to associate with each software method stored in objects.  I was clever enough to make the code indices version-stable, but I hadn't made the keys under which they were stored version-stable.  So this still violated version-proof save and restore, and I had to fix it.

One more revision to the basic idea, then.  The keys that the code triggers are stored under are also saved as symbolic names in the savefile, and these names will also mean the same thing across different versions, even if it doesn't happen to mean the same index number.

So, after rejecting several approaches, I hit on one that was "close" in terms of enabling all the gameplay ideas I had for items.  But it still failed save/restore.  With a few more refinements it passed save/restore, but if I hadn't found the appropriate refinements, I'd have had to reject it too.

So, after writing most of a book chapter....  It's not really the case that a gameplay idea refines itself into a design idea.  What it comes down to is brainstorming design ideas, then *REJECTING* design ideas that don't allow your gameplay ideas to be expressed.  Sometimes you can rehabilitate an implementation idea by a refinement that permits it to express a gameplay idea that it failed in its initial form, but sometimes, if you can't see this problem coming in the design phase, you have to scrap something you've already implemented and redesign that subsystem (which in the worst case involves redesigning other subsystems too).  Or, the alternative, which is scrapping a gameplay idea (Not a bad thing if implementing it really is intractable or not worth the effort; is Famine an "unreasonable" item?  Could you implement it in your design? I could have gone with a simpler idea if I'd chucked it. )


Fenrir

  • Rogueliker
  • ***
  • Posts: 473
  • Karma: +1/-2
  • The Monstrous Wolf
    • View Profile
Re: Organization and Workflows
« Reply #10 on: October 23, 2010, 07:46:08 PM »
Fenrir gleefully scribbles notes in that notebook he seems to be carrying around in his fur somewhere.

My sincerest thanks for the very in-depth explanation, Bear.

Bear

  • Rogueliker
  • ***
  • Posts: 308
  • Karma: +0/-0
    • View Profile
Re: Organization and Workflows
« Reply #11 on: October 27, 2010, 05:49:53 PM »
I don't plan every detail before starting programming. From technical side it has been difficult, because most of the things I did for Teemu were the first time, like before Teemu I didn't exactly know how to implement save/load game.

Krice is right here.  You can't possibly plan every detail.  It would take a long time and you'd probably find part way through that you'd committed yourself to doing something impossible and have to plan every detail again ... and again ... and again ....

The trick is to pick out things that are heavily constrained by your gameplay ideas, or which have to do something unusual and possibly difficult, and develop a plan for implementing those.  It doesn't have to be a very specific plan; you don't need to decide how the routines will break down or which operations you're going to abstract or what the variable names are going to be.  What you're doing when you plan these things is making sure you have an implementation strategy in mind where you don't need information before you have it, where you don't throw away information before you need it,  where you don't have to solve, eg, the halting problem or strong AI or something else you have no idea how to do, and where you're pretty sure the solution you've picked will scale reasonably as the program grows (so neither you nor the computer has to put in NX more work each time you add another X to the game).  Once you are sure you can do it and have a pretty good idea how, you make anywhere from a paragraph to maybe a page or two of notes and go on. 

A good plan, IMO, is more about making sure what you want to do is possible than it is about figuring out exact details in advance.  Things that are very straightforward (even if they're a lot of work) you don't need to plan when starting out.  A one-liner in your design file acknowledges that they need to be done, but if you don't see any problems or pitfalls with doing them or constraints that might be tricky to manage, then don't bother to plan until you're ready to do it.

And the trick, of course, is being right when you decide you don't need to plan something, and being right when you decide that a particular plan for something is in fact viable.   When inexperienced, you will make each type of mistake a few times.  Getting a long way into a project and then finding that you have no viable way forward is the symptom of this kind of mistake.  It sucks when it happens, but don't beat yourself up over it.  If we are not clue-resistant, we make mistakes and learn.   If you have not implemented something like save-and-restore before, you may not realize some of the issues that can make it hard.  At worst, you may have to scrap a lot of code that you wrote when you assumed you could solve a problem you don't actually have a decent way to solve.   But even then, that code isn't wasted.
That code is just the byproduct of a learning process that made you into someone more able to reach the solution. 

Bear