http://www.roguebasin.com/index.php?title=Roguelike_Intelligence is
the classic article series on Roguelike AI, of course. Also, if you dig through the archives over at
Ascii Dreams, you might find that Andrew D. has written something nice about this topic (and others of interest for budding RL devs). And here's
another take on the subject. And I think some people swear by neural networks.
I use a variant of the state machine AI described in the Roguebasin articles linked to above. Each actor is always in a current state (with a current target), which basically consists of a series of if/else statements. The basic "zombi attack" state would be:
if can_attack(target): attack(target)
elif can_approach(target): approach(target)
else: return_to_previous_state()
Functions like "can_attack", "can_approach", "can_heal", "can_debuff", "can_buff" iterate through the actor's skills and inventory and sees if the appropriate action can be performed. I put a tag on every command/action, so that a healing potion is tagged "heal" and a spell of slow monster is tagget "debuff". That in itself opens up for surprising behaviour, such as a goblin suddenly zapping a wand at you or tossing a potion of speed at one of it's comrades.
For more varied states, add a probability for each action. Say, if the function d100() spits out a random number between 1 and 100:
…
if can_shoot(target) and d100()<75: shoot(target)
elif can_debuff(target) and d100()<50: debuff(target)
…
In addition to states, each actor's personality is defined by a list of "bias switches" and a dictionary of "standard states". The "bias switches" are used to observe the playing field and react appropriately. For instance, a "self preservation" bias switch might dictate that whenever another being harms the actor, the actor turns hostile towards that being. Bias switches can also be contained within states, so that a "relaxed wandering" state could include a bias switch to turn and attack any enemy within line of sight. The "standard states" come in a few tactical categories: "attack", "flee", "buff", "relax", and just make the system a bit easier to maintain. For instance, a dog and an archer might share "reptile brain"-bias switches that sets them to "attack" any enemy that comes into sight. But they will have different states listed as their preferred "attack" states. This way, we don't need individually defined "wander" states for every single being.
It's a nice starting point for a system, since it's quite flexible. Let actors know about which group they belong to, where they have their home, and maybe important landmarks, and you can get seemingly complex behaviour, such as hyenas hunting in packs, villagers sacrificing at the temple or having a drink at the local tavern, quest givers behaving properly, etc.
And you can easily expand NPCs by plugging in extra bias switches. A "coward" personality trait might include "flee" as one of the standard "attack" states; a "dog hater" might have bias switches that makes him/her hate dogs on sight, and love people who kill dogs; a "drunk" may get "go to tavern" as a "relax" state, etc.
Remember that you don't need a too subtle system. To see if an actor heals itself, you don't need complex computations based on current hp, relative strength of opponents etc. It's suffient to say something like: "if I'm hurt and have healing remedies, 50% heal self." To the player, the end result will look almost exactly the same.
Also, make sure the beings don't do all their interesting stuff off-screen. It's probably good to make them "sleepy" when they're not actively chasing the player or something like that, and wake them up when they come into view. It's more fun to come across a knight fighting an orc, than just the remains of the battle.
As always,
Minotauros