Variety in NPC behavior

Ideas for various NPC behaviors, compiled from r.g.r.d.

Benhem

"crab" movement: fast sideways, slow up/down.

"turtle" -- can retreat into a high-defense "shell" mode.

"swarmer" -- monsters follow just outside of the player's light radius 
until there's a small gang, then swarm in.

"slug" -- leaves a slime trail when frightened, which slows the player 
down.

"nest-breeder" -- Two can combine to make a nest, which acts as a 
Gauntlet-like monster generator until bashed.

"chess moves" -- e.g. straight/diagonal limitations or things like the 
knight-jump.

"territorial" -- tends to patrol its "territory" and bully monsters 
within it.

"charmer" -- will try to charm nearby monsters and amass an army.

Gelatinous Mutant Coconut

"grid-bug crab" -- have movement costs for the monster be greater in the y direction than the x direction.
"realistic crab" -- have movement costs for the monster be greater in a direction parallel to its target than in directions perpendicular to its target. (If your roguelike implements facing and turning costs, obviously use those instead.)
"porcupine" -- can retreat into a zero-movement mode; melee attacks on the monster reactively damage the attacker while in this mode.
"Grue-like" -- this monster hates light. If the player has no light source, it closes in on the player to attack. If the player has a light source, it flees away from the player (at a faster speed) until it is outside of the light radius. A smart player might be able to trap the monster in a corner and kill it while it is helpless.
May be made more general by having the substance (blood, poison, confusion potion, etc) and condition (IS_FRIGHTENED, !IS_HUNGRY, IS_LIT, etc) given as parameters.
"nest-maker" -- like "nest-breeder", but only needs one to make a nest (a fertile ant queen, for instance).
"fusing" -- like "nest-breeder", but the resulting creature is mobile.
"evolving" -- like "nest-maker", but the resulting creature is mobile.

Radomir Dopleralski / The Sheep

I think the most basic monster behavior that changes the rules of 
game when that monster appears on the level is collecting items. There 
are three variants: item-eater, which destroys the items that it picks up, 
single-item-carrier,  which picks up a single item and drops it on death 
(or when it finds a better item), and item-collector, which collects many 
items and drops them all when dead. You can add variety by making the 
monsters only target some kinds of items (food, gold, magical items).
Examples:

 rust monster that eats all metal weapons and armor it finds

 leprechaun that picks up all gold it finds

 gelatinous cube/amoeba that swallows all items and expells them upon 
 death

 ants that pick up single items and carry them through all the dungeon
A variation on the topic is a monster that becomes somehow different when 
it finds specific item (and maybe reverts back when the item is 
destroyed). A goblin might turn into a goblin warrior after finding 
a weapon -- and it doesn't have to be simulated by it actually equipping 
the weapon. The item might also make the monster weaker: in Rogue orc 
becomes immobile after finding gold, for example.
Another common theme is special reaction to certain attacks. This gets 
interesting if you allow the player character to throw potions at 
monsters. For example, as someone mentioned on #rgrd recently, silencing 
should work as blinding on bats and bat-like creatures, while blinding or 
invisibility might not affect them at all.
Generally speaking, status-altering attacks are a source of great variety 
in the game, as they let monsters interact with each other through the 
player character. The status may include poisoning, blinding, immobilizing 
(lasting until the monster is defeated or as paralysis for a certain 
number of turns), teleporting, confusing, disarming, weakening, etc. all 
of which will change the relative strengths of other monsters around.

Billy Bissette

  There is another variation possible for some more magical/special creatures.  Not just item damage/destruction, but outright item changing.
  Azure Dreams had a monster that appeared on only one dungeon level 
that was effectively a living magical bag.  It attacked like other 
monsters, but if you threw an item at it, that item would disappear 
and it would throw a random item to you in return.  This behavior 
was meant to be abused to make building a winnable character easier. 
Indeed, you could go as far as to make one a pet, and thus have 
item transformation available on every level in trade for not being 
able to take a more effective attack animal in that spot.
  While not necessarily pure random transformation, you could have 
monsters that pick up one item and transform it into a different 
(but not necessarily inferior) item, even perhaps dropping said 
new item back to the ground.  Maybe a rock eater would turn metal 
objects into small piles of the related metal.  It eats a gold 
crown and leaves a few gold nugget "crumbs" on the ground (or 
simplify and just leave "coins".)
  A fire elemental might burn up some objects on the ground, but 
maybe others could be transformed.  To steal and slightly modify 
a couple of ideas from Tobal 2's quest mode, bad meat might be 
safely cooked if a flame creature briefly walks over it, but 
cooked meat will be burnt to a crisp and destroyed.  Potions 
might be transformed as well.  In Tobal 2's Quest mode, holding 
a potion inside a fire would cause its color to lighten and 
thus transform it into a new potion type.  Expose it too long 
though and it would evaporate, and thus be destroyed.  (Tobal 
randomly reassigned potion effects on every dungeon entrance and 
town visit, so this was sometimes quite useful and sometimes 
quite useless.  The mechanics of the game also made it take 
serious effort to abuse, though it was likely expected for 
serious players to take as much advantage of it as they could.)
  A gremlin or certain demons might curse objects, while a 
non-violent angel might bless them.  Temporarily or permanently, 
depending on what it means to your game system.

pdpl

Facehugger: rushes to you to infect you with something.

Screecher: mostly harmless, makes a huge noise if it's hurt, 
attracting more monsters. Can be deadly in the same room as several 
other monsters, since it'll mostly shut down your aoe (if your game 
implements any).

eggs: monster-on-a-timer, there might or might not be a mother nearby. 
unseen: your average invisible monster, turns visible when it attacks 
you, and stays visible for a turn after that.

guerilla fighter: ranged attacker, plays hit and run. Problematic if 
your PCs are mostly melee fighters.

E. Liddell

I'm surprised that no one has mentioned the obvious "ambusher" yet (monster hides/pretends to be something else, then springs a surprise attack on the player when he steps on it/tries to manipulate it/ whatever). Potentially interesting variant: the monster pretends to be an NPC or other harmless creature.
For a more complex interaction between monsters, consider the cultivator:

a more intelligent monster that "plants" less intelligent sessile 
monsters either around its lair/treasure stash for protection, or in a 
location where it can drive or lure the player into them. A variation on 
that would be a monster that uses traps for similar purposes. Nothing 
like chasing after a monster that moves in 2-square jumps every second 
turn, and being dropped into a pit trap in a square that it skipped 
over...

Timothy Pruett

• note -- All the monster's can take into account allied strength,

which is factored into their own when comparing against a percieved enemy.
Typical -- Attack percieved enemies with percieved strength lower than or equal to own. When wounded to a certain point, attempt retreat. If retreat is impossible, attempt to attack with full force (berzerk).
Scavenger -- Avoid enemies with percieved strength within certain percent of own, but follow within certain distance. Attack enemies if they are weakened. If valuables/food is found, take it. If wounded lightly, retreat.
Coward -- Run from any enemy, no matter what. Only fight back when cornered.
Patrol -- Patrol specified area, until enemy presence is detected.
When enemy is detected, attack. Other than the patroling part, the rest of this module is identical to the Typical module.
Special -- Specially scripted AI for unique enemies, bosses, and townsfolk.

Auric

Insane -- Always attack the enemy, no matter how strong it is, and fight until death. (Standard hack-n-slash game AI)
Coward 2 -- Hide from the enemy, and later attack from the enemy's blind side. (D&D thief)
Long-range -- Attempt to keep the enemy from getting close, while attacking with ranged attacks.
Trapper -- lays a trap and waits for the enemy to walk into it, only entering actual combat if the enemy isn't killed by the trap.
Experienced -- fights with good tactics: if wounded to a certain point, will use healing magic/potions on self; fights cautiously but not cowardly, if outmatched, attempts to bargain its way out; etc.
- flanking maneuvers: goblins attack you from both sides!
- ambushes: a buncha goblins jump down from a ledge, landing on your head!
- false retreats: (variation on ambush) the goblins run away! oh, wait, they just led you to another dozen goblins...
- overwhelming numbers: you are attacked by a hundred goblins! - hit-n-run tactics: the goblins hit you for X damage! the goblins run away! [later] the goblins return!
- relief combatants: you hit goblin A for X damage! goblin A retreats! you are attacked by goblin B! goblin A drinks a healing potion!

R. Dan Henry

Daredevil -- Defends if attacked or allies attacked, but only initiates combat if forces are "close enough" to equal. Represents groups that fight to prove courage; suicidal attacks are still stupid, but an attack with a strong advantage shows no heroism, so is pointless.
Greedy -- Largely ignores target strength. Attacks if target appears wealthy (including obvious magic goodies, like flaming sword). Fanatic -- Ignores others unless they break a religious taboo (so each Fanatic type needs AI modified for different taboos), then makes berserk attacks.

Kusigrosz

I think monster-monster interaction is definitely a good idea.
It is not easy to get right though; on a large map it could leave parts of the level in a 'strange' state. OTOH, if properly balanced, this could be part of the fun - maybe even worth running the level without the PC for some turns immediately after it is created, to allow some interesting situations to develop.

The Aging Minotaur

...For instance, the game features a kind of giant moth ("night swarmer") that is never hostile to anything, instead moving completely at random and performing an attack on whatever comes in its way. Crossing a cave with these beasts is a trying task, but you can also lead hostile monsters into such a flock and retreat to a safe distance. There are also groups of adventurers, who are neutral to the pc, but will attack certain monster types. OTOH, these bands tend to clog passages the pc needs to get through, or use area attacks that can harm the player if s/he gets in the way (or even their comrades, causing in-fighting, which is amusing in itself, and tends to leave some fair equipment lying around when the dust has settled). In a game where the player can only move orthogonally, there are a few monsters that move along the diagonals. Snakes became an interesting addition: they're only allowed to move diagonally, wants to attack everything, and retreats from enemies standing orthogonally adjacent to them. This makes them difficult to hit, but usually easy to avoid. If you are being chased by some brutish melee monster, however, a snake pit can turn out to be either a useful dungeon feature or a deadly trap.
...
The idea now is to generate random cultures for two species at the beginning of the game. One is loosely based on Swift's lilliputs, and one on Breughel's dancing skeletons. The customs will be "molecular" kits that can be patched onto the data objects used to create these critters and their environs. A lot will be cosmetic, or have only minor effects. Details pertaining to archetecture and inventory lists is one example. In one game, a species might live in small, connected rooms and favor books for treasure, in another they may prefer to build houses in larger caves and keep mummies in crypts[1]. Other quirks will set or rearrange switches in the state machine of species/individuals. Here you've got local and global taboos ("don't eat in the temple", "don't harm <species X>"), paired up with reprimands (dislike, attack, give penal quest, verbally abuse, etc.) You can also get stuff that's almost purely for show (for instance a quirk to make a species musical, adding instruments to their treasure lists, and making them susceptible to passing the time by playing when they're at ease).
The possibilies are many, of course. You can have herders (keeping flocks of animals, either behind fences or as pets in the wild), hunters (wanting to kill and collect the corpse of certain species), farmers (have gardens), etc. I'm aiming for a nonsensical setting, so the plan is to combine cultural quirks pretty much at random and hope for interesting effects. The game play value I hope to gain, is to make exploration of the game world fresh from game to game. In lots of RLs, you pretty soon learn how the setting is put together, eg. that you have "kobolds" and "kobold archers". Instead, I let some of the moving parts vary from game to game. For instance, a species might always contain a subset of <archers>, but they may be rock throwers in one game and bombadiers in the next.

links

r.g.r.d: "Some suggestions/help for my monster AI"
r.g.r.d: "Open call for monster behaviors"
r.g.r.d: "Emergent puzzles to beat the grind"