AI and - connected to it - emergent gameplay are two of my Holy Grails that I always look forward to in games, sadly quite often ignored. I guess these being complex issues is one of the reasons, though it`d be nice to see more people trying at least.
My favourite RLs - Linley`s Crawl, Brogue, Sil, Qud all have rather interesting monsters/behaviors. Some examples & ideas:
- different movement patterns: ye ole bat is the most obvious with it`s erratic, infuriating schemes, then frogs can jump and advance rapidly, bulls could charge blindly, centaurs and such are fast, nagas slow.
- depending on intelligence and type: different responses to the environs; avoiding traps as you said already, but also natural obstacles like water, pits etc. That charging bull would plunge into one, I suppose.
- depending on intelligence and type (2): different equipment and use of collectables: you mention it already. Crawl`s strong suit...it really makes the world more believable instead of countless monsters which differ only in name and HP count.
- trying to lure player into ambushes and traps - Sil excels here with its group tactics, retreats and suchlike
- tracking the player: there really should be more to this, in so many RLs having a stronger foe next to you means curtains no matter how far you run. But monster could run out of stamina or trip over. And so could the @. Also, scent and tracking in general: some monsters should be better and some worse at this, doors and environments again should aid or hinder the escapee (old trick with jumping into a stream? or throwing out loot for the greedy kobolds or meat at trolls/hounds).
-different reactions to @s presence depending on varying circumstances - maybe monsters don`t always feel like attacking? Could be alignment, player`s strength or rat`s allergy to leather. Expanding: intelligent monsters could ally with the player or let him off for a "fee"
-there could be other adventurers in the dungeon with agendas of their own, again these could aid or hinder - or just ignore - the player.
In case you missed it, great post regarding AI in general from Dr URR:
http://www.ultimaratioregum.co.uk/game/2016/03/13/alphago-lee-sedol-and-grand-ai-challenges/