A very interesting problematic, indeed. One issue I think can create difficulties, is to make a system that scales well. In my old project, I bumped into the problem that drop rates for everything got effectively lower as more items were added. This could mean that the spawn rates for rare items might drop so close to zero that they're practically out of the game. On the other end of the spectrum, there'll be some essential tools (healing potions, ammo, whatever your player will need to be able to play at the intended pace). Again, if you have balanced the frequency of such items within a set of 30 items total, the balance might get skewed as the number of items goes up (ie. at 60 items, you only get half the healing potions you need). For my next project, I hope to make a system where I don't have to go back a rebalance item frequencies all the time.
Mind you, the right system depends on the scale of the game. For some dungeon crawlers, it can be quite well balanced if, say, levels 1-3 feature "hand axes", levels 3-6 "battle axes" and levels 6-9 "executioner's axes". For a more open world, I've always felt it makes sense that anything *could* show up at any time.
Re: the issue of making sure essential items show up, I think ADOM has an interesting solution: In addition to items with a danger level of 1 or more, there are items with a danger level of 0, which may show up at any time. This way, healing potions are sure to spawn even at deeper levels. For my current project, I've been considering something similar, maybe giving each item a frequency category ("essential" , "common", "rare", "unique"), and weighing the game so that "essential" items are sure to show up at a certain rate, regardless of all else.
Re: who drops what, I've been swearing by "treasure categories" and "tags". Categories are used to tell a being it should carry, for instance, 1 weapon, 1 armor, 0-1 rings and 0-1 scrolls. Tags are less hard and fast, and are just used to skew the probabilities, making rooms/creatures more likely to spawn items with similar tags as themselves. For example, a knight and an orc may both be set to own one "weapon". But if the knight is tagged "military", that might up the chances for things like maces and plate mails (which would also be tagged "military"); the orc could be tagged with "chaos" or something to increase the chance of spiked clubs, bone helmets, etc. Similarly, a simple instruction to carry one "tool" might give yield a lockpick for a thief, a hatchet for a farmer and a severed arm for an orc. This can be a quite powerful system, especially if implemented on a large-ish scale, with thematic rooms and such.
As always,
Minotauros