As you say, a simple RPS-minigame might be difficult to pull off. In particular, programming a clever AI to make interesting decisions (I would think), given that random choices constitute perfect play in RPS. You should also consider that typical Roguelikes have your @ fighting multiple opponents at once, so a RPS-based system might be difficult just because of that. That's not to say you can't have something like battle screens as a proper mode in the game, like they do in certain CRPGs (Final Fantasy etc).
If you haven't, you could check out the 7drl Malleus Goblinficarium, which features interesting one-on-one combat. It's not based on RPS per se, but has a similar duel-y feel to it. Note how transparent the workings of the AI is to the player.
I'm also reminded of the board game "Fury of Dracula", which used a variant of RPS for combat. If a vampire hunter was fighting Dracula, the hunter had a number of options (garlic, stake, holy water, pistol, swordcane, dodge ...), and Dracula had his options (fangs, bat form, mist form, hypnosis ...). The two players would each choose an option and cosult a table to see the effects. Some actions had special rules or carried special risks. The stake was very effective against a few of Dracula's stances, but could be fatal to the hunter if met with one of many others. I seem to remember combat was ok implemented, not great.
Just rambling more random thoughts: Maybe all monsters and weapons have different actions, each action with a certain effect as well as a certain RPS-value or element. For instance, let's say a sword could have "lunge" (rock), "parry" (paper) and "focus" (scissors). If player plays "lunge" and opponent plays "parry", the opponent wins that round (perhaps not damage, but something like make player unable to choose "lunge" again next round). Other weapons and natural attacks could have different effects, whilst retaining the value of either rock, paper, or scissors. Getting to know the creatures might entail learning that eg. harpies strongest/preferred attack is scissors ("scream"), meaning anything that plays as rock ("lunge" in the sword example) generally plays well against them. Of course, they'll occationally dodge (paper) or use their claws (rock). The player could have a repertoire of actions based on their skills+inventory. For balance, especially powerful attacks might need something like an adverse effect if you try and fail to use them. Your basic sword would probably have no side-effects, but maybe magic. If you cast "paralyze spell" (scissors, say) and your opponent plays something with rock, then maybe you're confused for one round, in addition to getting hurt by the attack. Similarly, an executioner's sword might have a "behead" attack (scissors, again), which instakills any opponent unlucky enough to play paper that round. If failed, perhaps the player is disallowed to play scissors at all for one or two rounds.
Whatever you do, test, taste, and season. Maybe you'll need more than three elements, for instance. I think you won't be able to anticipate all the humps in the road in advance. In particular, if you're trying something that hasn't been done to death already, starting actual testing at an early point is probably just as valuable as a detailed design document. I'm notoriously bad at project managing, just to be clear.
Happy coding.
As always,
Minotauros
