Hi, that is a very broad question
Try to get a prototype that you can play as soon as possible. It's near impossible to assess how difficult a given system will be in advance (especially if you're not an experienced game designer already). And when you get to the point where you're testing your own game, keep in mind that you're a lot better at it than most players. Ie. enemies that are windshield kills to you, will be a challenge to a newbie, and the stuff that you find difficult will be unbeatable to some.
The prototype is a roguelike-like, like the aforementioned Legerdemain.
Non-random maps, easy to die, and when this happens, the player can choose to "die" or to be resurrected with a penalty (losing gold, decreasing attributes, losing one random object from inventory, or whatever), so death is something to be avoided.
He cannot save whenever he pleases, only like "Save and quit".
Difficult features in games should of course always be interesting. As you note, putting in overpowered enemies (or just scaling enemies' HP as you get deeper in the dungeon) isn't very fun at all. Enemies with special abilities, on the other hand, can be fun. Most RLs feature one or more basic enemies (aka "giant rats", "zombies") which will usually just charge the player and attack in melee. A step up from this is any enemy with a single special trait: One that is fast, or uses ranged attacks, or is slow but super strong, etc. It doesn't always have to be "special attacks" either. Mixing in enemies with different behaviour patterns can give rise to a lot of challenging game positions. It goes without saying that any enemy you normally encounter should be beatable (or avoidable/manageable if you play well). Strictly speaking, a game can be said to be broken if it ever generates unwinnable positions (given perfect play from turn 1), but you'll have to at least partially rely on your gut instincts.
But monsters' special attacks or abilities are something limited, and once you know how to deal with a monster with such ability, the game difficulty will difficulty and so the challenge it represents to the player.
Also, due to the nature of the game I'm developing, magic is a very limited and scarce resource or skill, but powerful.
If your game depends on limited resources (potions, mana points, power cooldowns, whatever), the right balance is usually that the player will find/have just enough to get by.
In the end, it all depends on what you want. Some games use procedural lock+key puzzles (Brogue and Ultima Ratio Regum spring to mind), some require intelligent usage of spells/abilities (like Caves of Qud or Legerdemain), others go for a more pure tactical tension (check out PrincessRL and Hoplite, and a plethora of other 7DRLs). Rogue itself is a notoriously difficult game, and yet supremely well designed. It more or less invented typical RL systems like hunger and identification, and stands still as a shining example of what these systems are really all about.
I kinda like this more. It looks like the player must manage the difficulty at all times. Using consumables early on could lead to problems later, while reserving these resources from the beginning could make hard to reach deeper dungeon levels since you are making the game harder in the begining.
But in this case, both low level mobs and high level mobs should represent a rather similar challenge for the player, otherwise there is no reason why the player should use potions or spells at the beginning. I mean, if a rat is not a challenge at all, then the player will just save these potions since he doesn't need them at all, so the logical decision is not to use them.
Nothing bad about this, but the problem could come if the player saves a huge amount of resources for later levels, then these later levels would be too easy.
You can also have strategic decisions which are "hard" in a sense, such as whether to invest skill points in weapon mastery or magic, or which wand to leave behind if you have a limited inventory space.
Limited inventory space is something that I have been working on already. I am more inclined to think about a weight-managed inventory.
In this case, the player again must manage how easy he wants the game to be now and in the future:
saving potions in the inventory for later will not allow him to carry one extra weapon for low level monsters (a problem if your current sword breaks). But if you have a pair of sword and do not pick up potions, these swords might become to weak when fighting dragons instead of rats.
So either drop the sword or drop the potions... Or find the best balance.
If you like to play RLs as well, take note of how your favourite games implement systems to be difficult/interesting/whatever-it-is-that-like-about-them. Also, read up on which games are out there and check out the ones that seem interesting. Steal and adjust the ideas that you like.
Hope some of these stray thoughts can serve to inspire in one way or the other.
Actually I have several favourite RLs, and their approach to difficulty is different, such as Nethack or DCSS.
Yes, your thoughts are really useful. Thank you.
I think I will experiment with the limited resources and volume-managed inventory by now.