Have you thought of having a cost for each enemy/weapon/item and matching it against a current potential value of the player?
IVAN has a system where monsters which are allowed to appear depend on the player's HP and equipment. This has counter-intuitive effects. Like, items which increase HP are considered worthless, because they cause stronger monsters to appear. Or, when going to a new level, it might be better to stash the best equipment and have temporary buffs off. This way, strong monsters won't be attracted. Overall, it is not clear whether you should make your character stronger, or not. I recommend against using such kind of systems.
You could also include modifiers, like if the player is a cleric, skeletons points value is halved, or if he doesn't have a ranged weapon green slime points value is increased by 15%
This gives a very balanced and fair game.
Again, in such a design it is not clear what is good for you and what is bad. Clerics have a some special power which helps against skeletons, but on the other hand, they face more monsters. You might consciously decide that you do not want to bring a ranged weapon, because you want less green slimes, or less monsters in the rooms containing the green slimes. And as the total number of points depends on your character, you learn to develop the character in a way which makes them strong, but the game does not see it.
IMO such a design is cheating, and the player will cheat you too.
I understand your reservations about such a system, and you have made me think more carefully about using it, but I think The system is sound as I described it.
When I used to DM paper and pencil RPGs in highschool I used to balance the encounters against my players, if they had a weak combination of party members or a couple of poor character builds I'd substitute some of the monsters in the module. I didn't want them to get killed too easily, but I would make it challenging and sometimes they did all end up getting slaughtered even with the substitutions because of making poor tactical choices. I'd do other things to make the game more survivable too, such as including more healing potions if the party (which would some times change from week to week as the club membership shifted) didn't have a healer, or less traps if they didn't have a thief.
When I've played RPG games on computer I've found myself almost always playing a cleric just because, they have a fairly good set of combat abilities, they can heal themselves and allies, they can summon creatures and they can buff themselves. Most games become a walkover in such a case. If I chose to play as a bard or a gnome illusionist it's going to be much much harder, probably impossible unless I play on an easier difficulty setting. This is a problem because I actually don't really like playing a cleric, I actually want to play a thief and sneak my way through the game, but it's rare to find a game that will adapt to my playing style.
I think an auto-tuning game system gives you more freedom to play the game anyway you want to instead of being forced to play only the "right" way.
Such a system could leave itself open to being "gamed" but as I showed above (but didn't thoroughly explain), it's important to have a mixed system. Although most encounters will be balanced to the player's abilities (generating monsters from a pool of XP), there will be some which have a set value. You could strip off all your armor and weapons and descend in to the dungeon in just your loin cloth expecting to fight only pixies and gnomes, but that rare encounter where you face a set of monsters pitched at what "should be" your value at that part of the game if you are playing as expected will soon wish you didn't leave your armor behind.
Besides, an item that gives more HP is always useful right? No matter what enemies you're facing, it's better to have 10 HP than 2HPs. Being stronger is always going to be better than being weaker, as usually some enemies will have special abilities which let them hit above their points value in certain situations. For example spiders have poison, giant rats have disease. You may be facing less of them because you are not a cleric, but still once bite can be enough to really ruin your day. If you haven't got a ranged weapon then that one rust monster (instead of two because of the XP modifier) is still going to be chomping on your fine plate mail if he scores a hit. Problems could arise if you set the XP modifiers too high, effectively pricing certain monsters out of the bell curve, but that's part of the design process and something that needs to be gotten right for the game to be any fun.
Every game becomes an arms race between the player who is trying to beat the DM's tactics and the DM, who is trying to beat the player's tactics. As a game designer you have to think ahead to find ways to confound the player without just throwing random stuff at them.
As a games designer I could cheat and use the system to single out a player's weaknesses so that I could kill them off easily, but then who's going to want to play my game then? It's important to get the balance just right so the game is challenging to every player but not impossible.