May I suggest having whatever resource used for spell casting being split into more than one resource? It could be as simple as having multiple types of MP that were used for different spells. Maybe spell 1 uses type A MP and type B MP, while spell 2 uses type C and D, spell 3 uses type B and C, and spell 4 uses type A and E. With such a set up, you could cast spell 1 without it affecting how often you can cast spell 2, but casting spell 3 would lower the number of times you can cast spells 1 and 2, while casting spell 4 lowers the number of times you can cast spell 1 without affecting how many times you can cast spell 2 or spell 3. Basically, the kind of thing you see in a multicolored Magic the Gathering deck.
One game that I remember from my childhood that I think made this interesting was Secret of Evermore, which relied on an ingredient-based system for its spell costs. I'm sure that there has probably been a roguelike or two that did something similar. It DOES allow a possibility of setting up a system for assigning random spells to the various pairs of ingredients, which allows the player to try combining random ingredients to see what happens. This brings up another potential type of spell casting cost, risk.
Having risk as a cost means having the spell have a chance of not working how the player expects it to. Maybe it fizzles out and wastes a turn, maybe some of the parameters change unexpectedly so that you get caught in the blast radius, Maybe the target is changed, or maybe the spell just flat out blows up in your face.
If you want to go with a Lovecraftian horror theme, there's always the possibility of having magic cost you sanity. Just have the player experience various effects like seeing stuff that isn't there or having the names of their items change (maybe change "green potion" to "green poison") or having one monster appear to be another if they've been using magic a bit too much. While you're at it, change the descriptions to be more terrifying. And if you really want to scare the player, have the game not react to any key presses for a few seconds.
Rather similar to the sanity cost, but you could also try having a spell leave its caster with a status effect of some sort.