Heheh, "mana hacking".
The concept appears similar to an idea I've had for procedurally generated objects (in particular, NPCs and items). If you break down an object into its components, or characteristics, random objects can be constructed by assigning random values to each characteristic. The same thing could apply to spells.
If you manage to come up with a set of sensible characteristics for spells, and assign meaningful values to them, you could allow players to compose spells according to a predetermined syntax. The easiest encoding I can come up with is a simple list of values, in which every position represents a characteristic. For example:
Position 1: mana consumption; determines spell power.
Values: 1 - 9, representing for how many magic points the player wishes to cast the spell.
Example: cast a Fire spell for 1 MP to light a cigarette; cast same spell for 4 MP to burninate a peasant.
Position 2: target
Values: 1 (target PC), 2 (target NPC), 3 (target wielded item), 4 (area effect)
Example: cast a Frost spell with target 2 to freeze an NPC; cast same spell with target 1 if you're on fire; cast same spell with target 4 to freeze nearby water.
Position 3: spell type
Values: 1 (conjuration), 2 (enchantment), 3 (divination) etc.
Example: cast a Fire spell of type 1 to throw a fireball, or type 2 to light an area, or type 3 to reveal unseen heat sources on the map.
Position 4: effect; allows the caster to reverse the effect of a spell.
Values: 1 (positive effect), 2 (negative effect)
Example: cast a Heal spel with effect 1 to heal wounds; cast same spell with effect 2 to inflict pain.
Position 5: element; determines the spell's elemental flavour.
Values: 1 (air), 2 (water/ice), 3 (earth), 4 (fire)
Example: cast a Bolt spell with element 4 to hurl a fireball, or with element 2 for a bolt of frost.
These are just silly examples, but you get the idea. A spell would be encoded as a five digit sequence. A spell "12112" could be used to target a small frost bolt at a NPC; "64313" would reveal the entire map; accidentally casting "33221" would curse the weapon you're wielding.
This example is too simple to accommodate or encode spells in such a way that they make sense. You'd have to define the common characteristics for your spells carefully, and come up with sensible values for each characteristic. Once you have those, you'd probably have to come up with a better grammar than a string of digits. One game that did this quite nicely is Loom. It allowed the player to cast spells by playing notes. Although most spells were pre-defined sequences, there was some logical arrangement; e.g. playing a tune backwards would produce the opposite effect.
Finally, you don't want to code the effects of every possible combination. You'd have to define as many generalisations as possible, although there are bound to be lots of specific exceptions. Similar spells could probably be handled as groups. There could be a general rule that higher mana consumption would make the spell more powerful, but more difficult to cast. And for nonsensical sequences, the spell parser could decide to produce some harmless random side effect based on some values in the input.