I like that you are getting away from the crazy DND AC. Also, removing randomness is a neat idea as well. I've played some games that did this, and it can help make the game more tactical and predictable.
You can also have different actions take different amounts of time (this would be a more offensive take). I suppose it depends on how you handle speed, but you could have agility reduce the time it takes to make a basic melee attack, quaf a potion etc.
Agility might not increase your speed per say, but it would increase your action speed to take a percent say 80% or something of your movement speed.
I have a really simple system of time right now. Basically given a players speed, it calculates the time it takes to act given that speed. It then gives that time to each monster on the level and they convert it to ap (action points). They then spend that, if they have enough to act until they no longer have enough ap to spend. I know there are a whole lot of implementations of time in roguelikes, but I think this way is super easy.
It would be pretty easy to make say an attack take 3/4's the time it takes to move or have a stat that reduces attack time while doing it in a more interesting way than angband variants, where you just attack multiple times in a row.
Granted, the player might not really *see* that attacks are taking 1/2 a turn or something that way. It might be hard to notice that type of mechanic. I guess it's debatable if that's a superior way. Angbands implementation where you just make several attacks at a time is certainly more noticeable to the player, although you lose the natural progression you could have otherwise and have break points where at X agility you get 3 attacks per round and at X + Y agility you get 4 attacks and between that there is no real point.
I can list the ways dexterity effects the game I am working on if that helps.
Base amount of armor weight before penalties (multiplied by a modifier based on armor skill, with armor weight you can have begin to have penalties to stealth/spell caster, dodge, unarmed damage, and speed starting at a set percentage of your base armor weight)
To hit (based on primary weapon stat), not sure how you would handle this though but you could have armor piercing, where your tohit limits the mitigation armor provides an enemy.
Damage (based primary weapon stat, certain weapon types gain damage bonuses from agility. A sword might get less dex bonus than a dagger and a mace might be pure strength)
Flat bonus to Dodge, you could do damage mitigation to ranged damage and certain spell types to avoid random factor. You could have a base dodge rate based on agility and have that modified by dodge skill.
Flat bonus to stealth (gains an indirect bonus from dex's effect on armor weight penalties)
Flat bonus to recharging a magic item.
For a dragon ball Z type game, DEF might work against physical attacks. And Dodge might provide mitigation vs certain types of ranged attacks. Actually you could have both, say DEF applies 3/4s mitigation vs physical and dodge provides 1/4 and vice versa.
So say you had 60% mitigation from dodge and 20% from armor. You might have 30% mitigation vs physical melee and 50% vs ranged attacks.
Also with speed, you really don't want to move 2x as fast as a monster, as you pointed out that's no challenge. But you want some method where speed can be any variable. So maybe you are moving at speed 25 and the monster is at 17. Under a system like that you get an extra 4th turn to the monsters 3 about. It gives a more natural progression. Also if you make a physical attack take a fraction of movement for both you and monsters, you can separate things out a bit. That would also lend to the type of mechanic where you might have a character with 2x speed, and 2x attack speed getting 4x attacks per round vs a slow tankish character that might do 4-5x damage. That would be cool.
Another thing you could do is have movement resolve for the player after giving enemies a chance to act, while attacking takes first priority. That would make it harder to abuse speed, as movement would resolve after the enemies use the time you give them. So your attacks would resolve prior to monsters having a chance to act, and movement would resolve post monster action. That would also resolve the movement abuse. So say you move 2x as fast, you'd attack, then move away. But the monster would have a chance to act before your movement resolves on your send turn. You'd end up a space away but the monster would have attacked as opposed to you moving a space away, the monster then not attacking and moving one space closer to you.
Of course, you have to ask do you want to get rid of movement abuse? If you have granular movement, some enemies might be half as fast as your character some might be 6/7ths as fast or 4/3rds as fast. When you run into a slower mob, is it wrong to abuse movement?
That's all I have for ideas on agility, I hope some of those are useful.