I posted the Zelda and Metroid analyses because I don't think we really have a firm grasp on what makes such hand-crafted level design good. I don't think the analyses are perfect but they are at least a start on the road to understanding the problem.
I think the art of level design might be to railroad a player down a path, but have him be completely oblivious to the fact that he's being railroaded. This falls apart if you look at sequence breaking in the metroid games, or maybe it's a testament to how solid the original path choice is that the game can still be completed if you forcefully go off the tracks.
Quite a bit of your rant is essentially you shouting NO BUT YOU'RE WRONG though. The Zelda analysis basically says that placing necessary items off the critical path is bad, which is a debatable point. No one is complaining about heart containers or other unnecessary upgrades being off the beaten track.
The metroid articles points out that 'dead-ends' always have items (optional upgrades) in there. Makes sense if you want to encourage exploration but it's also something that roguelikes don't really do except by random chance.
I tend to do that, yeah
You make a good point about whether or not having
necessary items off the critical path is good or bad. I feel, for a game like Zelda 1, that it's perfectly fine. There being a critical path is nice, and finding out that enemy placement on it is a conscious choice is impressive, but they DID go through the trouble of making the rest of the dungeon too. That a critical path was discovered for the final dungeon of Zelda 1 is nice, but it doesn't sit well with me that he complains about the silver arrows being in an out-of-the-way location. For two reasons mostly:
1) The dungeon is shaped like a skull! You need to get the map and go through some rooms before that starts being apparent to the player. Some players will notice it sooner, others will notice it only after they've been in almost every room. But once you notice it (speaking from MY experience) you'll be filled with a small sense of awe at the detail. So that's one of the reasons it pays to explore the full dungeon.
2) The silver arrows are still IN the final dungeon itself. This is a key element, and I don't really feel it needs explaining.
As an aside, having rooms off the critical path also serves as a bit of a gesture towards the players. Not everyone has the skill to beat the game without taking damage, so these optional rooms allow you to encounter more monsters to kill for more rupees (arrows), bombs or hearts. This gives the player a bit more room for error ON the critical path itself, which is nice.
As for roguelikes, I feel that they actively discourage exploration if anything. Most have a clock of some kind running, whether it's a food clock, ADOM's corruption clock, or Crawl's monster respawn clock, that urges the player to MOVE ALONG. The resources you have and other informed decisions the player makes determine exactly how FAST you move along (ie whether you explore the current floor a BIT more before descending) but in general I don't think there's many roguelikes that encourage the player to full explore EVERY floor. And there CERTAINLY isn't a roguelike that has random stairs UP!
That would be the true form of exploration that is missing from roguelikes.
Add: Dangit, I forgot completely that that's more or less exactly what Crawl does with its branches and mini-branches. So yeah, go Crawl!
Zelda 1 had far more free roaming than say Dark Souls or Super Metroid but LttP is pretty much on par with Dark Souls for openess. Also Dark Souls gives far more exploration opportunities than say Brogue. You can do the bells in either order (even missing out entirely on the Depths and Blighttown), then Sen's and Anor Londo then the 4 Lord's Souls in any order. One of the Lord's Souls can be obtained right at the start of the game.
Exactly so, but you still have to do both bells before Sen's, Sen's before Londo, and Londo before Lord Souls. Which is more or less exactly like how LttP paces the game, yeah.