I've toyed with this idea so long that I think it's time to try it. I have a plan to use 32x32 tiles with 2x2 or 2x4 (tall) atomic parts to create tiles. Those parts can be stuff like ground, pebbles, grass, sand etc. Then using those parts it's possible to create a set of tiles. Dividing 32x32 tile to 16x16 with 2x2/2x4 atomics is easier and still small enough with variations to types of atomic parts. There is an endless way to combine atomic parts, but the tileset should stay reasonably small to keep memory consumption low. With atomic parts it's easy to create in-between tile types with different terrain and I also like the idea that you only need to draw simple stuff and build tiles from them which reduce the tedious work of drawing tiles.
I don't think I've ever seen this in any roguelike game, but I think it's an idea that has been discussed before.