In regards to the legalities of using a published, trademarked, copyrighted set of tabletop RPG rules as the basis for computer algorithms in a game...
As long as all of the following are true you're fine:
A) You do not use the trademark names in your game.
B) You do not copy published material into your game (as descriptions, background, etc).
C) You do not attribute your game as being 'based upon' or, to be safe, 'inspired by' said rule set.
D) No one has patented the algorithm(s).
Case A is trademark law.
Case B is copyright law.
Case C is cover your ass law because there are lawsuit happy idiots out there in table top RPG rule set land who have a history of suing computer game manufacturers.
Case D is the sad fact of life as a programmer in the USA today with the fubar'd software patent laws, and applies to every bit of code you write, game or not, and ignorance of any possible patent is not a legal defense.
All of that said, no one can get blood from a stone (without bashing someone's head in with it first). If you're doing a free game, worst that will generally happen is a 'cease and desist' order which, depending on what sites are hosting the game software for download, may result in the game no longer being available for download. If you're doing a commercial game, well be smart and incorporate, then at worst you can't lose anything more than the time you put in, the expenses incurred in development and publishing, and any profits you gained.
Do keep an eye out on your country's laws though. In the USA the laws are getting really screwed up as lots of money is being thrown about by the movie and music industries.
Finally, if you really think you have something to lose and are considering treading on dangerous ground, consult a real lawyer who specializes in software copyright and patent law.
