Author Topic: Debugging and other helpful things  (Read 26288 times)

Kyzrati

  • 7DRL Reviewer
  • Rogueliker
  • *
  • Posts: 508
  • Karma: +0/-0
    • View Profile
    • Grid Sage Games
    • Email
Re: Debugging and other helpful things
« Reply #15 on: May 16, 2012, 03:23:07 PM »
Most vital: An AI that can mimic the user and be run indefinitely to test the program automatically.
Isn't that kind of hard to do?
Most certainly. I could've added that bit, but it seems obvious ;D It's only something to do if you love working with AI, and have lots and lots of free time... If anything, you can always write a simple AI that at least plays your own game using completely randomized decision-making, and run it in a loop. It works pretty well and isn't that hard; though the precise difficulty would be a function of the depth of your game, starting it early and expanding it as each new feature is added is actually quite trivial. Just have to be religious about updating it, otherwise it's not nearly as useful once the game reaches beyond its scope.

Another very helpful feature is to be able to toggle the random number
generation to be simple or noisy.
Or just reuse the same seed, if the system is automated to begin with. (Another advantage of AI-driven playtesting!)

ant

  • Newcomer
  • Posts: 25
  • Karma: +0/-0
    • View Profile
    • anthive
    • Email
Re: Debugging and other helpful things
« Reply #16 on: May 16, 2012, 09:28:02 PM »
Quote from: Kyzrati link=topic=2362.msg18423#msg18423 date=133718178
[quote author=ant link=topic=2362.msg18401#msg18401 date=1337145786
Another very helpful feature is to be able to toggle the random number
generation to be simple or noisy.
Or just reuse the same seed, if the system is automated to begin with. (Another advantage of AI-driven playtesting!)
[/quote]

  For the code I'm currently messing with I can do that, but it
still has enough things going on in the code to generate more
random numbers that it still will not reliably reproduce the
error/event I'm after.  What I need to do is to go in there and
find all the random noise making and put a flag around them
to turn them off for testing.

  Like it has been said though it is fun to watch even a weak
AI attempt to beat a game.  Running it in a loop, catching the
logs and looking for error messages is a great way to exercise
the code and it can also come up with combinations of events
that might not have been originally considered in the game
design.

  Another helpful feature on that front is a random command
sequence generator to feed as input so that you can test the
command interface to handle garbage input gracefully.

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #17 on: May 17, 2012, 11:30:31 AM »
Really nice ideas there. I have to try that AI bot thing at somepoint and see what I need to get it up and running. What kind of interface do you have for it Kyzrati? Like, is the bot running inside the game and interfacing it directly or do you have it as an external program that connects to the game in order to play it?

I guess I could expand my debug server to serve json or something equally simple that could describe what the bot sees and then accepts commands from it that it executes. I think it should be fast enough for turn based game like this.

I'm mostly using the random number generator created in the beginning of the program, but I think there are still some places where a new one is instantiated. This makes the game non-deterministic, since current system time is used as seed. I'm slowly removing those from the code though, so after that I can try out running games with same seed.
Everyone you will ever meet knows something you don't.
 - Bill Nye

Kyzrati

  • 7DRL Reviewer
  • Rogueliker
  • *
  • Posts: 508
  • Karma: +0/-0
    • View Profile
    • Grid Sage Games
    • Email
Re: Debugging and other helpful things
« Reply #18 on: May 17, 2012, 02:43:11 PM »
What kind of interface do you have for it Kyzrati? Like, is the bot running inside the game and interfacing it directly or do you have it as an external program that connects to the game in order to play it?
I create them as part of the game, using the same class that the player controls, so the game doesn't necessarily have to even know the difference. In previous games I've gone so far as to have both the player and the AI use the exact same command system to interface with the logic, but in my current project they're not completely the same since I found that a bit cumbersome and limiting in some situations. Also, after spending years working on AI, I've more recently gotten a bit lazy about making sure it covers absolutely everything so I can put more effort into game design instead.

I'm mostly using the random number generator created in the beginning of the program, but I think there are still some places where a new one is instantiated. This makes the game non-deterministic, since current system time is used as seed. I'm slowly removing those from the code though, so after that I can try out running games with same seed.
Even after you make sure everything's centralized, you can still use the system time, of course; just make sure the save the seed it results in each time in case you need it/them again.

purestrain

  • Rogueliker
  • ***
  • Posts: 172
  • Karma: +0/-0
    • View Profile
Re: Debugging and other helpful things
« Reply #19 on: June 01, 2012, 10:29:28 AM »
Um... is there really a reason to write a human-like AI instead of passing some predefined actions to the player controller in a specific a scenario?

Once you change a single part of the AI to match for a specific solution you can't guarantee that all other cases are not touched - but that is just my opinion.

i prefer the following:

Code: [Select]
@Test
public void shouldTriggerIfPlayerStartsThereButMovesBackAndForth() {
Game testGame = Game.newGame(database, new GameBuilderTestImplementation(false, 0));

Entity player = testGame.getPlayer();
CustomController controller = new CustomController();
controller.addAction(new IdleAction());
controller.addAction(new MoveAction(-1, 0));
controller.addAction(new MoveAction(1, 0));
                controller.addAction(new OpenDoorAction(1, 0));

player.setController(controller);

LocationTrigger toAdd = new LocationTrigger(testGame);
toAdd.setLocation(new Point(2, 1));
toAdd.setActivateCallback(new ActivateCallbackImplementation());
testGame.getCurrentZone().addTrigger(toAdd);

testGame.getLoop().rebuild();
while (testGame.step())
;

Assert.assertEquals(true, wasTriggered);
}

Kyzrati

  • 7DRL Reviewer
  • Rogueliker
  • *
  • Posts: 508
  • Karma: +0/-0
    • View Profile
    • Grid Sage Games
    • Email
Re: Debugging and other helpful things
« Reply #20 on: June 01, 2012, 11:57:07 AM »
Other than the benefit of being able to use it as an opponent, using it to test and debug is going to give you a lot more coverage than feeding specific actions. I'm not referring simply to code coverage here, but situation coverage.

Testing basic actions like moving and opening doors is not what it's for--it's for highly complex games (like expansive RLs) with a huge array of interconnected systems where there are more or less an infinite number of possibilities, any of which could, under special circumstances, combine to create a unique situation that code was never intended to deal with. It could be very difficult to find some hypothetical emergent bug that only occurs when an entity under the effect of a certain spell is hit by the special effect caused by a certain weapon carried only by a certain enemy, or even some other situation which could require an even greater depth of specialization. Using an AI on this kind of game can save a lot of testing time. Obviously if you can foresee every possibility then an AI would be unnecessary, yes, but once you start adding system upon system to a complex game with development spanning over many years, I'd wager it's just about impossible to predict everything.

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #21 on: June 02, 2012, 07:37:55 AM »
I think both ways have their merits. More than often it's good to have full control of what actions are taken during test, while sometimes (especially when testing how AI behaves), it is useful to give it a free reign.

Watching AI to play the game can be interesting, but I personally probably wouldn't use it to test the game. I like my tests small, fast and running continuously when I code. But I could probably use it to learn more about the game and how different AIs react to each other and observing if there are subtle bugs that tests don't cover yet (and then make sure that there is test for that subtle bug too when one is found).

I have been playing around on idea of writing some high level helpers for testing and especially for cases when I'm defining how things interact on a high level (ie. get hit by a monster, your hit points should go down, drop a fireball on top of potions, potions should boil and explode). Intent of following test is probably easy enough to understand, even if one reading it hasn't done much (if at all) programming:

Code: [Select]
def test_that_hitting_reduces_hit_points(self):
        """
        Getting hit should reduce hit points
        """
        Pete = strong(Adventurer())
        Uglak = weak(Goblin())
        level = Level()

        place(Uglak, middle_of(level))
        place(Pete, right_of(Uglak))

        make(Uglak, hit(Pete))

        assert_that(Pete, has_less_hit_points())

This is of course a lot simple case than what purestrain showed. I don't have game loop running (since I'm doing a single action). Eventually I need to figure out how to test those things too (making multiple actions and maybe having multiple characters to act even). Having queues of actions could be one solution for this.
Everyone you will ever meet knows something you don't.
 - Bill Nye

Kyzrati

  • 7DRL Reviewer
  • Rogueliker
  • *
  • Posts: 508
  • Karma: +0/-0
    • View Profile
    • Grid Sage Games
    • Email
Re: Debugging and other helpful things
« Reply #22 on: June 02, 2012, 07:50:27 AM »
I think both ways have their merits. More than often it's good to have full control of what actions are taken during test
Agreed, though personally I like to do the simple tests manually rather than automate them. This is made much easier by being able to take control of any entity at any time. So even if you're happily playing along in a normal game and something doesn't quite look right, just switch over control and check it out.

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #23 on: August 25, 2012, 07:21:45 AM »
Returning to an old thread..

I have been playing around with some tools and libraries again and found two that I like.

First is behave, small tool/library for behaviour driven development. It allows me to write tests with almost natural language that is much easier for non-programmers to read. Below is two simple tests/specifications:

Code: [Select]
Feature: Combat
  as an character
  in order to kill enemies
  I want to damage my enemies

  Scenario: hit in unarmed combat
     Given Pete is Adventurer
       And Uglak is Goblin
       And Uglak is standing in room
       And Pete is standing next to Uglak     
      When Uglak hits Pete
      Then Pete should have less hitpoints

  Scenario: hit in melee combat
     Given Pete is Adventurer
       And Uglak is Goblin
       And Uglak wields dagger
       And Uglak is standing in room
       And Pete is standing next to Uglak     
      When Uglak hits Pete
      Then Pete should have less hitpoints

Output:

Code: [Select]
1 features passed, 0 failed, 0 skipped
2 scenarios passed, 0 failed, 0 skipped
13 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m0.0s

If I were to work with something more complex than roguelike and had a team of programmers and non-programmers, this could be really useful tool. Now it's fun tool to play around and experiment.

Another one I tried briefly is quickcheck. It allows me to wrap test functions and feed random test data into them:

Code: [Select]
    @forall(tries = 5,
            loc_x = integers(low = 1, high = 19),
            loc_y = integers(low = 1, high = 19))
    def test_dropped_item_added_to_correct_location(self, loc_x, loc_y):
        """
        Test that dropped item is added to correct location
        """
        self.character.location = (loc_x, loc_y)
        self.item.level = None
        self.item.location = (0, 0)
        self.character.inventory = [self.item]

        self.character.drop_item(self.item,
                                 self.action_factory)

        assert_that(self.item.location,
                    is_(equal_to(self.character.location)))

Nose (test framework I'm using) detects that function automatically based on its name and executes it. It doesn't know however, that it has been wrapped with @forall decorator that causes the test to be run 5 times, with random data for loc_x and loc_y parameters. This can be useful if you suspect that the function might be working differently with different parameters. Because of the way quickcheck is written, it can even made to generate random complex objects for testing purposes.
Everyone you will ever meet knows something you don't.
 - Bill Nye

purestrain

  • Rogueliker
  • ***
  • Posts: 172
  • Karma: +0/-0
    • View Profile
Re: Debugging and other helpful things
« Reply #24 on: August 28, 2012, 06:11:05 PM »
Woot; thanks. I'll try that out too (JBehave) - Already heard about BDD, but never took a further look at it until now.

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #25 on: September 01, 2012, 01:13:24 PM »
What I like about using behave is that it lets met to separate things I'm working to two levels: long term goals and immediate goals. If I'm adding a new feature to the game, I usually try to express it as a behave specification/test. Then I can start working towards that and keep adding unit tests as I go. I don't mind if I have failing test written with behave, but I mind if I have a failing unit test (since it prevents me from commiting my changes to version control).

Let us know how you feel about using JBehave. I haven't used that one, but it looks really similar to behave (even more than SpecFlow for example).
Everyone you will ever meet knows something you don't.
 - Bill Nye

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #26 on: November 18, 2012, 09:14:34 AM »
I have been playing around with the interface and testing recently. I'm not doing any checks for layout or aesthetics, since those can be done by humans much more better than by computer (at least if I'm the one coding). But I figured out neat way to check if the correct information is shown to player and if basic functionality of a dialog is working.

Following snippet creates a character, who has cure minor wounds spell cast on him (that heals player slowly over couple of turns). It then creates a widget (control) that can show player's stats, spells being among them. Last two lines checks that there are two strings present at the dialog: name of the spell and short description.

It doesn't matter where those two strings are present, as long as they are there. This is just a quick check that the dialog can show the information to player. Human can then do the more complex testing if required.

Code: [Select]
def test_displaying_effects(self):
    """
    Displaying character should show effects
    """
    character = (CharacterBuilder()
                            .with_body(5)
                            .with_finesse(6)
                            .with_mind(7)
                            .with_effect(EffectBuilder()
                                                    .with_title('Cure minor wounds')
                                                    .with_description('Cures small amount of damage')
                                            )
                            .build())

    surface_manager = mock()
    when(surface_manager).get_icon(any()).thenReturn(QPixmap())

    widget = CharacterWidget(surface_manager = surface_manager,
                                         character = character,
                                         parent = None)

    assert_that(widget, has_label('Cure minor wounds'))
    assert_that(widget, has_label('Cures small amount of damage'))

Second test has a character standing in the same location with a dagger. Test creates inventory widget that shows the player's equipment, inventory and items laying on the ground. It then locates control which has dagger in it and clicks it. Final step is to check that the dagger is no longer laying on the ground.

What I like about the second example is how I don't have to tell exactly where to click, the test can figure it out by itself (within limits, of course). If I were to move things around on the screen, it should still work, as long as the basic idea stays the same.

Code: [Select]
def setup(self):
    """
    Setup test case
    """
    self.application = QApplication([])
    self.surface_manager = mock(SurfaceManager)
    when(self.surface_manager).get_icon(any()).thenReturn(QPixmap())

    self.action_factory = (ActionFactoryBuilder()
                                .with_inventory_factory()
                                .build())
    self.level = (LevelBuilder()
                        .build())
    self.character = (CharacterBuilder()
                            .with_level(self.level)
                            .with_location((5, 5))
                            .build())

def test_picking_up_item(self):
    """
    Test that item can be picked up
    """
    item = (ItemBuilder()
                .with_name('dagger')
                .build())
    self.level.add_item(item, (5, 5))

    dialog = InventoryDialog(surface_manager = self.surface_manager,
                             character = self.character,
                             action_factory = self.action_factory,
                             parent = None,
                             flags = Qt.Dialog)

    QTest.mouseClick(satin.widget(dialog,
                                  slot_with_item('dagger')),
                     Qt.LeftButton)

    assert_that(self.level, does_not_have_item(item.name))
Everyone you will ever meet knows something you don't.
 - Bill Nye

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #27 on: December 12, 2012, 07:55:24 AM »
Been playing with new toys again, this time with IPython, which is enchanced interactive Python shell. I have only scratched the surface, but already fell in love with the system.

IPython is really great for prototyping and experimenting with the data. Below I have dump of an interactive console, where I start up part of my game and generate a level. After generating one level, I change some of the parameters and generate a new level to see how they affect the end result. I also coded a whole new room generator interactively and checked while coding what kind of results it produces. Sadly I don't have logs from that at hand.

I didn't get rendering of my levels to work yet, but that shouldn't be too far anymore. Then I could display what the level actually looks in game with tiles and everything, instead of ascii representation.

I had fun playing briefly with this on a browser, which has the added benefit of leaving page that others can view and edit too. Might be interesting to use it for fast prototyping and collaborating with friends.

Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
%guiref   -> A brief reference about the graphical user interface.

In [1]: import os.path
   ...: from herculeum.config import Configuration
   ...: from pyherc.data.model import Model
   ...: import sys
   ...: import logging
   ...: import herculeum.config.levels
   ...: import herculeum.gui.resources
   ...:
   ...: from herculeum.gui import MainWindow, HelpProvider
   ...: from PyQt4.QtGui import QApplication
   ...: from PyQt4.QtCore import QFile, QLatin1String, Qt
   ...:

In [2]: qt_app = QApplication([])

In [3]: world = Model()
   ...: config = Configuration('', world, herculeum.config.levels)
   ...: config.initialise()
   ...:
Initialising generators
Generators initialised
Initialising level generators
Level generators initialised
Initialising action sub system
Action sub system initialised

In [4]: level_generator_factory = config.level_generator_factory
   ...: level_generator = level_generator_factory.get_generator('upper crypt')
   ...:
   ...: level = level_generator.generate_level(None)
   ...:

In [5]: level
Out[5]:

#################################################################################
########################################################........#################
########################################################.######.#################
####............#########..*X..##############........*##.######.####.....########
####............#########...........######...............######..........########
####...X........#########......####........##.........##############.....########
#########.###############*.....##################.##################X....########
########..#################.##############........###############################
########.#############......##############.######################################
########.#############.###################.######################################
########.#############.###################.######################################
########....##########........############.........##############################
###########.#################.####################.##############################
########........#.......####...################..*....############.......X.######
########..........#####........################.......############...*.....######
########........############...################.......############........*######
###########.###################################.......################.##########
###########.######################################.##############......##########
###########.######################################.##############.###############
###########.######################################.##############.###############
###########.######################################.##############.###############
#########...###################################....##############.....###########
#########.#####################################.#####################.###########
########...####........##...........#########......##############.........#######
########........######..............#########.....*.....######............#######
########*..##############..X........#########......####........##...*.....#######
########...##################################......##############################
#################################################################################
#################################################################################
#################################################################################
#################################################################################

In [6]: from pyherc.generators.level.partitioners.grid import GridPartitioner
   ...: from random import Random
   ...:
   ...: new_partitioner = GridPartitioner('upper crypt', 2, 2, Random())
   ...:
   ...: level_generator.partitioner = new_partitioner
   ...: new_level = level_generator.generate_level(None)
   ...:

In [7]: new_level
Out[7]:

#################################################################################
#################################################################################
#################################################################################
###################################################........................######
###################################################......................*.######
###################################################........................######
############..........#############################........................######
############..........########..............................X........*.....######
############...................####################......*......X..........######
############..........#############################........................######
############..........#############################.....X..................######
############.....*....#############################........X............*..######
################.################################################################
################...##############################################################
##################.##############################################################
##################.##############################################################
##################...############################################################
####################.############################################################
################..........######...........######################################
################.....*....######.#########.###........................###########
################..........######.#########.###.....X..................###########
################..........######.#########..........................X.###########
################.................#############.....X..................###########
################..........####################*.......................###########
################..........####################........................###########
################..........#######################################################
################..........#######################################################
#################################################################################
#################################################################################
#################################################################################
#################################################################################

In [8]:
Everyone you will ever meet knows something you don't.
 - Bill Nye

MorleyDev

  • Newcomer
  • Posts: 5
  • Karma: +0/-0
  • "It is not enough for code to work."
    • View Profile
    • http://www.morleydev.co.uk/
    • Email
Re: Debugging and other helpful things
« Reply #28 on: December 12, 2012, 05:27:37 PM »
Tests glorious tests. The tricky part is figuring out where the problem is so I can put a unit test around it and fix. If it isn't obvious, Integration tests, debug logging and if all else fails step through. Then it's isolate the parameters that cause the erroneous behaviour with failing unit tests, then fix.

Though none of the existing unit testing or mocking libraries I found in C++ ever felt right or fit my needs and preferred testing style quite right. But I'm growing my own in my latest projects, taking advantage of C++11.
« Last Edit: December 12, 2012, 06:10:08 PM by MorleyDev »
All code is guilty until proven innocent, unworthy until tested, and pointless without singular and well-defined purpose.

tuturto

  • Rogueliker
  • ***
  • Posts: 259
  • Karma: +0/-0
    • View Profile
    • pyherc
Re: Debugging and other helpful things
« Reply #29 on: December 12, 2012, 09:27:32 PM »
Though none of the existing unit testing or mocking libraries I found in C++ ever felt right or fit my needs and preferred testing style quite right. But I'm growing my own in my latest projects, taking advantage of C++11.

Have you had a look at Googletest http://code.google.com/p/googletest/? I have heard many good things about it, but haven't personally tried it. It might give ideas if nothing else.

I'm really happy with the testing tools that I have currently in my disposal. I'm mostly using existing tools (nose, nosy, mockito, hamcrest and QTest) and had added just a little bit sugar coating here and there to make things easier to read. Testing C++ is a whole different beast than testing Python code though!
Everyone you will ever meet knows something you don't.
 - Bill Nye