(un)acceptable memory usage

[AgingMinotaur]

Here goes: Actually kind of a support question from a n00b

I've been working on this roguelike, (badly) written in python+pygame. I have no formal training as a programmer, instead learning as I go along, and focusing on effect rather than good programming habits. At the outset, it seemed a good idea to make "everything" (monsters, items, walls, etc.) instances of the same class. This does indeed provide me with a very flexible system, but it turns out to have its down sides. At the moment, I am concerned with memory usage. Even an "empty" instance of this class, where all variables (a little more than 70) are set to False, takes up about 5.5 kB of memory. Obviously I don't understand how python handles memory, since I can't understand why. Does the functions defined in the class (more than a few) take up space, or something like that?

A few days ago, a single dungeon level would take up more than 50 megabytes of RAM. That boggles the mind. Even after some ugly tweaks, a level with monsters, treasure and various features requires something like 5 mB. This leaves me with a potential application that will eat up hundreds of megabytes for a reasonably long game.

So I hope I might benefit from the wisdom of more seasoned programmers. Is it to be expected that python objects take up this crazy amount of memory? Are there some generic, nifty tricks to reduce memory usage that I should know of? Or simply a doc describing the basics of how python allocates memory (articles I find on the web, seem mostly to focus on particular cases)?

Any hints (or just friendly ridicule) would be greatly appreciated.

As always,
Minotauros

[Nathan Stoddard]

I don't think it's too unreasonable for a dungeon level to take 5 MB. It is probably higher than average, but as long as you only keep one dungeon level in memory at once (saving the rest to a file), it should be fine.

I'm not familiar with Python, so I'll let someone else explain what is probably taking up so much memory.

[magellan]

False or True doesnt matter for the memory usage. In one case you store 1 in a byte, in the other 256 or whatever. All cases use of course one byte of memory. I am not familiar with python but you might be able to shave off a bit by looking at the ranges you need for your variables, and use the according types. (A flag that tells you 1 or 0 doesnt need to be a double etc.)
Another way to reduce memory usage would be to not hold everything in memory, and store what is not in use right now on the HD.

[Anvilfolk]

Also, if you're using pygame, it is based on SDL. I'm reckoning the images you're displaying need to be stored somewhere. Even if you're just using ASCII, I think it might render it to a bitmap. Bitmaps can take absurd amounts of space. For example, a 800x600 pixel screen, with typical 32bit per pixel representation will take up almost 2 megabytes.

On another note, and if you're using bitmaps and the like, don't forget to store them in a single place. For example, ground tiles can be represented by a simple dot. If you render this dot for each ground tile, you'll use up a lot of memory for bitmaps. If you render it once, and then draw that single bitmap several times in different places, you'll be saving a lot of memory. Don't know if you're doing it this way.

This can be widened to other things other than bitmaps. For example, a TerrainTile class could be instantiated once for each type of terrain (wall, ground, water, etc), defining attributes such as whether it's passable, if it's an obstacle, base movement speed over it, and some other things (not position). Then each ACTUAL, positioned tile (with an X, Y position) only has to point to these terrain information classes, and not maintain a copy of that information (which would lead to duplicates/redundancy, increasing memory usage).

I'm never sure if I explained stuff in a sensible way, so feel free to ask any questions at all! I'd love to help further!

[Krice]

How did you measure that memory consumption?

[AgingMinotaur]

Thanks for your answers. I was afraid the post was going to be a dud.

Writing the data to disk... that's very clever. Why didn't I think of that?

@Anvilfolk: That's sensible stuff, and in fact I did consider just what you are suggesting a while back, so each tile is loaded only once, when the game starts. And making the grid coordinates point to, instead of copying, floor & obstacle Beings is exactly what I did to get reduce memory consumption by 45 megabytes per level.

@Krice: I used the python console to measure how much memory an instance uses. Something like:

import beings # critters etc. defined in this module
l=[]
for i in range(0,1000):
   tmpbeing=beings.Being() # creates an "empty" class instance
   l.append(tmpbeing) # and watch memory consumption skyrocket

As always,
Minotauros

[Slash]

Yeah, I think the most important advise here is to NOT CREATE A NEW OBJECT INSTANCE FOR EACH CELL IN THE MAP. Instead, create one of each kind and make the structure point there. This will take away from you some flexibility (For example, if you want cells to be modifiable, you can't change the properties of a cell because it would affect all of the same kind), however this can be worked around with some "magic".

Also, a wise usage of inheritance means you will only have meaningful fields in your objects; if you use only one class with all the possible fields, you'll end up with half of the fields not being used for one kind of entity.

[Vanguard]

Yeah, I think the most important advise here is to NOT CREATE A NEW OBJECT INSTANCE FOR EACH CELL IN THE MAP.

Is this your advice for AgingMinotaur specifically, or a general rule everyone should follow?  In my own project, I was planning to make each tile on the map its own instance.

[Ex]

Yeah, I think the most important advise here is to NOT CREATE A NEW OBJECT INSTANCE FOR EACH CELL IN THE MAP.

Is this your advice for AgingMinotaur specifically, or a general rule everyone should follow?  In my own project, I was planning to make each tile on the map its own instance.

I've made every map tile an instance of a complex object in C++ without problems. But this is C++, and the memory usage in such an event is highly dependant on the complexity of the object used, and the size of maps. For instance, an 80x25 map is not going to take nearly as much memory as a 100x100 map. But in C++, generally the memory usage is extremely small even for maps of fairly complex objects. Just need to remember not to store graphical data in each tile

The best solution IMO would be to have a very, very small class that either points toward a common set of objects, or holds a copy of any one of those common objects. Thus, each map tile starts out pointing toward a set of more complex object, and if any map tiles need alteration the map tile creates a copy of the object it points to and points to the copy instead. The big thing here would be to remember to delete all copied objects on exit (Unless you have automatic garbage collection..), and also to save all copied objects.

[Anvilfolk]

Yeah, I think the most important advise here is to NOT CREATE A NEW OBJECT INSTANCE FOR EACH CELL IN THE MAP.

Is this your advice for AgingMinotaur specifically, or a general rule everyone should follow?  In my own project, I was planning to make each tile on the map its own instance.

This is just programming good practice. Generally speaking, you should never have duplicate information. There no point! Just pass a pointer to where the data is, and you have the same functionalities without occupying more space.

You won't get away from needing a different instance of a class in each tile. You'll need it to store what objects are in each tile. The point here is that there's other, more general and reusable kinds of information (graphics to display, movement bonus over tile, etc) that can be stored elsewhere only ONCE, and then pointed to. Graphics is just a bigger issue because it takes up enormous amounts of memory.

In generic object-oriented programming,  this would amount to something along the lines of:

Code: [Select]

class TerrainType:
  int movementBonus;
  graphicsStuff image;
  boolean passable;
  ...
This class would have several instances, say, HillyTerrain, MountainousTerrain, GrassyTerrain, etc. Then, you have

Code: [Select]

class MapTile:
   TerrainType* type;
   List items;
   List monsters;
   ...
Or something along these lines. In case you're not familiar with the * pointer notation, it just means that there doesn't need to be a different instance. You can create just the one, and have several pointers pointing at it. Say, 20 tiles will point towards HillyTerrain, and you won't need 20 HillyTerrain instances, just one. Pointers are much smaller in size than classes, so you gain!

I hope this makes it a bit clearer and "down to earth"

[Vanguard]

That's helpful.  I wouldn't have thought of that.  Thanks!

[Jotaf]

Ya know, in Python land, if the tiles are stored in a list, you can "use pointers" with very little modification. Just make sure you create one instance of a tile type that you want to reuse -first-, like wall=Foo(), and then just merrily set it somewhat like this: mylist[i ]=wall. In the case of nested "2D" lists as is usual with maps, that would be map[x ][y]=wall. Python's copy semantics, unlike other languages like C++, won't make a different instance for each, instead they will all point to the original "wall". On the other hand, if you had map[x ][y]=Foo(), THAT would make a different instance for each tile.

BTW you can always check an instance's "true identity" / memory address using the function id(). So when you're not sure just print some id()'s.