I haven't updated this page in quite some time, so I figured I'd
use this space to go over some of the most recent fixes with Sintendo
and how I finally managed to fix two of the most major problems
that have plagued Sintendo since nearly the beginning: Transparencies
and ROM loading.
Transparencies
This fix actually resulted from the need to fix another problem.
When Sintendo Beta v0.01 was first released on August 1st last year,
it was plagued with a rather major bug that prevented it from playing
all but about 8 ROMs. The next day, I realized the problem was happening
when transparencies were being displayed (but didn't know why),
turned of transparencies (it's quite simple in Snes9x) and put out
the fixed version the next day. Up until recently, I was uncertain
about the reason for this bug, but Sintendo ran pretty well without
transparencies (except for a few games that absolutely needed them
- most noticably Zelda and the introductory rain scene, as well
as a few other games where monster became completely invisible),
so I dealt with it.
Not long ago, I accidently came upon the realization that Snes9x
was drawing all of the SNES graphics data to the video memory of
the Dreamcast. This was done because there just simply wasn't enough
RAM to set up a properly sized buffer for video in the Dreamcast's
system memory. The problem is this: Dreamcast video memory, much
like many other sorts of video memory at present, can only be written
to. I know from graphics programming experience (and common sense)
that in order to render a transparency, first the destination pixel
must be read from video memory, then it is combined using alpha
arithmetic, and a new pixel is generated, about half of it (depending
on the transparency level) is based on the colors of the original
pixel, and the other half based on the color value of the transparent
overlay. So rendering in Dreamcast video memory was causing the
problem. Now the only problem was getting enough memory to set up
the large buffers.
It was during testing of the frameskip that I realized that I had
to move the video buffer to system memory instead of video memory
for reasons I'm not sure about (as they apply to frameskip). This
consumed a reasonable ammount of memory, but it still wasn't enough.
In order to render transparencies, a second buffer of the same size
is required (actually, a second and a forth, which I'll explain
in a moment).
In ideal circumstances, Snes9x requires 4 memory buffers to render
video: A screen buffer, a subscreen buffer (for transparency information),
a zbuffer (required for normal operation - has always been located
in system memory), and a subscreen zbuffer (for transparency z axis
information). The screen buffers both require 2 bytes per pixel
and the zbuffers require 1. At its current state, Sintendo uses
only the low-res mode of Snes9x (256x239) or 61,184 pixels. For
all buffers, 367,104 bytes are required, which doesn't seem like
a large ammount compared to the 16 Megs the Dreamcast has available,
but it didn't take me long to figure out that there just wasn't
enough when all was said and done.
Why? I did the calculations and figure out Snes9x says it takes
up about 8 Megs. WindowsCE about 2. The executable (SintendoDC.exe)
must reside in memory as well. At one point, SintendoDC.exe exceeded
over 2.5 Megs before I decided that the background bitmaps need
to exist in external files rather than as resources inside of the
executable. Right now, SintendoDC.exe is just over 900k, after being
optimized for size rather than speed, getting it down from just
under a Meg. It still didn't quite add up, but I figured I was being
some sort of vicious memory hog, started looking in vain for any
major leaks (ROM file names, bitmaps left in memory, anything),
but eventually gave up and went back to programming Sintendo under
some harsh memory constraints.
This is where the other problem came along. At some point late
in programming the Flash (VMU) saving/loading functions, I had to
create a function that checked the VMU to see if a file for the
game loaded existed. This function refused to work for me for some
reason, so I went to the old stand-by and checked the error message.
Not enough memory. Ouch. I've already been down this futile path
and came back empty handed (virtually), so at this point is when
I actually optimized Sintendo for size rather than speed, saving
me a cool 900k. Not enough memory. I'm not sure if the whole executable
actually resides in memory at any given point, but I knew something
was up here.
I went looking through the WindowCE documentation and found a neat
little function GlobalMemoryStatus() that nicely filled a structure
for me with all kinds of memory information. So I programmed this
function into a menu to display this information and ran this menu
at several key moments checking for memory consumtion. I followed
it through until emulation and discovered that I was running with
under 100k of free RAM by the time all was said and done. I rebooted
my good old Dreamcast and followed it through again, trying to see
where all of it was going. Nothing seemed out of the ordinary until
I ran through it again, discovering that one single function, DoSnes9xInit()
had reduced it from 12 Megs to 800k (rather than 8 Megs, like I
previously thought). Somewhere in there, I had lost a 0. I knew
what this function was and that it was, in fact, a giant memory
hog, but I wasn't so sure about 11 Megs! So I put a call to the
Memory manager between each function call inside DoSnes9xInit()
that allocated memory. It was a great surprise to me to discover
that Memory.Init() (an Snes9x function) was allocating over 10 Megs
of memory. Wait a minute? Don't the docs on Snes9x say it takes
up about 8 Megs? And to top all that off, do I really want to be
going through Snes9x's code looking for where I can clean up memory?
Then I remembered that horrible little FindFlashFile() function
and it's very certain lack of memory and realized I had to.
At first glance everything looked in order. All of the numbers
were in hex, so I had to break out the old Windows calculator to
convert them and I was coming up with numbers like 64k and 128k,
nothing anywhere near the 11 Megs this function was consuming, except
perhaps:
ROM = new uint8 [MAX_ROM_SIZE + 0x200 + 0x8000];
So I did a file search to figure out where MAX_ROM_SIZE was defined,
and what it was defined as, which ended up being:
enum { MAX_ROM_SIZE = 0xa00000 };
Calculator time again. A00000 hex = 10,485,760 decimal. You got
it. 10 Megs. So I asked my good friends on the #dcemu chat channel
if they've ever heard of a Super Nintendo ROM being 10 Megs
in size. Someone mentioned they had heard of a 7 Meg ROM, but never
10. I personally have only seen Chrono Trigger and Mario RPG, both
running at just over 5.
Whether or not there actually exist ROMs that are 10 Megs in size
or whether this is just a logical number based on how much an SNES
cart can actually store, I'm not sure. What I am sure of, however,
is that for most intents and purposes, 10 Megs of memory simply
isn't needed to load an SNES ROM. What I was even more sure of was
that this space could most certainly be used for a better purpose.
So, by changing one number, I managed to steal 2 Megs of memory
from Snes9x:
enum { MAX_ROM_SIZE = 0x800000 }; // Changed July 30, 2001 by Jamin Vander Berg
// enum { MAX_ROM_SIZE = 0xa00000 };
I commented out the old define and replaced it with my own, allocated
what I thought was a more than responable 8 Megs for ROM storage.
Through all of this, I had also learned another thing that surprised
me. Sintendo actually occupied very little system memory by itself.
Most of the memory Sintendo occupied was in those video buffers
I spoke of earlier. Besides that, I had actually been doing a good
job of cleaning up used memory and keeping Sintendo from being a
hog. And all this time I thought it was me.
So this being done, I tested out the FindFlashFile() routine and
everything went perfectly in order. Next, I uncommented all the
screen buffer allocation deep inside Sintendo and turned transparencies
on. I tried out (of course) Zelda first, and the results were astounding.
It ran a little slower because of the transparenies, so I had to
turn the frameskip up a notch to compensate, but even still Zelda
was running very smooth. What was really important, however, were
the graphics, which were simply astounding as I ran arround playing
in the beautiful rain.
All together, this problem that had plagued Sintendo for a year
minus 2 days was solved in only a few short hours - almost on accident.
It's wonderful how things fall together sometimes... :)
ROM Loading
Now this problem took me about two days of hardcore searching,
hacking, and testing to figure out - and that's not even fair to
the problem. If I actually took the time to figure out exactly how
much time I spent, how many sessions of brute force searching for
the problem, rewriting code two or three times; if I were to add
all those hours together, it wouldn't be pretty.
This problem began with Sintendo Build #209, in which I finally
added a running In-Game menu, which seemed to run perfectly until
the comments started pooring in. "When I go back to the game
menu and select a game, some of the time it doesn't load it and
just goes right back to the game I was playing". The key, very
evil, phrase in that sentence (at least if you're a programmer)
is "some of the time". Some of the time equates to an
unpredictable bug, one which can not be replicated every time, which
means it is the hardest to find. If a bug can be replicated every
time, then you know where it is, and you go to that peice of code,
look only at the peice of code and don't emerge until you've found
it. If a bug can not be replicated, it can potentially come from
anywhere. For exmple, a NULL pointer pretty much anywhere in the
code may sometimes over lap code or memory that the ROM loading
functions need. I did know, however, that it never happened the
first time you loaded a ROM, which mean the problem may be in the
In-Game menu, or perhaps the way in which the game menu was called
the second time, but not the first. Or it could be in the ROM loading
code itself, but why then did it never happen the first time?
These kind of problems are a disease. I rewrote the ROM loading
code at least twice and the problem remained. I looked through everything
until my eyes turned glossy time and time again. I'd go to sleep,
suddenly think of something it could be and pop out of bed to try
it out with no luck. I debugged through it several times, only to
find out that the reason it wasn't loading was due to a 'File not
found' error. But why? Obviously, when I set up the game list, the
file is there otherwise it would have never shown up in the first
place.
Something, somehow, got overlooked the first few times I debugged
through the code. I was checking the wrong filenames or something,
because everything seemed in order. When I tried to load, for example,
'Chrono Trigger.smc', the debug code listed the filename correctly,
and still game me a 'File not found' error. I beat at my head and
tried everything. Maybe I wasn't closing files properly and I was
running out of files. Maybe I wasn't closing directory handles properly
and loading them more than once. Anything... everything.
So today I discovered that the code I was trying to fix was never
getting run. A minor oversite that was quickly fixed, but the bug
still existed. 'File not found' it still said, so I checked the
filenames again. The first filename looked fine, and the second,
and then I got something like this: 'Chrono Trigger.smce Past.smc',
and then 'File not found'. Hmmm, it's pretty obvious why the file's
not found, but why does the filename look like that?
Its relatively obvious to anyone who knows C that the filename
string, stored in memory, is not getting terminated. That is to
say, any string in C always ends with a character 0, which isn't
actually a character, but a terminator to tell C that this is the
end of the string. Since the string was not getting terminated properly,
whatever was previously in memory (the last ROM filename) became
a part of the new filename. The terminator at the end of that string
(which was there only because the memory was zeroed first), was
when the string was actually ended.
So this is to say, the ROM loading problem was this: The first
ROM always loaded ok, because the memory was zeroed first, which
gave the string its own terminator by accident. The second time,
if the filename was longer, it worked in much the same way. If,
however, the string was shorter, overlap would occur and the filename
would be corrupted. This is why I could almost never get 7th Saga.smc
to load, but Zelda: A Link to the Past.smc loaded almost every time.
So I went through and zeroed out all my memory each time but was
still having the same troubles. But this problem never existed
on the PC version of Sintendo. Why? UNICODE. The filename string
is actually converted twice because of UNICODE compatibility. When
I get the list of filenames, they are stored as UNICODE strings,
two bytes per character. Snes9x and its Memory.LoadROM() function
expect character strings, or C strings, which are 1 bytes per character.
Memory.LoadROM() then calls S9xOpenFile() (a function I wrote) to
open the file. S9xOpenFile has to convert this string back into
UNICODE in order to properly interface with the OS. This is were
the problem was occuring. Looking at the following code:
#if defined UNDER_CE
mbstowcs(superstr, f, strlen(f));
#else
strcpy(superstr, f);
#endif
It is not obvious that the entire problem behind ROM loading lies
here unless you know exactly what you are looking for. It
is somewhat, obvious, however, once you know where the bug is, why
the PC version never had the same problem. The PC version doesn't
deal with UNICODE at all. Instead, under the #else part, the string
is just copied from one memory location to another using the very
standard, very well-known strcpy() routine. The mbstowcs() function,
however, converts a UNICODE string into a character string, and
is only run if UNDER_CE is defined. UNDER_CE is only defined if
the code is intended to be run on the Dreamcast.
Another not about mbstowcs() is that the third parameter is that
it is supposed to be the size of the buffer used to store the new
string, superstr, rather than the size of the source string, f.
In truth, the third parameter could have simply been strlen(f) +
1 and everything would have worked fine. Instead, it was only converting
the string until it reached the end of the string, not counting
the terminating zero, so the terminating zero was never copied over.
Changing this line of code over to:
mbstowcs(superstr, f, 255);
Made the world a better place for Super Nintendo junkies everywhere.
Now, the terminating zero was finally getting copied over and the
Dreamcast knew, properly, where the filename ended. Testing this
proved my theory was correct: ROMs load from file every time, correctly.
It still makes me a little sad to think exactly how much time I
spent trying to figure this problem out, and after all that it ended
up being such a simple problem. I suppose that the lesson to be
learned from both of these issues is that in programming, one number
can mean the difference between life and death. Whether it be that
you are allocating far too much memory or just one number away from
the difference between a string ending properly or it going on until
it felt like ending... This is not the first time I've run into
a problem that was solved simply by changing one number to another,
and it certainly won't be my last. I have learned from experience,
however, that these are typically, and by far, some of the hardest
to find. One number lying against a sea of 1s and 0s kindof puts
the idea of a needle in a haystack to shame.
Some final notes about transparencies and memory. While in the
PC word, 2 Megs or RAM is hardly worth spit, in the Dreamcast world,
this is little to scoff at. This RAM afforded me not only the ability
to check for Flash files and to enable transparencies; there is
still plenty left for all of the things I wished I could have done,
memory allowing. The current ROM limit per directory is 128. This
will soon be changed to a more reasonable 256. The SNES hi-res mode
may soon be supported simply by resizing the buffers (making them
about 4 times as large - which may in fact end up being too large),
and finally, when I get arround to it, there is plenty of memory
available to enable Save States, an idea I've been considering for
quite some time...
Until next time,
--Sin
|