I haven't updated this page in quite some time, so I figured I'd use this space to go over some of the most recent fixes with Sintendo and how I finally managed to fix two of the most major problems that have plagued Sintendo since nearly the beginning: Transparencies and ROM loading.

Transparencies

This fix actually resulted from the need to fix another problem. When Sintendo Beta v0.01 was first released on August 1st last year, it was plagued with a rather major bug that prevented it from playing all but about 8 ROMs. The next day, I realized the problem was happening when transparencies were being displayed (but didn't know why), turned of transparencies (it's quite simple in Snes9x) and put out the fixed version the next day. Up until recently, I was uncertain about the reason for this bug, but Sintendo ran pretty well without transparencies (except for a few games that absolutely needed them - most noticably Zelda and the introductory rain scene, as well as a few other games where monster became completely invisible), so I dealt with it.

Not long ago, I accidently came upon the realization that Snes9x was drawing all of the SNES graphics data to the video memory of the Dreamcast. This was done because there just simply wasn't enough RAM to set up a properly sized buffer for video in the Dreamcast's system memory. The problem is this: Dreamcast video memory, much like many other sorts of video memory at present, can only be written to. I know from graphics programming experience (and common sense) that in order to render a transparency, first the destination pixel must be read from video memory, then it is combined using alpha arithmetic, and a new pixel is generated, about half of it (depending on the transparency level) is based on the colors of the original pixel, and the other half based on the color value of the transparent overlay. So rendering in Dreamcast video memory was causing the problem. Now the only problem was getting enough memory to set up the large buffers.

It was during testing of the frameskip that I realized that I had to move the video buffer to system memory instead of video memory for reasons I'm not sure about (as they apply to frameskip). This consumed a reasonable ammount of memory, but it still wasn't enough. In order to render transparencies, a second buffer of the same size is required (actually, a second and a forth, which I'll explain in a moment).

In ideal circumstances, Snes9x requires 4 memory buffers to render video: A screen buffer, a subscreen buffer (for transparency information), a zbuffer (required for normal operation - has always been located in system memory), and a subscreen zbuffer (for transparency z axis information). The screen buffers both require 2 bytes per pixel and the zbuffers require 1. At its current state, Sintendo uses only the low-res mode of Snes9x (256x239) or 61,184 pixels. For all buffers, 367,104 bytes are required, which doesn't seem like a large ammount compared to the 16 Megs the Dreamcast has available, but it didn't take me long to figure out that there just wasn't enough when all was said and done.

Why? I did the calculations and figure out Snes9x says it takes up about 8 Megs. WindowsCE about 2. The executable (SintendoDC.exe) must reside in memory as well. At one point, SintendoDC.exe exceeded over 2.5 Megs before I decided that the background bitmaps need to exist in external files rather than as resources inside of the executable. Right now, SintendoDC.exe is just over 900k, after being optimized for size rather than speed, getting it down from just under a Meg. It still didn't quite add up, but I figured I was being some sort of vicious memory hog, started looking in vain for any major leaks (ROM file names, bitmaps left in memory, anything), but eventually gave up and went back to programming Sintendo under some harsh memory constraints.

This is where the other problem came along. At some point late in programming the Flash (VMU) saving/loading functions, I had to create a function that checked the VMU to see if a file for the game loaded existed. This function refused to work for me for some reason, so I went to the old stand-by and checked the error message. Not enough memory. Ouch. I've already been down this futile path and came back empty handed (virtually), so at this point is when I actually optimized Sintendo for size rather than speed, saving me a cool 900k. Not enough memory. I'm not sure if the whole executable actually resides in memory at any given point, but I knew something was up here.

I went looking through the WindowCE documentation and found a neat little function GlobalMemoryStatus() that nicely filled a structure for me with all kinds of memory information. So I programmed this function into a menu to display this information and ran this menu at several key moments checking for memory consumtion. I followed it through until emulation and discovered that I was running with under 100k of free RAM by the time all was said and done. I rebooted my good old Dreamcast and followed it through again, trying to see where all of it was going. Nothing seemed out of the ordinary until I ran through it again, discovering that one single function, DoSnes9xInit() had reduced it from 12 Megs to 800k (rather than 8 Megs, like I previously thought). Somewhere in there, I had lost a 0. I knew what this function was and that it was, in fact, a giant memory hog, but I wasn't so sure about 11 Megs! So I put a call to the Memory manager between each function call inside DoSnes9xInit() that allocated memory. It was a great surprise to me to discover that Memory.Init() (an Snes9x function) was allocating over 10 Megs of memory. Wait a minute? Don't the docs on Snes9x say it takes up about 8 Megs? And to top all that off, do I really want to be going through Snes9x's code looking for where I can clean up memory? Then I remembered that horrible little FindFlashFile() function and it's very certain lack of memory and realized I had to.

At first glance everything looked in order. All of the numbers were in hex, so I had to break out the old Windows calculator to convert them and I was coming up with numbers like 64k and 128k, nothing anywhere near the 11 Megs this function was consuming, except perhaps:

    ROM     = new uint8 [MAX_ROM_SIZE + 0x200 + 0x8000];

So I did a file search to figure out where MAX_ROM_SIZE was defined, and what it was defined as, which ended up being:

enum { MAX_ROM_SIZE = 0xa00000 };

Calculator time again. A00000 hex = 10,485,760 decimal. You got it. 10 Megs. So I asked my good friends on the #dcemu chat channel if they've ever heard of a Super Nintendo ROM being 10 Megs in size. Someone mentioned they had heard of a 7 Meg ROM, but never 10. I personally have only seen Chrono Trigger and Mario RPG, both running at just over 5.

Whether or not there actually exist ROMs that are 10 Megs in size or whether this is just a logical number based on how much an SNES cart can actually store, I'm not sure. What I am sure of, however, is that for most intents and purposes, 10 Megs of memory simply isn't needed to load an SNES ROM. What I was even more sure of was that this space could most certainly be used for a better purpose. So, by changing one number, I managed to steal 2 Megs of memory from Snes9x:

enum { MAX_ROM_SIZE = 0x800000 };         // Changed July 30, 2001 by Jamin Vander Berg
// enum { MAX_ROM_SIZE = 0xa00000 };

I commented out the old define and replaced it with my own, allocated what I thought was a more than responable 8 Megs for ROM storage.

Through all of this, I had also learned another thing that surprised me. Sintendo actually occupied very little system memory by itself. Most of the memory Sintendo occupied was in those video buffers I spoke of earlier. Besides that, I had actually been doing a good job of cleaning up used memory and keeping Sintendo from being a hog. And all this time I thought it was me.

So this being done, I tested out the FindFlashFile() routine and everything went perfectly in order. Next, I uncommented all the screen buffer allocation deep inside Sintendo and turned transparencies on. I tried out (of course) Zelda first, and the results were astounding. It ran a little slower because of the transparenies, so I had to turn the frameskip up a notch to compensate, but even still Zelda was running very smooth. What was really important, however, were the graphics, which were simply astounding as I ran arround playing in the beautiful rain.

All together, this problem that had plagued Sintendo for a year minus 2 days was solved in only a few short hours - almost on accident. It's wonderful how things fall together sometimes... :)

ROM Loading

Now this problem took me about two days of hardcore searching, hacking, and testing to figure out - and that's not even fair to the problem. If I actually took the time to figure out exactly how much time I spent, how many sessions of brute force searching for the problem, rewriting code two or three times; if I were to add all those hours together, it wouldn't be pretty.

This problem began with Sintendo Build #209, in which I finally added a running In-Game menu, which seemed to run perfectly until the comments started pooring in. "When I go back to the game menu and select a game, some of the time it doesn't load it and just goes right back to the game I was playing". The key, very evil, phrase in that sentence (at least if you're a programmer) is "some of the time". Some of the time equates to an unpredictable bug, one which can not be replicated every time, which means it is the hardest to find. If a bug can be replicated every time, then you know where it is, and you go to that peice of code, look only at the peice of code and don't emerge until you've found it. If a bug can not be replicated, it can potentially come from anywhere. For exmple, a NULL pointer pretty much anywhere in the code may sometimes over lap code or memory that the ROM loading functions need. I did know, however, that it never happened the first time you loaded a ROM, which mean the problem may be in the In-Game menu, or perhaps the way in which the game menu was called the second time, but not the first. Or it could be in the ROM loading code itself, but why then did it never happen the first time?

These kind of problems are a disease. I rewrote the ROM loading code at least twice and the problem remained. I looked through everything until my eyes turned glossy time and time again. I'd go to sleep, suddenly think of something it could be and pop out of bed to try it out with no luck. I debugged through it several times, only to find out that the reason it wasn't loading was due to a 'File not found' error. But why? Obviously, when I set up the game list, the file is there otherwise it would have never shown up in the first place.

Something, somehow, got overlooked the first few times I debugged through the code. I was checking the wrong filenames or something, because everything seemed in order. When I tried to load, for example, 'Chrono Trigger.smc', the debug code listed the filename correctly, and still game me a 'File not found' error. I beat at my head and tried everything. Maybe I wasn't closing files properly and I was running out of files. Maybe I wasn't closing directory handles properly and loading them more than once. Anything... everything.

So today I discovered that the code I was trying to fix was never getting run. A minor oversite that was quickly fixed, but the bug still existed. 'File not found' it still said, so I checked the filenames again. The first filename looked fine, and the second, and then I got something like this: 'Chrono Trigger.smce Past.smc', and then 'File not found'. Hmmm, it's pretty obvious why the file's not found, but why does the filename look like that?

Its relatively obvious to anyone who knows C that the filename string, stored in memory, is not getting terminated. That is to say, any string in C always ends with a character 0, which isn't actually a character, but a terminator to tell C that this is the end of the string. Since the string was not getting terminated properly, whatever was previously in memory (the last ROM filename) became a part of the new filename. The terminator at the end of that string (which was there only because the memory was zeroed first), was when the string was actually ended.

So this is to say, the ROM loading problem was this: The first ROM always loaded ok, because the memory was zeroed first, which gave the string its own terminator by accident. The second time, if the filename was longer, it worked in much the same way. If, however, the string was shorter, overlap would occur and the filename would be corrupted. This is why I could almost never get 7th Saga.smc to load, but Zelda: A Link to the Past.smc loaded almost every time.

So I went through and zeroed out all my memory each time but was still having the same troubles. But this problem never existed on the PC version of Sintendo. Why? UNICODE. The filename string is actually converted twice because of UNICODE compatibility. When I get the list of filenames, they are stored as UNICODE strings, two bytes per character. Snes9x and its Memory.LoadROM() function expect character strings, or C strings, which are 1 bytes per character. Memory.LoadROM() then calls S9xOpenFile() (a function I wrote) to open the file. S9xOpenFile has to convert this string back into UNICODE in order to properly interface with the OS. This is were the problem was occuring. Looking at the following code:

#if defined UNDER_CE 
     mbstowcs(superstr, f, strlen(f)); 
#else 
     strcpy(superstr, f); 
#endif

It is not obvious that the entire problem behind ROM loading lies here unless you know exactly what you are looking for. It is somewhat, obvious, however, once you know where the bug is, why the PC version never had the same problem. The PC version doesn't deal with UNICODE at all. Instead, under the #else part, the string is just copied from one memory location to another using the very standard, very well-known strcpy() routine. The mbstowcs() function, however, converts a UNICODE string into a character string, and is only run if UNDER_CE is defined. UNDER_CE is only defined if the code is intended to be run on the Dreamcast.

Another not about mbstowcs() is that the third parameter is that it is supposed to be the size of the buffer used to store the new string, superstr, rather than the size of the source string, f. In truth, the third parameter could have simply been strlen(f) + 1 and everything would have worked fine. Instead, it was only converting the string until it reached the end of the string, not counting the terminating zero, so the terminating zero was never copied over. Changing this line of code over to:

mbstowcs(superstr, f, 255);

Made the world a better place for Super Nintendo junkies everywhere. Now, the terminating zero was finally getting copied over and the Dreamcast knew, properly, where the filename ended. Testing this proved my theory was correct: ROMs load from file every time, correctly.

It still makes me a little sad to think exactly how much time I spent trying to figure this problem out, and after all that it ended up being such a simple problem. I suppose that the lesson to be learned from both of these issues is that in programming, one number can mean the difference between life and death. Whether it be that you are allocating far too much memory or just one number away from the difference between a string ending properly or it going on until it felt like ending... This is not the first time I've run into a problem that was solved simply by changing one number to another, and it certainly won't be my last. I have learned from experience, however, that these are typically, and by far, some of the hardest to find. One number lying against a sea of 1s and 0s kindof puts the idea of a needle in a haystack to shame.

Some final notes about transparencies and memory. While in the PC word, 2 Megs or RAM is hardly worth spit, in the Dreamcast world, this is little to scoff at. This RAM afforded me not only the ability to check for Flash files and to enable transparencies; there is still plenty left for all of the things I wished I could have done, memory allowing. The current ROM limit per directory is 128. This will soon be changed to a more reasonable 256. The SNES hi-res mode may soon be supported simply by resizing the buffers (making them about 4 times as large - which may in fact end up being too large), and finally, when I get arround to it, there is plenty of memory available to enable Save States, an idea I've been considering for quite some time...

Until next time,

--Sin

Happy Holloween.

Development News: The Direct3D fix I mentioned before going to Chicago worked quite well and has gotten me very far. As you may have noted from the main news page, the menu now successfully renders a transparent blue box atop the background bitmap. The unfortunate news is this doesn't work on the Dreamcast. THe Dreamcast seems to clear the video buffer whenever you call BeginScene(), and since the bitmap can't be blt'd after this call, the bitmap disappears entirely. I have the menu, however, running up to speed despite that. So far, I've filled the background with a nice gradient white to black and you can definitely tell that the menu is transparent.

To solve the problem, I'm going to have to treat the background bitmap as though it werer a texture and texture map it only a polygon that covers the entire screen. This I am still working on. I've got it to do it somewhat, but the results are quite weird, such as: rendering the background bitmap in strange ways, or invalidating the menu code so the menu shows up as a black box rather than a blue one. Once working, however, the effect on the Dreamcast should be very similar to that which you see on the screenshot provided on the main page.

The code in question is as follows:

void AlphaBlt(LPDIRECT3DDEVICE3 dev, LPDIRECTDRAWSURFACE4 dds, 
                       int x1, int y1, int x2, int y2,
			   float fade) {


	LPDIRECT3DTEXTURE2 texture;
	D3DTEXTUREHANDLE htex;

	D3DTLVERTEX rgverts[5] = {
		{ 0, 0, 0, 1.0f, 0x00000000, 0, 0.0f, 0.0f },
		{ 0, 0, 0, 1.0f, 0x00000000, 0, 0.0f, 0.0f },
		{ 0, 0, 0, 1.0f, 0x00000000, 0, 0.0f, 0.0f },
		{ 0, 0, 0, 1.0f, 0x00000000, 0, 0.0f, 0.0f },
		{ 0, 0, 0, 1.0f, 0x00000000, 0, 0.0f, 0.0f }
	};

	HRESULT hr;
	D3DCOLOR c = D3DRGBA(1, 1, 1, fade);

	rgverts[0].color = c;
	rgverts[0].sx = (D3DVALUE)x1;
	rgverts[0].sy = (D3DVALUE)y1;
	rgverts[0].sz = 1.0f;
	rgverts[0].specular = D3DRGBA(0, 0, 0, 0);
	rgverts[0].rhw = 1.0f;
	rgverts[0].tu = 0.0f;
	rgverts[0].tv = 0.0f;

	rgverts[1].color = c;
	rgverts[1].sx = (D3DVALUE)x2;
	rgverts[1].sy = (D3DVALUE)y1;
	rgverts[1].sz = 1.0f;
	rgverts[1].specular = D3DRGBA(0, 0, 0, 0);
	rgverts[1].rhw = 1.0f;
	rgverts[1].tu = 1.0f;
	rgverts[1].tv = 0.0f;

	rgverts[2].color = c;
	rgverts[2].sx = (D3DVALUE)x1;
	rgverts[2].sy = (D3DVALUE)y2;
	rgverts[2].sz = 1.0f;
	rgverts[2].specular = D3DRGBA(0, 0, 0, 0);
	rgverts[2].rhw = 1.0f;
	rgverts[2].tu = 0.0f;
	rgverts[2].tv = 1.0f;

   	rgverts[3].color = c;
	rgverts[3].sx = (D3DVALUE)x2;
	rgverts[3].sy = (D3DVALUE)y2;
	rgverts[3].sz = 1.0f;
	rgverts[3].specular = D3DRGBA(0, 0, 0, 0);
	rgverts[3].rhw = 1.0f;
	rgverts[3].tu = 1.0f;
	rgverts[3].tv = 1.0f;


	dev->SetRenderState(D3DRENDERSTATE_SRCBLEND,
                           D3DBLEND_SRCALPHA);
	dev->SetRenderState(D3DRENDERSTATE_DESTBLEND,
                           D3DBLEND_INVSRCALPHA);
	dev->SetRenderState(D3DRENDERSTATE_ALPHABLENDENABLE,
                           TRUE );
	dev->SetRenderState(D3DRENDERSTATE_SRCBLEND, 
                           D3DBLEND_BOTHSRCALPHA);

	hr = dds->QueryInterface(IID_IDirect3DTexture2, 
                                (LPVOID*)&texture);

//	hr = dev->SetTextureStageState(0, D3DTSS_ALPHAOP, 
//                                    D3DTOP_BLENDTEXTUREALPHA);
	hr = dev->SetTexture(0, texture);
	
	// Draw the surface to the screen via DrawPrimitive.
	hr = dev->DrawPrimitive(D3DPT_TRIANGLESTRIP, 
                               D3DFVF_TLVERTEX, rgverts, 4, 0);

	texture->Release();

}

The function doesn't neccessarily need to be an alpha blt per-se, but I added that functionality in case I need it in the future. I'm imagining that the problem is that I'm just not setting the render states correctly, though I haven't discarded the posssibility that the texture itself isn't set up correctly. Anyone who is willing to provide any insight is welcome, though I don't imagine this is a large deal and I'm hoping it will be solved soon.

After this is done, all that remains is fixing the in-game menu with the same format, adding folders to the in-game menu, and a few other small enhancements including a selectable frameskip.

--Sin