[MOD] Mundanely Better Textures 1.80d (Nov-26-2013)

Sounds interesting, but how much difference is there between the TSOP release and yours (along the lines of graphics size)?  Most mods try to use .dds format as much as possible, anyway.

IIRC, I tried out before, and did some reading on RLE. they will be uncompressed when loaded in the ram which uses the same ram space but just increases load time. This was a few years ago and maybe the software has caught up but I doubt it.

I was under the impression that most modders did all this compression stuff just to reduce the size of their download, not to boost performance....

I looked into TSOP, and the two don't really overlap. TSOP mainly concentrated on optimizing particle effects and meshes, while this mod focuses on optimizing texture files. The only overlap is that TSOP converted most .tga files to .dds, something which I hadn't done because I was unsure whether it would work, while I kept the .tga files and just recompressed them with RLE. Also, AFAIK, the devs fixed a lot of the problems TSOP addressed in later patches. Still, I'll look into the whole tga-dds thing, something interesting might come out of it.

EDIT: Did some actual comparing, looks like TSOP does, in fact, change some the same textures I changed. TSOP reduced their size, while I kept the same size and just applied the maximum compression settings.

Experimentally, loading times seem to be unaffected, and I'm running the game from a 7200rpm HDD. I was unaware of Sins possibly uncompressing .tga's, so I'll try to check the bottom of the RAM usage and texture overlays to see if you're correct.

EDIT: You are correct, SoaSE uncompresses TGA files.

You're confusing compressed folders (zips, rars, etc.) with file compression. Compressed folders do just what you say they do: they can pack a large amount of files in a much smaller storage space. File compression is based upon the different settings a filetype supports. For example, .jpg files have a quality setting that drastically reduces filesize as you lower it, but you start getting more obvious "compression artefacts", like noticeable blocks of same colored pixels, a prominent grid-like pattern when zoomed in, etc. File compression is either lossless, meaning there is no loss of quality (e.g. FLAC is a lossless compression of WAV), or lossy, meaning there is some loss in quality, but the trade-off is usually worth it up to a certain point (e.g. 320kbps audio still sounds OK on most systems compared to ~8000kbps for uncompresed audio, but 128kbps audio files often sound hollow and flat, even if they take up 67% less space than 320kbps ones). In this mod, I changed the file compression settings of a lot of the textures from a lossless A8R8G8B8 to a lossy DXT5, DXT3, or DXT1c setting; as mentioned in the description, though there is some loss in quality, I didn't notice it at all in-game, so the reduced filesize and memory usage of the texture outweigh the possible quality loss caused by the lossy compression.

The bonus of dds is it loaded into the GPU's ram where tga is not, or it was when I worked on TSOP.

Major Stress I think turned all tga into DDS5. along with finding and repairing many text errors, not many meshes were optimized, fighters were which was a major help late game.

DDS5 was a little to compressed for my liking on a lot of the UI stuff, I am happy with all the small research buttons and some of the smaller particle textures as tga

If you wish to incorporate anything from TSOP, go ahead, improve on it, the community needs a new leader of the TSOP genre as Stress is retired and I am way to busy with SOA2

No, I'm not...

I don't for sure remember the exact numbers, but IIRC using DXT3 cut down the file-size by a factor of 2 or 4 for planet textures (probably comparable for other textures too)...yet when in-game, the memory usage seemed unaffected regardless of how "zoomed in" you were on planets...I tried on maps with lots of planets and lots of different planets -- did my best to keep other things constant, but I didn't see any differences in resources....essentially, the DXT3 compression did not seem to reduce the memory usage of the game, only the filesize...

I've discussed this with some other modders to hear their thoughts and the general gist I got was basically that they used file compression on their textures just to keep their download size small...

Now, I may be wrong on this....DXT3 may also reduce the memory used by a factor of 2 or 4, I just didn't test it properly...but I assure you I'm not confusing file compression with compressed folders.....

Very odd, because the textures overlay for the dev exe clearly showed a reduction in texture memory usage, for BackdropInGame for example. In addition, I ran test case at the end of my dds pass (same game, same map, same conditions; 9 players, triumverte, all 6 factions present): for unmodded, the first 729 textures loaded took up exactly 814.37 MB, while for modded, the first 729 textures loaded took up exactly 735.00 MB.

Yes the Devs worst textures is a great way to see what is loaded, now I remember that the rle's were showing the uncompressed size, and the different DDS types showed the same as the file size.

Great job so far  

Keep us posted on your testing please  

Yeah, myfist is correct.

How you compress the TGA's really just affect their load times.

The power of DDS is mainly two fold-

1. You can use mipmaps to improve performance of game elements when zoomed out.
2. They are sent compressed to the graphics card and uncompressed in hardware by the GPU.

The downside of compressed DDS is they are lossy and add artifacts. Using compressed DDS on UI elements can result in noticeable artifacts in addition to the picture just being a bit blurrier. 

Keep working on it and let us know if you learn anything new.

On compressing the effect TGA's in the game- be careful as some are not in multiples of 4 on their length and width. They will not compress correctly to DDS if this is the case. I've had one example that also became corrupted after conversion when the file was 1 pixel width which I didn't catch when doing a batch conversion of the files.

So, there's actually a reason not to compress TGA files? I assume you mean the game loads faster at start up?

Yes, load time, but it was negligible. I just didnt see the point when the sheet used up the same RAM uncompressed.

Like Myfist0 said, best to incorporate TSOP into whatever you're working on, and work out from there.  Do you have all the DLCs?

What exactly were you changing?  Were you comparing just normal files to DXT3 or .dds to .tga?

For v1.8a, I changed most existing .dds textures to DXT5 or DXT1, depending on alpha channel information. However, while converting the effect/UI files from .tga to .dds for v1.8b, I've recently discovered an odd way I can further optimize some textures: quite a few textures aren't colored (just shades of grey), yet they are stored as RGBA8. Converting these files to A8L8 or L8 will reduce their size by at least 50% with no backside, other than the fact that this method is limited to grayscale textures. Plus I don't know of any automation tools I could use to process all 800+ textures, so it will definitely take some time.

EDIT: Scratch that, I forgot DXTx has its own way of saving color, so it's either DXT3/5 or L8/A8L8, not both, making sorting through possible L8/A8L8 optimizations even more of a nightmare.

I will have to re-look at the planet textures I was working with...I compared normal .dds to DXT3, but like I said, I may not have tested it properly...I also did not use the dev.exe memory stats but rather task manager (I know it sounds sketch but I decided using something NOT put in the game may be a fairer way of analyzing)....

Next version of the mod is done and uploaded. In this version, I removed the RLE compressed .tga files and replaced them with .dds conversions of said .tga files. Since all 400 or so .tga files are different, I went through each one manually to do things like test to see if compression ruins the texture quality, remove unused alpha channels, check to see whether saving the texture with smaller color palettes e.g. R5G6B5 produced any noticeable quality degradation, rescale and expand textures that didn't have dimensions of a power of 2, etc. I also updated the UI files to force loading of the newly converted .dds textures, and also created an addon for optimized versions of maps' main menu graphics, since I had to edit the maps' .galaxy files and didn't want to integrate potentially multiplayer-breaking stuff into the main mod.

Compared to v1.80a, v1.80b lowers memory usage even further (~5% improvement for the update, ~15% improvement in total), has even more stuff mipmapped (weapon and spell effects mainly), and has slightly faster load times. The mod requires approximately 33 MB more disk space compared to v1.80a due to RLE compressed .tga files still being smaller than mipmapped and optimized .dds files, though the mod's textures still take up less space overall than the vanilla textures.

While testing another mod I have in development, I found something nasty I overlooked in v1.80b. This is fixed in v1.80c, and since the modified file is only 7 kB in size, I uploaded a patch for the people who are already using v1.80b to download and decompress into the mod's folder. The full version will follow soon.

Like Fist0 said. If you want to incorporate your changes into TSOP you are more than welcome to do so. In fact i encourage it. Being that i am retired and all.

The main reasons i converted the TGA's to DDS was that DDS is pretty much the gaming standard. To be able to use Mip Maps in the particle textures, and it saved a crap ton of in game memory, because the textures load into the GPU ram instead of system ram. I know TGA is a better quality format, but the benefits of DDS outweighed the drawbacks IMO.

The Dev's did in fact fix many of the issues TSOP addressed. Like the text errors, missing files, and bad file path name issues. Which is one of the main reasons why it was never updated. TSOP was created for one main reason, and that was to eliminate the 2 gig crash issue. The 2 gig issue is no longer a problem as of Rebellion.

However Rebellion, and the DLC's could have new minor issues that we are not aware of yet. That a simple text edit can fix. Which is why i encourage the community to take over TSOP, and incorporate projects like this one into it, or vice versa.

for outstanding work

For mods that is no longer true, I have to tell people to reduce their graphics settings again just like the good old Diplomacy days. 

I meant the 2 gig issue was fixed for the core game itself. Of course for mods it is a different story. People want to put even more eye candy in the game, but do not realize that they will hit the 2 gig brick wall if they over do it.

More like- people don't think your serious when you tell them they can't run mods on HIGHEST/HIGHEST/HIGHEST... Well technically they could- I could reduce the -da and -nm maps down to low resolution and let them feel better about running effects on all highest.

Silliness aside, doing something like that might actually pay off; the map will have a lower resolution, sure, but it might not be not as noticeable if you can do some good sharpening. If I have a bit of time, I might try lowering the maximum resolution of some of the larger textures/maps by 25% (starbases, titans, planets, etc.), then doing a bit of wavelet sharpening (a little trick I learned way back to sharpen images with minimal noise by taking advantage of Photoshop's/GIMP's image resize algorithm, or "wavelet algorithm", hence the name), and see how they hold up.

If you can reduce the starbase textures (including the -da, and -nm maps) to 2048 res, and make the quality look like the original 4096's. Then that will be a feat in itself. The game hates odd sized textures that are not in a power of 2. Despite a few odd sized ones being in game. Sharpening will be very hard to do without pixelating the texture. This is just from my experience using photoshop. Not saying it cant be done. I personally just wouldnt know how to do it.

You may know some things that i do not. So go ahead, and experiment. Worst that could happen is it fails. If it does work then outstanding.