Request/Proposal for better (more granular) modding support

implementing Extensible Markup Language (XML) Patch Operations Framework (RFC 5261)

from GalCiv3 Forums

As it is.

The current modding support for GC3 is completely based on full file replacement, during the load of a mod. While this is very easy to implement and create a mod, it is not very maintenance friendly or suitable for multi mod loading.

The ease comes from the fact that you simply copy a standard file to your mod directory, make the changes you wish and that is it.

The problems arrive when there is an update of the game (that changes the original of your modded file), in which case to have to re-copy the standard file re-apply your changes. This is not very friendly for maintenance, especially when your mod not just changes a single file but multiple files.

Also if you want your mod to coexist with another mod, it limits you to mods that do not make changes to any of the files you altered. For if this is the case then the last file loaded is the one that will be in effect, even when the two files only change different elements.

RFC 5261.

Therefor i propose that, the already existing standard for, xml patching is implemented in the game. This standard known as "RFC 5261: An Extensible Markup Language (XML) Patch Operations Framework Utilizing XML Path Language (XPath) Selectors" describes how to patch xml documents on element level.

This would allow for far less maintenance and would allow mods to coexist without any problems - as long the mods don't have competing operations, like one mod altering an element and the other mod deleting the same element.

I am fully aware that most people don't want to go and read a RFC so i will show the functionality with some examples. These examples i have taken from the file that i myself use to patch the GC3 xml files after an update from steam arrives.

Be warned that the examples are for version 1.0.x of GC3 (i have not tested them against version 1.1) and that i have copy-pasted them from my far larger patch file.

The patch (example) file.

<diffs>
    <diff file="ImprovementDefs.xml">
        <replace sel="/DefaultImprovementList/Improvement[InternalName='OmegaResearchCenter']/PlacementType/Text()">ResearchUnique</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='ResearchCloisters']/PlacementType/Text()">ResearchUnique</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='ResearchCloisters']/Stats[EffectType='Research' and BonusType='Multiplier']/Target/TargetType/Text()">Colony</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='QuantumResearchFoundation']/Stats[EffectType='Research' and BonusType='Multiplier']/Target/TargetType/Text()">Colony</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='AscensionGate']/Stats[@EffectType='Research' and BonusType='Multiplier']/Target/TargetType/Text()">Colony</replace>
        <add sel="/DefaultImprovementList/Improvement[InternalName='AscensionGate']/Prerequ/UpgradesFrom/Text()">QuantumResearchFoundation</add>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='XenoIrrigation ']/DisplayName/Text()">XenoIrrigation_Name</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='XenoIrrigation ']/ShortDescription/Text()">XenoIrrigation_ShortDec</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='XenoIrrigation ']/InternalName/Text()">XenoIrrigation</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='HyperionSupplySystem']/LevelEffectTriggers/Lifetime/Text()">Target</replace>
        <replace sel="/DefaultImprovementList/Improvement[InternalName='HyperionShipyard']/LevelEffectTriggers/Lifetime/Text()">Target</replace>
        <remove sel="/DefaultImprovementList/Improvement[InternalName='BiosphereManipulator']/IsColonyUnique"/>
    </diff>

    <diff file="..\Game\English\Text\ImprovementText.xml">
        <replace sel="/StringTableList/StringTable[Label='XenoIrrigation _Name']/Label/Text()">XenoIrrigation_Name</replace>
        <replace sel="/StringTableList/StringTable[Label='XenoIrrigation _ShortDec']/Label/Text()">XenoIrrigation_ShortDec</replace>
        <replace sel="/StringTableList/StringTable[Label='XenoIrrigation _Dec']/Label/Text()">XenoIrrigation_Dec</replace>
    </diff>

    <diff file="ShipComponentDefs.xml">
        <add pos="before" sel="/ShipComponentList/ShipComponent[InternalName='AdvancedTransportModule']/Stats[1]">
            <Stats>
                <EffectType>Logistics</EffectType>
                <Target>
                    <TargetType>Ship</TargetType>
                </Target>
                <BonusType>Flat</BonusType>
                <Value>2</Value>
            </Stats>
        </add>
    </diff>

    <diff file="*FactionDefs.xml">
        <replace sel="/FactionList/Faction/CreditsInit/Text()">1000</replace>
    </diff>
</diffs>

The explanation.

Lets take the line:

        <replace sel="/DefaultImprovementList/Improvement[InternalName='OmegaResearchCenter']/PlacementType/Text()">ResearchUnique</replace>

it looks in the ImprovementDefs.xml file for the Improvement whose InternalName is OmegaResearchCenter, then it alters the text for the PlacementType tag to ResearchUnique. (in the standard GC3 file it is set to WealthUnique, which it should not be).

The same for the ResearchCloisters where in the standard xml file it is set to ManufacturingUnique while it should also be ResearchUnique.

        <replace sel="/DefaultImprovementList/Improvement[InternalName='ResearchCloisters']/Stats[EffectType='Research' and BonusType='Multiplier']/Target/TargetType/Text()">Colony</replace>

This line finds the ResearchCloisters improvement, looks for the Stats that has 'Research' for EffectType and has 'Multiplier' for the BonusType, and then alters the TargetType tag to 'Colony'

        <add sel="/DefaultImprovementList/Improvement[InternalName='AscensionGate']/Prerequ/UpgradesFrom/Text()">QuantumResearchFoundation</add>

Adds an UpgradesFrom tag, with 'QuantumResearchFoundation' as text value, to (the end) of the Prerequ node.

        <replace sel="/StringTableList/StringTable[Label='XenoIrrigation _Name']/Label/Text()">XenoIrrigation_Name</replace>

Find the string 'XenoIrrigation _Name' (with the erroneous space in it) and changes it to the name without the space in it.

To show it is not limited to single value changes i have added the example of the ShipComponentDefs.xml where an entire (Logistics) stat is added to the AdvancedTransportModule ship component. Because the xml files are order sensitive and all ship components have at least one stat i insert the new stat before the first existing stat (note the pos attribute in the add command).

Elements can also be removed:

        <remove sel="/DefaultImprovementList/Improvement[InternalName='BiosphereManipulator']/IsColonyUnique"/>

removes the IscolonyUnique from the BiosphereManipulator (to allow it to be build multiple times).

and finally:

    <diff file="*FactionDefs.xml">
        <replace sel="/FactionList/Faction/CreditsInit/Text()">1000</replace>
    </diff>

will change the starting credits for all factions (as long as their file name matches *FactionDefs.xml) to 1000 bc. Also since a XPath (the text after the 'sel' attribute) selects a set of elements and not just a single element, the change will be applied even when there are multiple factions defined in a single xml file (like the standard GC3 FactionDefs.xml file).

 Notes.

If implemented then since GC3 reads all the xml into memory (and into the game save file) the <diff> node file attribute should be removed, since there are no xml files in memory, so the element <diff file="ImprovementDefs.xml"> should become just <diff>.

Afterword.

I really want to see this (or something similar) implemented to empower the modding community.

16,591 views 8 replies
Reply #1 from GalCiv3 Forums Top

I agree with this.  This game is so very mod-able yet mod-unfriendly because so few mods will ever be able to coexist with other mods.

Reply #2 from GalCiv3 Forums Top

Would be still hard to have them be valid.

A bit off topic. Thecw, you seem to have experience with XML and possibly XML Schema's. Is it possible for a XML Schema to check if a value is available within another element in another file? In this case I am thinking to check if an InternalName exists for an element that tries to reference such value.

CursorFX - Customize your Windows cursor
Reply #3 from GalCiv3 Forums Top

Seconding this. Full file replacement is highly undesirable for mods.

Reply #4 from GalCiv3 Forums Top

>Therefor i propose that, the already existing standard for, xml patching is implemented in the game

Your suggested solution would certainly make it quite flexible and allow new ways to combine stuff but is it perhaps too complex? If they went for it I'm all in favor of that but usually the more costly it is to do the less chance it gets picked up...

At the most basic level what we are missing in modding is the ability to modify single values in existing files without having to copy the entire file. What I'm thinking is that since there's *already* built in the game the implementation of adding new XML entries with files that have different names, how about simply changing that part a bit: Leave everything else "as is" but if the "add element from new file" happens to encounter an XML entry that's already loaded previously (either in the base game file or from some other mod), replace the already loaded value with the new value. I think that would keep the needed implementation at minimum.

So, if the base game file SomeBaseGameFile.xml had:

... more XML stuff ...

<SomeValue>20</SomeValue>

... more XML stuff ...

you could create MyMod.xml that had only two elements (with the same XML-paths, of course):

<SomeValue>15</SomeValue>

<ACompletelyNewValue>22</ACompletelyNewValue>

When the game loads it sees the file name is unique and different from the base game files so it loads it as an addition (like now) adding the <ACompletelyNewValue> entry to the game database (like now) but when it encounters the <SomeValue> entry it notices that XML-path-enty already exists and so it replaces the base game value "20" with the modded value "15". This would allow you to make "surgical" modifications targeting just those entries that the mod needs to modify and everything else would come from the actual base game file (or another mod) meaning they would automatically get updated if the game gets updated.

Yes, that would still leave the problem of deleting stuff with similar precision but at least it's a step forward... :)

Reply #5 from GalCiv3 Forums Top

Yep this would make a huge difference for most modding, I can think of a few examples it wouldnt fix but that is another issue.

However what I would like to know is why no one ever seems to use this system, I've moddded a number of games that use xml append/replace mechanics like GC3 does and never seen one use any kind of in place or inject-able code replacement.

One thing I could definitely wish for in addition to this is real math operators like you see in most other games that use xml Multiplier and Flat just don't cut it for alotta things. 

Reply #6 from GalCiv3 Forums Top

Quoting pendrokar, reply 2

A bit off topic. Thecw, you seem to have experience with XML and possibly XML Schema's. Is it possible for a XML Schema to check if a value is available within another element in another file? In this case I am thinking to check if an InternalName exists for an element that tries to reference such value.

Not according to the XML standards. A XML schema defines the syntax (and limited validation) for a xml structure, consistency enforcement neither external nor internal is defined by a schema.

Only XML engines that support a directable tokenizer phase can do this, but these engines are rare and not-standard-compliant. Therefor they are rarely used.

Unfortunately consistency checking (after load completion) is one thing that GC3 does not support. And it is, together with RFC 5261 support, on my wishlist see my reply (#12) in the The GalCiv III wishlist thread.

Reply #7 from GalCiv3 Forums Top

Quoting Deathwynd, reply 5

However what I would like to know is why no one ever seems to use this system, I've moddded a number of games that use xml append/replace mechanics like GC3 does and never seen one use any kind of in place or inject-able code replacement.

There are a number of games that do support it, but it is not widely advertised. From the top of my head: 7 days to die and X:Rebirth, both of which i myself have not played but i have taken a look at their mod files for friends. One of these two might actually need a plugin to activate it.

Some game engines have support for it because they have it embedded, so it is in some games and the game developer does not know it or has not exposed the interface to the players.

I can understand why developers don't like it, because it would require multi stage schema validation: first after (or during) the initial load of the base file, and then after each time a <diff> node closes all the altered nodes must be re-validated.

+1 Loading…
Reply #8 from GalCiv3 Forums Top

Well it'd sure be nice to have more games openly support it, and would love to see GC3 support it, you've got my vote.