|
|
Post by jcd007 on Oct 14, 2024 14:16:18 GMT -5
I'm trying to do a bulk update of package files, and to that end I've been trying to write a script that can read the contents of the file, hex edit whatever is necessary and save it back. I'm having trouble doing this because there doesn't seem to be an easy way to decode the files; S4S can read a package file and get the internal name, the tuning, tags, etc. but I have no idea how to replicate that functionality.
Basically I want to be able to parse the package files so that they're human readable enough that I can tell what corresponds with what.
|
|
|
|
Post by jcd007 on Oct 14, 2024 17:36:56 GMT -5
Here's the error I'm getting when I try to load a modified package file into S4S. What I'm doing is using this project: github.com/thequux/s4py/blob/master/README.md, to load up the package file and writing some new data back (i.e. update the description). S4S does not know how to read the resource afterwards, and I'm not 100% sure why. The Sims 4 Studio - Version 3.2.2.9
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> S4Studio.Data.IO.Package.UnreadableResourceException: Resource: 220557DA-80000000-000000004667854A (Zlib) in package:C:\Users\[UserName]\Documents\Charly Pancakes\CharlyPancakesINSOMNIABirdSkullCandle.package could not be read.
at S4Studio.Data.IO.Package.DBPFPackage.FetchResource(Type t, IDBPFResourcePointer index, Boolean nocache) in X:\TheS4Studio\S4Studio.Shared\IO\Package\DBPFPackage.cs:line 698
at S4Studio.Data.IO.Package.DBPFPackage.FetchResource[T](IDBPFResourcePointer index, Boolean nocache) in X:\TheS4Studio\S4Studio.Shared\IO\Package\DBPFPackage.cs:line 602
at S4Studio.Data.IO.Locale.StringTableManager.FixLanguageSTBL(IDBPFPackage package) in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 44
at S4Studio.Data.IO.Locale.StringTableManager.Init() in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 161
at S4Studio.Data.IO.Locale.StringTableManager..ctor(IDBPFPackage localPackage) in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 120
at S4Studio.ViewModels.Generic.CatalogSims4CustomContent..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id, Type type) in X:\TheS4Studio\S4Studio.Shared\ViewModels\Generic\CatalogSims4CustomContent.cs:line 90
at S4Studio.ViewModels.Generic.BuyBuildSims4CustomContent..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id, Type type) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\BuyBuildSims4CustomContent.cs:line 34
at S4Studio.ViewModels.Generic.BuyBuildSims4CustomContent`1..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\BuyBuildSims4CustomContent.cs:line 658
at S4Studio.ViewModels.ObjectCustomContentViewModel..ctor(IWindow window, IResourceProvider globalFiles, IDBPFPackage localPackage, UInt64 prototype_id) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\Objects\ObjectCustomContentViewModel.cs:line 38
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
at S4Studio.Shared.StudioContentViewModelFactory.CreateStudioView(IWindow win, IResourceProvider globalResources, IDBPFPackage package, IPackedResource rs) in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioContentViewModelFactory.cs:line 79
at S4Studio.Shared.StudioContentViewModelFactory.CreateStudioView(IWindow win, IResourceProvider globalResources, IDBPFPackage package) in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioContentViewModelFactory.cs:line 68
at S4Studio.Shared.StudioDocumentModel.LoadViewModel() in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioDocumentModel.cs:line 282
===========================================================
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> S4Studio.Data.IO.Package.UnreadableResourceException: Resource: 220557DA-80000000-000000004667854A (Zlib) in package:C:\Users\amer_\Documents\Charly Pancakes\CharlyPancakesINSOMNIABirdSkullCandle.package could not be read.
at S4Studio.Data.IO.Package.DBPFPackage.FetchResource(Type t, IDBPFResourcePointer index, Boolean nocache) in X:\TheS4Studio\S4Studio.Shared\IO\Package\DBPFPackage.cs:line 698
at S4Studio.Data.IO.Package.DBPFPackage.FetchResource[T](IDBPFResourcePointer index, Boolean nocache) in X:\TheS4Studio\S4Studio.Shared\IO\Package\DBPFPackage.cs:line 602
at S4Studio.Data.IO.Locale.StringTableManager.FixLanguageSTBL(IDBPFPackage package) in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 44
at S4Studio.Data.IO.Locale.StringTableManager.Init() in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 161
at S4Studio.Data.IO.Locale.StringTableManager..ctor(IDBPFPackage localPackage) in X:\TheS4Studio\S4Studio.Shared\IO\Locale\StringTableManager.cs:line 120
at S4Studio.ViewModels.Generic.CatalogSims4CustomContent..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id, Type type) in X:\TheS4Studio\S4Studio.Shared\ViewModels\Generic\CatalogSims4CustomContent.cs:line 90
at S4Studio.ViewModels.Generic.BuyBuildSims4CustomContent..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id, Type type) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\BuyBuildSims4CustomContent.cs:line 34
at S4Studio.ViewModels.Generic.BuyBuildSims4CustomContent`1..ctor(IWindow window, IResourceProvider remoteSource, IDBPFPackage localPackage, UInt64 prototype_id) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\BuyBuildSims4CustomContent.cs:line 658
at S4Studio.ViewModels.ObjectCustomContentViewModel..ctor(IWindow window, IResourceProvider globalFiles, IDBPFPackage localPackage, UInt64 prototype_id) in X:\TheS4Studio\S4Studio.Shared\ViewModels\BuildBuy\Objects\ObjectCustomContentViewModel.cs:line 38
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
at S4Studio.Shared.StudioContentViewModelFactory.CreateStudioView(IWindow win, IResourceProvider globalResources, IDBPFPackage package, IPackedResource rs) in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioContentViewModelFactory.cs:line 79
at S4Studio.Shared.StudioContentViewModelFactory.CreateStudioView(IWindow win, IResourceProvider globalResources, IDBPFPackage package) in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioContentViewModelFactory.cs:line 68
at S4Studio.Shared.StudioDocumentModel.LoadViewModel() in X:\TheS4Studio\S4Studio.Shared\ViewModels\App\StudioDocumentModel.cs:line 282 |
|
|
|
Post by andrew on Oct 15, 2024 0:39:19 GMT -5
Hi jcd007, The error message means that the string table 220557DA-80000000-000000004667854A is not formed correctly. If you are manually creating/modifying this, check the format again to make sure you are writing the resource correctly. |
|
|
|
Post by jcd007 on Oct 15, 2024 18:59:21 GMT -5
Let me give some more context. I'm using the Github code to read in a package file, make modifications and write a new package file.
When I read and write without modifying the package file in any way, it works. So it's not anything wrong with the Github code. But when I update the byte data in one of the resources it fails. It's not clear why because it looks perfectly normal.
Example:
A normal resource before I modify it:
ResourceID(group=2147483648, instance=72057595219117386, type=570775514)
dbfile[resource_id].content = b'STBL\x05\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x00\x00\x00\x04\xc0\xbbN\x00N\x00This poor little bird is wearing a candle on his head.\r\n\r\n\r\nby Charly Pancakes\xef\x8c\xbf\xac\x00,\x00INSOMNIA - Birdskull Candle| Charly Pancakes'}
After modification:
ResourceID(group=2147483648, instance=72057595219117386, type=570775514)
dbfile[resource_id].content = b'STBL\x05\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00|\x00\x00\x00\x04\xc0\xbbN\x00N\x00This poor little bird is wearing a candle on his head.\r\n\r\n\r\nby Charly Pancakes\xef\x8c\xbf\xac\x00,\x00INSOMNIA - Birdskull Candle| Charly Pancakes happy'}
Basically all I'm doing is extending the byte data with more byte data (i.e. adding a tag programmatically) but that's enough to cause the whole thing to break. The only point of failure is where I modify the byte data, but I'm not seeing why the way I'm doing it is wrong.
|
|
|
|
Post by jcd007 on Oct 15, 2024 19:30:56 GMT -5
I can confirm that if I make the byte data smaller it also fails with a different error, but if it's exactly the same size then it's fine (even if the data itself is different). I'm guessing the offsets aren't being calculated properly, but I'm checking them manually and they seem fine to me.
|
|
|
|
Post by jcd007 on Oct 15, 2024 20:04:05 GMT -5
I edited the string via the edit items button in the S4S warehouse and it shows that the entire byte data changes when I append text, so it's actually not as simple as just doing that.
Also all I'm really trying to do is copy that functionality so I can add some batching logic (i.e. rename multiple packages at once) so if there's an easier way to just do that I'd really appreciate hearing it.
|
|
|
|
Post by andrew on Oct 16, 2024 17:31:42 GMT -5
You cannot just add to the end of the bytes. Each string has to be prefixed with the length of the string. S4S reads the entire file based on the lengths provided and if there are leftover bytes at the end, it shows an error message because that means the file is formed incorrectly. Please see the format provided by EA below. I couldn't find the original post on their new forum, so I just pasted it below.
//--------------------------------------
//--- 010 Editor v5.0.2 Binary Template
//
// File: StringTable Template.bt
// Revision: 1.0
// Purpose: Document The Sims 4 String Table Resource
//--------------------------------------
LittleEndian();
BitfieldRightToLeft();
// Header
char mnFileIdentifier[4];
if (mnFileIdentifier != "STBL")
{
Warning("Not a String Table file!");
return -1;
}
// The Sims 4 base game version is 5.
uint16 mnVersion;
if (mnVersion != 5)
{
Warning("Only version 5 is supported.");
return -1;
}
// Compression is not currently used by The Sims 4.
byte mbCompressed;
uint64 mnNumEntries;
byte mReserved[2];
uint32 mnStringLength;
typedef struct
{
uint32 mnKeyHash;
byte mnFlags;
uint16 mnLength;
if (mnLength > 0)
char mString[mnLength];
} StringEntry <read=GetStringValue>;
wstring GetStringValue(StringEntry &entry)
{
if (entry.mnLength > 0)
return StringToWString(entry.mString, CHARSET_UTF8);
return L"";
}
// The rest of the file comprises the string table.
StringEntry mStrings[mnNumEntries] <optimize=false>;
|
|
|
|
Post by jcd007 on Oct 26, 2024 19:13:47 GMT -5
How I ended up fixing it (for any future users with this problem).
original_length = len(content.split(b"\x00")[-1])
original_byte = struct.pack(">H", original_length)
updated_content = content + b" more bytes"
length = len(updated_content.split(b"\x00")[-1])
byte_overwrite = struct.pack(">H", length)
final_content = byte_overwrite.join(updated_content.rsplit(original_byte, 1))
Basically it gets the length of the string before and after adding content, so it knows what byte to replace and what the new byte should be.
Is the best, most efficient or most robust way to do it? Probably not. I can imagine it failing if the content gets too long, or other edge cases. But it's good enough for the case I'm going for (tagging a bunch of files the same way for personal use).
|
|
|
|
Post by jcd007 on Oct 26, 2024 20:21:22 GMT -5
Ah, new problem I'm facing. Sometimes the name is first, other times it's the description.
Looking in S4S I can see text like this:
"Entries": [
{
"Key": "0x4EBBC004",
"Value": "This is the description of the items\r\n\r\n\r\n"
},
{
"Key": "0xACBF8CEF",
"Value": "This is a name with tags | tags"
}
],
I can read the keys and values easily, but there's no guaranteed order. How do I use the data given to me to tell which is which? How does S4S do it?
|
|
|
|
Post by andrew on Oct 26, 2024 20:39:27 GMT -5
|
|
|
|
Post by jcd007 on Nov 2, 2024 18:10:51 GMT -5
That was the final missing piece. I've gotten it to work for all my use cases now. Thank you so much, I literally could not have done this without you.
Side note: never had the chance to work with binary templates before. It was fun trying to parse everything properly.
|
|
