ERF

2011-03-03T00:29:41Z

Mephales: /* Encryption */ described XOR encryption and decryption

[[Image:ERF editor.png|thumb|ERF editor]]

'''ERF''' ("Encapsulated Resource File") files are container files for game resources.

= List of common ERF Files =

ERF Files for Dragon Age Game Resources are stored in the (Dragon Age Install Dir)\packages\core\ Folder and Subdirectories. ERF's for Addons or Mods are usually in their respective Addon or Mod Folder.

Common ERF's of certain interests are,

*Game Resources
**packages\core\data\2da.erf
**packages\core\data\anims.erf
**packages\core\data\consolescripts.erf
**packages\core\data\designerareas.erf
**packages\core\data\designercreatures.erf
**packages\core\data\designercutscenes.erf
**packages\core\data\designerdialogs.erf
**packages\core\data\designeritems.erf
**packages\core\data\designerplaceables.erf
**packages\core\data\designerplots.erf
**packages\core\data\designertriggers.erf
**packages\core\data\face.erf
**packages\core\data\gui.erf
**packages\core\data\guiexport.erf
**packages\core\data\iterationtests.erf
**packages\core\data\lightprobedata.erf
**packages\core\data\materialobjects.erf
**packages\core\data\materials.erf
**packages\core\data\misc.erf
**packages\core\data\modelhierarchies.erf
**packages\core\data\modelmeshdata.erf
**packages\core\data\pathfindingpatches.erf
**packages\core\data\postprocesseffects.erf
**packages\core\data\resmetrics.erf
**packages\core\data\scripts.erf
**packages\core\data\shaders.erf
**packages\core\data\states.erf
**packages\core\data\subqueuefiles.erf
**packages\core\data\textures.erf
**packages\core\data\tints.erf
**packages\core\textures\high\texturepack.erf
**packages\core\textures\medium\texturepack.erf

*Patch Resources
**packages\core\patch\... --- Contains Resources added by Game Patches

*Toolset specific Resources
**packages\core\data_tools\... ---Contains Resources for the Toolset

= Extracting Resources =


The Toolset is capable of extracting ERF Files Contents.

If you are modding Resources, the first step would be extracting Resources. You may start of by extracting all Resources of certain interest to you, or just a single File. To Maintain a simple and clean resource place, it is best to use a similar Folder Names to extract in.

One may create a Folder on the HD named for example "gameresources". Then added with subfolders that suit the Purpose, for example "Materials" in which one extracts only Materials. Preferably the Folderscheme is similar to that of the ERF's, to make browsing simple and easy to remember.

= Packing Resources =

The [[builder to player]] process packages up resources into ERF files for you, storing the generated ERFs inside [[DAZIP]]s.

Additionally, you can edit any ERF by opening it directly in the Toolset, and add or remove files by drag& dropping them into the Toolset with the ERF open manually. In cases the above process fails or whenever.

= Overriding Resources =

ERF's are named by their general purpose. But not all material files need to necessarily be put into Materials.ERF as an example. Patches do add every Patched Resource into one single ERF for example. The Original Game Resource ERF Files however are best left alone, as it has a high risk of breaking the game.

Files do not need to be present in ERF Files at all. When Overriding Gameresources as when modding, this is a pretty easy method. Overriding Files are of equal name to their original Gameresource counterpart, and are to be placed into packages\core\override\. The Game will then choose this File over the original.

= File Format =

== Version 2.0 ==

All values are [http://en.wikipedia.org/wiki/Endianness#Little-endian little-endian].

=== Header ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| magic
| utf16[8]
| "ERF V2.0", a total of 16 bytes.

|-
| fileCount
| uint32
| Number of files encapsulated in this ERF.

|-
| year
| uint32
| Year since 1900.

|-
| day
| uint32
| Day since January 1st.

|-
| unknown
| uint32
| Always 0xFFFFFFFF?

|}

=== Table of Contents ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| toc
| [[#Entry|entry]][fileCount]
|

|}

==== Entry ====

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| name
| utf16[32]
|

|-
| offset
| uint32
| Offset to entry file data, from start of ERF file.

|-
| size
| uint32
| Length of file data.

|}

== Version 2.2 ==

This version introduces support for encrypted and compressed file data.

All values are [http://en.wikipedia.org/wiki/Endianness#Little-endian little-endian].

=== Header ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| magic
| utf16[8]
| "ERF V2.2", a total of 16 bytes.

|-
| fileCount
| uint32
| Number of files encapsulated in this ERF.

|-
| year
| uint32
| Year since 1900.

|-
| day
| uint32
| Day since January 1st.

|-
| unknown
| uint32
| Always 0xFFFFFFFF?

|-
| flags
| uint32 [http://en.wikipedia.org/wiki/Bit_field Bit flags]
|

Flags:
* 0x00000001 (bit 0) : Unknown
* 0x000000F0 (bits 4 - 7) : [[#Encryption|Encryption]] scheme
* 0xE0000000 (bits 29 - 31) : [[#Compression|Compression]] scheme

[[#Encryption|Encryption]] scheme:
''(flags >> 4) & 0xF''
* 0 : None
* 1 : XOR
* 2 : Blowfish
* 3 : Blowfish ''(Dragon Age 2 only)''
* Any other value is invalid.

[[#Compression|Compression]] scheme:
''(flags >> 29) & 0x7''
* 0 : None
* 1 : Bioware Zlib ''(Dragon Age 2 only)''
* 7 : Headerless Zlib ''(Dragon Age 2 only)''
* Any other value is invalid.

|-
| moduleId
| uint32
| This id is used to look up the password for the ERF when encryption is present, for example, DLC.

|-
| passwordDigest
| byte[16]
| [http://en.wikipedia.org/wiki/MD5 MD5] sum of password (as a string).

|}

=== Table of Contents ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| toc
| [[#Entry_2|entry]][fileCount]
|

|}

==== Entry ====

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| name
| utf16[32]
|

|-
| offset
| uint32
| Offset to entry file data, from start of ERF file.

|-
| packedSize
| uint32
| Packed length of file data.

|-
| unpackedSize
| uint32
| Unpacked length of file data.

|}

== Version 3.0 ==

Found in Dragon Age 2.

This version introduces hashed file names (and extensions) for faster file lookup.

All values are [http://en.wikipedia.org/wiki/Endianness#Little-endian little-endian].

=== Header ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| magic
| utf16[8]
| "ERF V3.0", a total of 16 bytes.

|-
| stringTableSize
| uint32
| Size of the file name string table.

|-
| fileCount
| uint32
| Number of files encapsulated in this ERF.

|-
| flags
| uint32 [http://en.wikipedia.org/wiki/Bit_field Bit flags]
|

Flags:
* 0x00000001 (bit 0) : Unknown
* 0x000000F0 (bits 4 - 7) : [[#Encryption|Encryption]] scheme
* 0xE0000000 (bits 29 - 31) : [[#Compression|Compression]] scheme

[[#Encryption|Encryption]] scheme:
''(flags >> 4) & 0xF''
* 0 : None
* 1 : XOR
* 2 : Blowfish
* 3 : Blowfish ''(Dragon Age 2 only)''
* Any other value is invalid.

[[#Compression|Compression]] scheme:
''(flags >> 29) & 0x7''
* 0 : None
* 1 : Bioware Zlib ''(Dragon Age 2 only)''
* 7 : Headerless Zlib ''(Dragon Age 2 only)''
* Any other value is invalid.

|-
| moduleId
| uint32
| This id is used to look up the password for the ERF when encryption is present, for example, DLC.

|-
| passwordDigest
| byte[16]
| [http://en.wikipedia.org/wiki/MD5 MD5] sum of password (as a string).

|-
| stringTable
| char[stringTableSize]
| Zero-padded strings referenced by nameOffset in [[#Entry_3|TOC entries]]. Strings are stored as ASCII. Note that not all TOC entries have to reference an entry in the string table.

|}

=== Table of Contents ===

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| toc
| [[#Entry_3|entry]][fileCount]
|

|}

==== Entry ====

{| border="1" cellspacing="2" cellpadding="2" width="100%"

! width="20%" | Name
! width="20%" | Type / Size
! width="60%" | Info

|-
| nameOffset
| int32
| Offset in stringTable of the file name for this entry. -1 if none.

|-
| nameHash
| uint64
| [http://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function FNV64 hash] of lowercased file name, including path and extension.

|-
| typeHash
| uint32
| [http://en.wikipedia.org/wiki/Fowler–Noll–Vo_hash_function FNV32 hash] of lowercased file extension.

|-
| offset
| uint32
| Offset to entry file data, from start of ERF file.

|-
| packedSize
| uint32
| Packed length of file data.

|-
| unpackedSize
| uint32
| Unpacked length of file data.

|}

== File Data ==

=== Encryption ===

When the appropriate flags are set in the header, a digest of the password will be present. File data will be encrypted depending on what scheme is specified.

You may not extract such third party ERF files unless you have permission to do so. This is provided only for purposes when you want to encrypt your own ERF files.

==== Blowfish ====

File data will be encrypted using [http://en.wikipedia.org/wiki/Blowfish_(cipher) Blowfish] in ECB.

'''To encrypt:'''
* Generate random number (represented as string, base 10), such as "12345678".
* Generate digest of the string (without null-terminating character) using MD5 and store it in the header.
* Convert those digits to 64-bit integer (0x0000000000BC614E).
* Store it in byte array in little-endian encoding (0x4E, 0x61, 0xBC, 0x00, 0x00, 0x00, 0x00, 0x00).
* Use that array as key to initialize Blowfish.
* Pad your content to multiple of 8 with zeroes.
* Encrypt the data.

'''To decrypt:'''
* You already know your password ("12345678").
* Generate its MD5 digest and compare it with the digest in the header.
* If they don't match file is either damaged or you forgot your password. If you forgot, brute-force attack should take no more than 1 minute. You '''may not''' perform brute-force attack on ERF files that are not yours unless you have permission from its owner to do so.
* Convert those digits to 64-bit integer (0x0000000000BC614E).
* Store it in byte array in little-endian encoding (0x4E, 0x61, 0xBC, 0x00, 0x00, 0x00, 0x00, 0x00).
* Use that array as key to initialize Blowfish.
* Decrypt the data.

==== XOR ====

Much of the process for protecting data with [http://en.wikipedia.org/wiki/Exclusive_or XOR] is the same as for Blowfish.

'''To encrypt:'''
* Generate random number (represented as string, base 10), such as "12345678".
* Generate digest of the string (without null-terminating character) using MD5 and store it in the header.
* Convert those digits to 72-bit integer (0x000000000000BC614E).
* Store it in byte array in little-endian encoding (0x4E, 0x61, 0xBC, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00).
* XOR each block of 9 bytes with the array.
* If the final block is not 9 bytes, XOR the block with the first N bytes of the array, where N is the length of the block.

'''To decrypt:'''
* You already know your password ("12345678").
* Generate its MD5 digest and compare it with the digest in the header.
* If they don't match file is either damaged or you forgot your password. If you forgot, brute-force attack should take no more than 1 minute. You '''may not''' perform brute-force attack on ERF files that are not yours unless you have permission from its owner to do so.
* Convert those digits to 72-bit integer (0x000000000000BC614E).
* Store it in byte array in little-endian encoding (0x4E, 0x61, 0xBC, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00).
* XOR each block of 9 bytes with the array.
* If the final block is not 9 bytes, XOR the block with the first N bytes of the array, where N is the length of the block.

=== Compression ===

'''''Compression is not supported by Dragon Age: Origins.'''''

Compression is supported by Dragon Age 2.

Note that if encryption is enabled:
* When decompressing, file data must first be decrypted.
* When compressing, file data must be encrypted after compression.

==== Bioware Zlib ====

File data is compressed with [http://en.wikipedia.org/wiki/Zlib zlib], with a custom 1-byte (commonly observed as 0xF9) header (vs the normal 2-byte zlib header).

The upper 4 bits of the header byte represent the window bits. The lower 4 bits are currently unknown.

==== Headerless Zlib ====

File data is compressed with [http://en.wikipedia.org/wiki/Zlib zlib], the normal 2-byte zlib header is omitted.

[[Category:File types]]

Talk:GFF

2011-02-28T04:20:10Z

Mephales: /* GFF V4.1 */ new section

Here's some more info on it for consideration. As is, the wiki article leaves you to guess at a lot of what's going on. Listing it here for now so I don't lose track of it.
* http://social.bioware.com/forum/1/topic/8/index/464166
* http://social.bioware.com/forum/1/topic/72/index/508109
--[[User:FollowTheGourd|FollowTheGourd]] 06:56, 26 January 2010 (UTC)

----
== Field information incorrect ==
After checking a .das file with the GFF editor that comes with the toolset, it would appear that FieldType is TypeID *then* flags. Also the flags are Reference (MSB, bit 16), List (bit 15), and Struct (bit 14).

This can be verified with:

<pre>
# let data be an array of the gff file
# let offset be the start of a field in the field array
(id, type, flags, index) = unpack("I H H I),data[offset:offset+12])

isList = (flags & (1 << 15) >0)
isStruct = (flags & (1 << 14) >0)
isRef = (flags & (1 << 16) >0)
</pre>

Given this, I'm going to change the page. If I'm incorrect, feel free to revert and explain why here.

--[[User:Overture1943|Overture1943]] 19:22, 6 November 2009 (UTC)

:I say this is not easy, however i think the original was correct. I dont have big explanation at hand, the Bonesused List in MMH for example has Bit 1 set. The "Generic List" has Bit 1 and 3 set (Where with Bit3 Data contains a Struct reference). And lastly "Struct List" has Bit 1 and 2 set. (Where with bit2 the Data contains the Struct itself) All of them data are referenced in itself, however i think that is a given for Lists.
:With which if only Bit3 is set, then the Data isnt referenced, but a struct is referenced ,and it is not a list or so it seems to me. [[User:Eshme|Eshme]] 07:25, 15 March 2010 (UTC)
::This appears true in GAD files as well, solely Bit1 is set for a Float32 list. [[User:Eshme|Eshme]] 22:51, 15 March 2010 (UTC)

== GFF V4.1 ==

Due to the new ERF V3.0 format, we don't have a lot to go on for the new GFF V4.1 format. The only example I know of is profile.dap. However, here are my thoughts. The header has two additional uint32 fields, giving us:
<pre>struct header {
char[8] magic = "GFF V4.1";
char[4] platform;
char[4] file_type;
char[4] file_version;
uint32 struct_count;
uint32 unknown_1;
uint32 string_offset;
uint32 data_offset;
}</pre>
I haven't figured out what the first new field is, but the field I call string_offset appears to be an offset from the beginning of the file to a new list of zero-terminated strings, just like the ERF V3.0 format has for filenames. And as a result of this, it appears that ECStrings are now represented as offsets into this string section. They also do not start with the length of the string, and it appears the strings are most probably encoded in utf-8, but definitely not in utf-16 anymore.

I had an alternate theory that involved the the strings still being part of the data section and the root structure being able to start later in the data section, but the offsets for lists are still relative to the root structure's start, so they appear to be different sections.

I imagine there will be a few more surprises in store for us beyond these, like what the purpose of the first new field is, but it looks like we need to puzzle out the ERF V3.0 format or wait for a toolset so we can get more GFF V4.1 samples. [[User:Mephales|Mephales]] 04:20, 28 February 2011 (UTC)

Dragon Age Toolset Wiki - User contributions [en]

ERF

Talk:GFF