109
edits
Changes
From Final Fantasy Inside
m
This section explains how LGP is an archive format used by Final Fantasy VII to store game assets. The format is used in the LGP archives from FF7 PC are constructed. If you're looking for a tool that already manages LGP archivesversion of the game to package various game resources including textures, models, scripts, try [[User:Ficedula|Ficedula]]'s [http://www.ficedula.com LGP Editor]and other data files.
Essentially the An LGP file is split up into four archive consists of a header, table of contents (maybe lessTOC), hash lookup table, optional path table for directories, depending on how you count it) sectionsand the actual file data. All multi-byte integers in the format are stored in '''little-endian''' format.
# File header/Table The archive terminates with the magic string "FINAL FANTASY7" to mark the end of contents# CRC code# Actual data# File terminatorthe file.
This contains two partsAn LGP archive is divided into six sections in the following order: A header of fixed size, then the table of contents.
Next '''Note:''' The hash function is case-insensitive, treating 'A' and 'a four-byte integer saying how many files the archive contains' identically.
Following this is the table of contents (TOC): One entry per file.=== Lookup Algorithm ===
Each entry in # Compute hash from the filename (first two characters of stem)# Read HashTable[hash]# If index is 0, file not found (empty bucket)# Search TOC has the following structure:entries starting at index-1 (converting to 0-indexed) for count entries# Match by exact filename comparison
{| border='''Example:''' To find "0test.dat" cellpadding="3" cellspacing:# Compute hash: hash('t') × 30 + hash('e') + 1 ="19 × 30 + 4 + 1" style="background575# Read HashTable[575]: {index: 42, count: rgb3}# Search TOC entries 41-43 (0,0,0-indexed: index-1)" align# Find entry where name =="centertest.dat"! style== Path Table == The path table immediately follows the hash table and stores directory paths for files. It uses a variable-length structure organized into "background:rgb(204,204,204); width:80px;path groups" alignfor files that share the same filename but exist in different directories. === Path Table Header === {| class="centerwikitable" | ! Offset! style="background:rgb(204,204,204); width:200px;" | Length! Size !! Type !! Description
This code is used to validate the LGP archive. The bad news is I have no idea how to make it (I've figured out how to decode it, ie. find out whether the archive is valid, but I can't create my own). The good news is you don't need to! The only thing this CRC is based on is the number of files Notes:'''* Path index in the archive (maybe the filenames too, haven't checked that). Anyway, the TOC entries is the only thing this check relates to. So if you're replicating an archive from FF7 for use in the game 1-indexed into path groups* Multiple files with the same number of files and filenames you can just copy the CRC section from an existing file.name but different paths share a path group* Empty path string means root directory* Maximum path length is 127 characters plus null terminator
Normally it's 3602 bytes long (one archive may be different, possibly MAGIC.LGP). Anyway, one normally-safe way of calculating the CRC size is to find the end of the TOC and the beginning of the first file. Anything in between is probably CRC code (this is not guaranteed to work. It works with "official" archives but editors - such as [http://www.ficedula.com LGP Editor] - can alter the TOC to achieve extra things).== File Data ==
==== Section 3: Actual Data ====File data blocks follow the path table. Each file consists of a 24-byte header followed by the raw file content.
The data from the files. However it's not that simple: the TOC doesn't list how long each file is (somewhat useful). It's done here. The offset in the TOC is actually the position of yet another file header. Format is: === File Header ===
After the last piece of data comes == Size Limits == The LGP format has the following technical limitations: {| class="wikitable"! Limit !! Maximum Value !! Reason|-| Files per archive || 65,535 || uint16 in header|-| File size || 4 GB || uint32 in file descriptor. This is a simple string, except instead of being header|-| Archive size || 4 GB || uint32 offsets in TOC|-| Filename length || 19 characters || 20 bytes with null terminator|-| Path length || 127 characters || 128 bytes with nullterminator|-terminated it's terminated by the end of the file. It's "FINAL FANTASY 7" for all archives, except LGP patches, where it's "LGP PATCH FILE".| Hash buckets || 900 || Fixed 30 × 30 grid (letters × letters)|}
→Introduction
=== LGP Archive format for PC by [[User:Ficedula|Ficedula]] =Introduction ==
==== Section 1: File Header ==Structure Overview ==
{| class="wikitable"! Section !! Size !! Description|-| Header || 16 bytes || Archive metadata and file count|-| Table of Contents || 27 bytes × file count || File entries with names and offsets|-| Hash Table || 3600 bytes (fixed) || Lookup table for fast file access|-| Path Table || Variable || Optional directory paths for files|-| File Data || Variable || Actual file contents with individual headers|-| Footer || 14 bytes || "FINAL FANTASY7" terminator string|} == Header == The first item header is 12 always 16 bytes containing and contains the archive's magic identifier and file creatorcount. This is a standard {| class="wikitable"! Offset !! Size !! Type !! Description|-| 0x00 || 2 bytes || uint16 || Reserved (always 0)|-| 0x02 || 10 bytes || char[10] || Magic string, except it is : "rightalignedSQUARESOFT". In other words the blank space comes before the actual text(ASCII, not after. In FF7 itno null terminator)|-| 0x0C || 2 bytes || uint16 || Number of files in archive|-| 0x0E || 2 bytes || uint16 || Reserved (always 0)|} '''Validation:'''s always The magic string at offset 0x02 must exactly match "SQUARESOFT" preceded by two nulls to make it 12 (ASCII, no null terminator within the 10 bytes). The only other thing you might see is == Table of Contents (TOC) == Immediately follows the header and contains one 27-byte entry for each file in the archive. {| class="FICEDULAwikitable"! Offset !! Size !! Type !! Description|-| 0x00 || 20 bytes || char[20] || Filename (null-padded, no path)|-| 0x14 || 4 bytes || uint32 || Absolute file offset in archive|-| 0x18 || 1 byte || uint8 || File type (always 0x0E / 14)|-LGP"| 0x19 || 2 bytes || uint16 || Path index (0 = no path, which I use 1+ = path table index)|} '''Notes:'''* Filenames are stored without directory paths* Filenames are null-terminated within the 20-byte field* The offset field points to indicate a the start of the File Header for this file* The path field references the Path Table (1-indexed); 0 means the file is an LGP in the root directory*patch* one Maximum filename length is 19 characters plus null terminator == Hash Table == The hash table immediately follows the TOC and is always exactly 3600 bytes (900 entries × 4 bytes). It provides fast O(1) lookup of my programs has constructedfiles by filename. === Hash Table Entry === Each entry is 4 bytes: {| class="wikitable"! Offset !! Size !! Type !! Description|-| 0x00 || 2 bytes || uint16 || Index into TOC (1-indexed, not 0 = empty bucket)|-| 0x02 || 2 bytes || uint16 || Count of consecutive entries in this bucket|} === Hash Function === The hash is computed from the filename (without path or extension) using only the first two characters of the file stem: <pre>hash_value = hash(first_char) × 30 + hash(second_char) + 1</pre> For filenames with only one character in the stem: <pre>hash_value = hash(first_char) × 30</pre> === Character Hash Values === The hash function maps characters to numeric values as follows: {| class="wikitable"! Character !! Hash Value|-| a complete archive.-z (case insensitive) || 0-25|-| 0-9 || 0-9|-| _ (underscore) || 10 (same as 'k')|-| - (hyphen) || 11 (same as 'l')|}
|-
|style0x00 || 2 bytes || uint16 || Number of path groups|} === Path Group ==="background Each path group contains all directory paths for files sharing the same filename:rgb(255,255,255);" | 20 bytes {|styleclass="background:rgb(255,255,255);wikitable" | Null terminated string, giving filename ! Offset !! Size !! Type !! Description
|-
|style="background:rgb(255,255,255);" 0x00 | 4 byte integer|style="background:rgb(255,255,255);" 2 bytes | Position | uint16 || Number of paths in this file where data starts for the filegroup
|-
|style="background:rgb(255,255,255);" 0x02 | 3 bytes|style="background:rgb(255,255,204);" Variable | Some sort | Path[] || Array of check code. Normally seems to be<br />14,0,0 but it does vary. Unsure about this. Path entries
|}
==== Section 2: CRC Code Path Entry === Each path entry is 130 bytes: {| class="wikitable"! Offset !! Size !! Type !! Description|-| 0x00 || 128 bytes || char[128] || Directory path (null-padded, no trailing slash)|-| 0x80 || 2 bytes || uint16 || TOC index this path belongs to|}
{| borderclass="0" cellpadding="3" cellspacing="1" style="background: rgb(0,0,0)" align="centerwikitable"! style="background:rgb(204,204,204); width:80px;" align="center" | Offset !! Size! style="background:rgb(204,204,204); width:200px;" | ! Type !! Description
|-
|style="background:rgb(255,255,255);" 0x00 || 20 bytes|style="background:rgb| char[20] || Filename (255,255,255same as in TOC);" | Null terminated string, giving filename
|-
|style0x14 || 4 bytes || uint32 || File size in bytes|} === File Content === Immediately follows the file header. The size is specified in the header's file size field. {| class="background:rgb(255,255,255);wikitable" ! Offset !! Size !! Type !! Description|-| 0x00 || file_size || byte[] || Raw file data| 4 bytes} == Footer == The archive ends with a 14-byte terminator string: {|styleclass="background:rgb(255,255,255);wikitable" | File length! Offset !! Size !! Type !! Description
|-
|style="background:rgb(255,255,255);" 0x00 || 14 bytes || char[14] | Varies|style="background:rgbFINAL FANTASY7" (255,255,255no null terminator);" | The file data itself
|}
==Reading Algorithm == Section 4 To read an LGP archive: Terminator # '''Read Header''' - Validate magic string "SQUARESOFT" and extract file count# '''Read TOC''' - Read file_count entries of 27 bytes each, storing filename, offset, type, and path index# '''Skip Hash Table''' - Skip 3600 bytes (or read for validation/fast lookup)# '''Read Path Table''' - Read path group count, then for each group read path count and path entries# '''Resolve Full Paths''' - For each TOC entry with path index > 0, look up path group at path_index-1, find path entry matching TOC index, and prepend path to filename# '''Read File Data''' (on demand) - Seek to TOC entry's offset, read 24-byte file header, read file_size bytes of content ==Writing Algorithm == To create an LGP archive: # '''Build Metadata''' - Compute hash for each file, group files by hash bucket, identify files needing path entries# '''Write Header''' - Write magic string, file count, and reserved fields# '''Write TOC''' - Write entries with placeholder offsets (will be updated later)# '''Write Hash Table''' - Populate 900 entries based on file hash groupings# '''Write Path Table''' - Group paths by filename collisions and write path groups# '''Write File Data''' - For each file write header and content, recording actual offset# '''Update TOC''' - Seek back to TOC section and update offsets with actual values# '''Write Footer''' - Append "FINAL FANTASY7" terminator
==== Notes == ==
The game is remarkably flexible about LGP archives. So long as the TOC and the CRC data is intact it'll accept just about anything.
* Example 1: The filename in the TOC and in the actual file header don't have to match. It only checks the TOC. * Example 2: You can point two entries in the TOC at the same data and it works. * Example 3: You can have ANY junk in the data section so long as all the TOC entries point to a valid file header. Not every piece of data has to be "accounted" for by the TOC. There can be data not used.
[http://www.ficedula.com / LGP Editor] uses this to its advantage in the Advanced Editor. If you want to replace a file in an LGP archive with your own copy, it just puts the file on the end of the LGP, writes a new file terminator, and updates the TOC to point at the new file. It even lets you link two TOC entries to the same data or have "inactive" files in the archive that aren't referenced by any TOC entry.
I don't know whether the file terminator has to be intact, but for safety's sake my editor preserves it. The CRC must be present and correct. Also, if you're replacing an archive with you're own custom version make sure it has filenames in the TOC matching the ones in the old one.
The game doesn't check archive sizes as long as all filenames are present. So if you want, you could replace an archive containing 95 files with a 98-file archive, so long as 95 of those 98 names matched those present in the original 95-file archive. (However there's no point in doing this when the game won't use any files other than the 95 it's expecting to find).
There are reports on [http://forums.qhimm.com / Qhimm's board] that once you've altered an archive and the game refuses to read it, it won't ever read it until you reinstall - even if you fix the problem/restore from a backup. The idea was generally scorned and ignored, but I'll mention it because something like that happened to me. No solid conclusion can be drawn here.
Sometimes, there are data "gaps" in the file that don't appear to be referenced by any file - even by an inactive file. If you're only using the TOC method to get at files (the easy way) then you won't notice this anyway. However, if you're stepping through the file header by header, even reading the unused ones, this can cause problems. If you use my program to update a file with one that's smaller than the original (can happen) then it writes it in, but leaves a gap after it (of course). However, to help you out, after the end of the file, it writes a 4 byte integer saying how much more space to skip over to reach the next file header. This really doesn't affect many things - only tools (like my Advanced LGP Editor) that bypass the TOC to construct their own file lists. FF7 never notices a thing.
===Useful downloads===
Below there are links to known programs that are capable to edit LGP archives:
*[http://www.sylphds.net/f2k3/programs/lgptools/lgptools160.zip LGP Tools] - with an Advanced LGP Editor allowing edit archive thoughoutly*[http://elentor.com/Projetos/FF7-Tools/Extracting/Emerald.zip Emerald] - has mass extracting/repacking function*[http://mirex.mypage.sk/FILES/unm_w082index.rar php?selected=1#Unmass Unmass] - general file extractor with LGP archives support
