Difference between revisions of "FF7/Kernel/Low level libraries"

From Final Fantasy Inside
< FF7‎ | Kernel
Jump to navigation Jump to search
my_wiki>Halkun
(LZS Archive)
m (59 revisions imported)
 
(51 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 
== PC to PSX comparison ==
 
== PC to PSX comparison ==
  
The files and data formats used in the PSX version of FF7 and it's PC port are conceptually the same thing, and accomplish the same tasks. That being said, they both have wildly different formats. Both ofwhich were derived from a third original format that is also somewhat different that the first two.
+
The files and data formats used in the PSX version of FF7 and it's PC port are conceptually the same thing, and accomplish the same tasks. That being said, they both have wildly different formats, both of which were derived from a third original format that is also somewhat different to the first two.
  
The original PSX FF7 was created in part using Sony's Psy-Q development library. This library usescommon formats that are "native" to the PSX. Often times a toolkit was used to convert commondevelopment- based formats, such as a TGA bitmap or a palleted GIF file, to something a little moresuited to Psy-Q, which would be a TIM file.
+
The original PSX FF7 was created in part using Sony's Psy-Q development library. This library uses common formats that are "native" to the PSX. Often, a toolkit was used to convert common development-based formats, such as a TGA bitmap or a palleted GIF file, to something a little more suited to Psy-Q, which would be a [[PSX/TIM_format|TIM file]].
  
During the porting process to the PC, some of the original artwork, (and artists for that matter), wereno longer available. This resulted in the port team having to use the Psy-Q versions of many files,which were ill suited for the PC architecture. In our example, the TIM file was converted to a TEX file,which would be manipulated in the PC's video memory a little more efficiently. Sometimes theoriginal artwork was available, such as the pictures of the characters within the menu, or the original MIDI files. Most often times it was not.
+
During the porting process to the PC, some of the original artwork (and artists for that matter) were no longer available. This resulted in the port team having to use the Psy-Q versions of many files, which were ill suited for the PC architecture. In our example, the [[PSX/TIM_format|TIM file]] was converted to a TEX file, which would be manipulated in the PC's video memory a little more efficiently. Sometimes the original artwork was available, such as the pictures of the characters within the menu, or the original MIDI files. Most often times it was not.
 +
 
 +
To make things a little more confusing, both systems also archive their data files in different ways, making the extraction and rendering of each file a bit of a bear. For the most part the data within each file is the same thing, just a little switched around. Here, we will cover the more generic files first, and then common files used in each module.
  
To make things a little more confusing, both systems also archive their data files in different ways,making the extraction and rendering of each file a bit of a bear. For the most part the data within eachfile is the same thing, just a little switched around. This manual will cover the more generic files first,and then common files used in each module.
 
  
 
== Data Archives ==
 
== Data Archives ==
Line 15: Line 16:
 
=== BIN archive data format ===
 
=== BIN archive data format ===
  
The BIN format comes as two different types. They both have the same extension, so one must open the file to see which format is which. They are best described as BIN Types and Bin-GZIP types
+
The BIN format comes as two different types. They both have the same extension, so one must open the file to see which format is which. They are best described as BIN Types and BIN-GZIP types.
 +
 
 +
==== BIN Type Archives ====
  
==== BIN Types ====
+
These are uncompressed archives. The header is 4 bytes long and gives the length of the file without the header and then the data beyond that.
  
These are uncompressed archives. The header consists of a 4 byte header that gives the length of the file without the header and then the data beyond that.
 
  
==== BIN-GZIP Types ====
+
==== BIN-GZIP Type Archives ====
  
 
Unless otherwise noted, these have a 6 byte header. After this are many gziped sections concatenated together.
 
Unless otherwise noted, these have a 6 byte header. After this are many gziped sections concatenated together.
  
{| border="0" cellpadding="3" cellspacing="1" style="background: rgb(0,0,0)" align="center"
+
{| class="wikitable"
! style="background:rgb(204,204,204); width:50px;" align="center" | Offset
+
! Offset
! style="background:rgb(204,204,204); width:50px;" | Length
+
! Length
! style="background:rgb(204,204,204); width:250px;" | Description
+
! Description
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | 0x0000
+
| 0x0000
|style="background:rgb(255,255,255);" | 2 bytes
+
| 2 bytes
|style="background:rgb(255,255,255);" | Length of gzipped section 1
+
| Length of gzipped section 1
 
|-
 
|-
|style="background:rgb(255,255,204)" align="center" | 0x0002
+
| 0x0002
|style="background:rgb(255,255,204);"  | 2 bytes
+
| 2 bytes
|style="background:rgb(255,255,204);" | Unknown
+
| Length of ungzipped section 1
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | 0x0004
+
| 0x0004
|style="background:rgb(255,255,255);" | 2 bytes
+
| 2 bytes
|style="background:rgb(255,255,255);" | File number
+
| File type*
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | 0x0006
+
| 0x0006
|style="background:rgb(255,255,255);" | Varies
+
| Varies
|style="background:rgb(255,255,255);" | [0x1F8B080000000000...] - Gzip header 1 and data
+
| [0x1F8B080000000000...] - Gzip header 1 and data
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | Varies
+
| Varies
|style="background:rgb(255,255,255);" | 2 bytes
+
| 2 bytes
|style="background:rgb(255,255,255);" | Length of gzipped section 2
+
| Length of gzipped section 2
 
|-
 
|-
|style="background:rgb(255,255,204);" align="center" | Varies
+
| Varies
|style="background:rgb(255,255,204);" | 2
+
| 2 bytes
|style="background:rgb(255,255,204);" | Unknown
+
| Length of ungzipped section 2
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | Varies
+
| Varies
|style="background:rgb(255,255,255);" | 2 bytes
+
| 2 bytes
|style="background:rgb(255,255,255);" | File number
+
| File type*
 
|-
 
|-
|style="background:rgb(255,255,255);" align="center" | Varies
+
| Varies
|style="background:rgb(255,255,255);" | Varies
+
| Varies
|style="background:rgb(255,255,255);" | [0x1F8B080000000000...] - Gzip header 2 and data
+
| [0x1F8B080000000000...] - Gzip header 2 and data
 
|-
 
|-
 
|style="background:rgb(255,255,255);" colspan="3" align="center" | ...
 
|style="background:rgb(255,255,255);" colspan="3" align="center" | ...
 
|}
 
|}
 +
<br />
 +
''*'' This particular value might be ignored by the whatever method is decompressing these archive types. Within archives it declares that the compressed file is a particular type. These values seem to be unique to the particular archive that is being opened and is not consistent between archives.<p />
 +
Example 1: Within the [[FF7/Kernel/Kernel.bin|KERNEL.BIN]] the first nine files are all different data sets so are numbered sequentially 0-8. All remaining files are text types and get labeled as type 9.<br />
 +
Example 2: Within the WINDOW.BIN file there are three files. The first two are type "0" and are textures. The third file is type "1" and not a texture.
  
 +
=== LZS Archives ===
  
=== LZS Archive ===
+
The [[FF7/LZS format|LZS format]] is used throughout the PSX version of Final Fantasy 7, often ending with the .lzs extension. LZS itself stands for Lempel-Ziv-Shannon-Fano, Statistical plus Arithmetic. It was originally developed by [http://oku.edu.mie-u.ac.jp/~okumura/index-e.html Professor Haruhiko Okumura] based on the work of [http://www.hpl.hp.com/about/bios/abraham_lempel.html Abraham Lempel] and [http://www.marconifoundation.org/pages/dynamic/fellows/fellow_details.php?roster_id=23 Jacob Ziv].
Under construction. I'm placing a [[FF7/LZS format|link here]] so I can move things about.
 
 
 
  
  
=== LGP Archive format for PC by [[User:Ficedula|Ficedula]] ===
+
=== LGP Archives ===
  
This section explains how the LGP archives from FF7PC are constructed. There's probably no reason why you'd need to know this (Plug: Use my [http://www.ficedula.com LGP Editor] !) but the file format might be useful to SOMEBODY.
+
The LGP file format is only used for the PC port of Final Fantasy 7. These are large "volume" type archives that hold most of the game's data. These archives can hold thousands of files. Unlike the BIN or LZS type files, this archive does reference the data within it by filename. Its file format is explained [[FF7/LGP format|here]].
  
Essentially the LGP file is split up into four (maybe less, depending on how you count it) sections.
+
== Textures ==
  
# File header/Table of contents
+
A texture is just a picture that is placed into video memory. It is later manipulated by the engine and displayed on the screen. The native format of a texture was the Psy-Q [[PSX/TIM format|TIM]] (Texture Image Map). This is used as the native format for the PSX version as well, with a few caveats explained below. The file can hold multiple color look up tables. This was one of the reasons why a video card on the PC that could do palleted data at high color depths was needed.
# CRC code
 
# Actual data
 
# File terminator
 
 
 
==== Section 1: File Header ====
 
 
 
This contains two parts: A header of fixed size, then the table of contents.
 
 
 
The first item is 12 bytes containing the file creator. This is a standard string, except it is "rightaligned". In other words the blank space comes BEFORE the actual text, not after. Oh: In FF7 it's always "SQUARESOFT" preceded by two nulls to make it 12 bytes. The only other thing you might see is the header "FICEDULA-LGP", which I use to indicate a file is an LGP *patch* one of my programs has constructed, not a complete archive.
 
 
 
Next is a four-byte integer saying how many files the archive contains.
 
 
 
Following this is the table of contents (TOC): One entry per file.
 
 
 
Each entry in the TOC has the following structure:
 
 
 
{| border="0" cellpadding="3" cellspacing="1" style="background: rgb(0,0,0)" align="center"
 
! style="background:rgb(204,204,204); width:80px;" align="center" | Offset
 
! style="background:rgb(204,204,204); width:200px;" | Length
 
|-
 
|style="background:rgb(255,255,255);" | 20 bytes
 
|style="background:rgb(255,255,255);" | Null terminated string, giving filename
 
|-
 
|style="background:rgb(255,255,255);" | 4 byte integer
 
|style="background:rgb(255,255,255);" | Position in this file where data starts for the file
 
|-
 
|style="background:rgb(255,255,255);" | 3 bytes
 
|style="background:rgb(255,255,204);" | Some sort of check code. Normally seems to be<br />14,0,0 but it does vary. Unsure about this.
 
|}
 
 
 
Simple!
 
 
 
==== Section 2: CRC Code ====
 
 
 
This code is used to validate the LGP archive. The bad news is I have no idea how to make it (I've figured out how to decode it, ie. find out whether the archive is valid ... but I can't create my own). The good news is you don't need to! The ONLY thing this CRC is based on is the number of files in the archive (maybe the filenames too ... haven't checked that). Anyway, the TOC is the only thing this check relates to. So if you're replicating an archive from FF7 for use in the game with the same number of files and filenames (and what ELSE would you use LGP archives for?) you can just copy the CRC section from an existing file. Cheap but effective :)
 
 
 
Normally it's 3602 bytes long. I think one archive was different? Maybe MAGIC.LGP - can't remember. Anyway, one normally-safe way of calculating the CRC size is to find the end of the TOC and the beginning of the first file. Anything in between is probably CRC code. (Not guaranteed to work! It works with "official" archives but editors - such as mine - can screw around with the TOC to achieve extra things).
 
 
 
==== Section 3: Actual Data ====
 
 
 
The data from the files. However it's not that simple: the TOC doesn't list how long each file is (somewhat useful!). It's done here. The offset in the TOC is actually the position of yet another file header. Format is:
 
 
 
{| border="0" cellpadding="3" cellspacing="1" style="background: rgb(0,0,0)" align="center"
 
! style="background:rgb(204,204,204); width:80px;" align="center" | Size
 
! style="background:rgb(204,204,204); width:200px;" | Description
 
|-
 
|style="background:rgb(255,255,255);" | 20 bytes
 
|style="background:rgb(255,255,255);" | Null terminated string, giving filename
 
|-
 
|style="background:rgb(255,255,255);" | 4 bytes
 
|style="background:rgb(255,255,255);" | File length
 
|-
 
|style="background:rgb(255,255,255);" | Varies
 
|style="background:rgb(255,255,255);" | The file data itself
 
|}
 
  
Simple!
+
=== TIM texture data format for PSX ===
  
 +
The [[PSX/TIM_format|TIM files]] are found both on raw format and also within several archives, including [[FF7/Kernel/Low_level_libraries#BIN_archive_data_format|BIN]], [[FF7/Kernel/Low_level_libraries#LZS_Archives|LZS]], or even MNU. The format proper has the ability to contain 24 bit bitmaps, but is not used in FF7. The format was created because the PSX does not have direct access to it's VRAM, and must go through the [[PSX/GPU|GPU]] for any graphic access. [[PSX/TIM format|A TIM file]] is a clean way to load a texture and color look up table into VRAM.
  
==== Section 4: Terminator ====
+
=== TEX texture data format for the PC ===
  
After the last piece of data comes the file descriptor. This is a simple string, except instead of being null-terminated it's terminated by the end of the file. It's "FINAL FANTASY 7" for all archives, except LGP patches, where it's "LGP PATCH FILE".
+
TEX files are texture files for the PC. The format for these files are located [[FF7/TEX format|here]].
  
==== Notes ====
 
  
The game is remarkably flexible about LGP archives. So long as the TOC and the CRC data is intact it'll accept just about anything.
+
== File formats for 3D models ==
  
* Example 1: The filename in the TOC and in the actual file header don't have to match. It only checks the TOC.  
+
During the development process, 3D models contain a good deal of information needed by the artist every time they save or load the model. When the model is finished, it is often exported and broken up into smaller files with many unneeded attributes stripped from them. When the models for FF7 were created, they were exported into Psy-Q's 3D library formats. These include [[PSX/RSD|resource data (.RSD)]], polygon data (.PLY), polygon groups (.GRP), materials (.MAT), [[PSX/TIM_file|textures (.TIM)]], [[PSX/HRC|skeletal hierarchy (.HRC)]], and animation (.ANM).
* Example 2: You can point two entries in the TOC at the same data and it works.  
 
* Example 3: You can have ANY junk in the data section so long as all the TOC entries point to a valid file header. Not every piece of data has to be "accounted" for by the TOC. There can be data not used.  
 
  
My LGP Editor uses this to its advantage in the Advanced Editor. If you want to replace a file in an LGP archive with your own copy, it just sticks the file on the end of the LGP, writes a new file terminator, and updates the TOC to point at the new file. (Advantage: Fast). It even lets you link two TOC entries to the same data :) or have "inactive" files in the archive that aren't referenced by any TOC entry.
+
The models are handled differently between modules. The models in the "battle" modules have a different animation system than the field models. When the models were converted to the PC version, they were taken from the Psy-Q formats to a more PC-friendly one. Some are even the original, uncompiled, Psy-Q files.  
  
I don't know whether the file terminator has to be intact, but for safety's sake my editor preserves it. The CRC DEFINTELY has to be present and correct. Also, if you're replacing an archive with you're own custom version make sure it has filenames in the TOC matching the ones in the old one, ne?
+
===  Model formats for PSX ===
  
Oh: The game doesn't check archive sizes so long as all filenames are present. So if you want, you could replace an archive containing 95 files with a 98-file archive, so long as 95 of those 98 names matched those present in the original 95-file archive! (There's no point in doing this - after all, the game won't use any files OTHER than the 95 it's expecting to find).
+
The Playstation models are stored in the following directories, \ENEMY1 \ENEMY2 \ENEMY3 \ENEMY4 \ENEMY5 \ENEMY6 (battle models), \FIELD (field models and field character models), \MAGIC (Summon magic), and \STAGE1 \STAGE2 (battle scenes). Battle model names for special characters and party characters are stored in \ENEMY6, all models of this type end in an .LZS extension.  The same goes with summon magic used they are stored with there animations etc. in \MAGIC with a .LZS extension.  The only exception to this extension is the FIELD models, which use the extensions BSX and BCX for scene models and character models respectively. The [[FF7/Playstation Battle Model Format|Playstation battle model format]], is different than the [[FF7/Field/BSX|Playstation field model format]], also the [[FF7/Playstation battle scene format]], is similiar but not identical to the [[FF7/Playstation Battle Model Format|Playstation battle model format]]. The [[FF7/Playstation magic model|Playstation magic model]] format is a work in progress.
  
Other point: I've heard reports on Qhimm's message board that once you've f***ed an archive and the game refuses to read it, it won't EVER read it until you reinstall - even if you fix the problem/restore from a backup. The idea was generally scorned and ignored, but I'll mention it because something like that happened to me. Then again, it COULD have been because I upgraded basically everything in my PC; so no solid conclusion to be drawn here.
+
=== Model Formats for PC ===
  
Further point: (due to changes in my LGP Tools/Cosmo programs) Sometimes, there are data "gaps" in the file that don't appear to be referenced by any file - even by an inactive file. This happens due to the way my programs update archives. If you're only using the TOC method to get at files (the easy way) then you won't notice this anyway. However, if you're stepping through the file header by header, even reading the unused ones, this can cause problems. If you use my program to update a file with one that's smaller than the original (can happen) then it writes it in, but leaves a gap after it (of course). However, to help you out, after the end of the file, it writes a 4 byte integer saying how much more space to skip over to reach the next file header. This really doesn't affect many things - only tools (like my Advanced LGP Editor) that bypass the TOC to construct their own file lists. FF7 never notices a thing.
+
The PC models are stored in the LGP files in the /DATA directory. The names for the models were obfuscated a little. The data can be found in the [[PSX/HRC|Hierarchy files (.HRC)]], [[PSX/RSD|Resource data files (.RSD)]], and [[FF7/P|Polygon files (.P)]].

Latest revision as of 05:19, 23 May 2019

PC to PSX comparison

The files and data formats used in the PSX version of FF7 and it's PC port are conceptually the same thing, and accomplish the same tasks. That being said, they both have wildly different formats, both of which were derived from a third original format that is also somewhat different to the first two.

The original PSX FF7 was created in part using Sony's Psy-Q development library. This library uses common formats that are "native" to the PSX. Often, a toolkit was used to convert common development-based formats, such as a TGA bitmap or a palleted GIF file, to something a little more suited to Psy-Q, which would be a TIM file.

During the porting process to the PC, some of the original artwork (and artists for that matter) were no longer available. This resulted in the port team having to use the Psy-Q versions of many files, which were ill suited for the PC architecture. In our example, the TIM file was converted to a TEX file, which would be manipulated in the PC's video memory a little more efficiently. Sometimes the original artwork was available, such as the pictures of the characters within the menu, or the original MIDI files. Most often times it was not.

To make things a little more confusing, both systems also archive their data files in different ways, making the extraction and rendering of each file a bit of a bear. For the most part the data within each file is the same thing, just a little switched around. Here, we will cover the more generic files first, and then common files used in each module.


Data Archives

To save space, quicken access time, and to obfuscate the file structure a little, most of the data files are stored in some kind of archive format. The archives remove such useful items as subdirectories and logical data placement. There is no real "native" format these are based on.

BIN archive data format

The BIN format comes as two different types. They both have the same extension, so one must open the file to see which format is which. They are best described as BIN Types and BIN-GZIP types.

BIN Type Archives

These are uncompressed archives. The header is 4 bytes long and gives the length of the file without the header and then the data beyond that.


BIN-GZIP Type Archives

Unless otherwise noted, these have a 6 byte header. After this are many gziped sections concatenated together.

Offset Length Description
0x0000 2 bytes Length of gzipped section 1
0x0002 2 bytes Length of ungzipped section 1
0x0004 2 bytes File type*
0x0006 Varies [0x1F8B080000000000...] - Gzip header 1 and data
Varies 2 bytes Length of gzipped section 2
Varies 2 bytes Length of ungzipped section 2
Varies 2 bytes File type*
Varies Varies [0x1F8B080000000000...] - Gzip header 2 and data
...


* This particular value might be ignored by the whatever method is decompressing these archive types. Within archives it declares that the compressed file is a particular type. These values seem to be unique to the particular archive that is being opened and is not consistent between archives.

Example 1: Within the KERNEL.BIN the first nine files are all different data sets so are numbered sequentially 0-8. All remaining files are text types and get labeled as type 9.
Example 2: Within the WINDOW.BIN file there are three files. The first two are type "0" and are textures. The third file is type "1" and not a texture.

LZS Archives

The LZS format is used throughout the PSX version of Final Fantasy 7, often ending with the .lzs extension. LZS itself stands for Lempel-Ziv-Shannon-Fano, Statistical plus Arithmetic. It was originally developed by Professor Haruhiko Okumura based on the work of Abraham Lempel and Jacob Ziv.


LGP Archives

The LGP file format is only used for the PC port of Final Fantasy 7. These are large "volume" type archives that hold most of the game's data. These archives can hold thousands of files. Unlike the BIN or LZS type files, this archive does reference the data within it by filename. Its file format is explained here.

Textures

A texture is just a picture that is placed into video memory. It is later manipulated by the engine and displayed on the screen. The native format of a texture was the Psy-Q TIM (Texture Image Map). This is used as the native format for the PSX version as well, with a few caveats explained below. The file can hold multiple color look up tables. This was one of the reasons why a video card on the PC that could do palleted data at high color depths was needed.

TIM texture data format for PSX

The TIM files are found both on raw format and also within several archives, including BIN, LZS, or even MNU. The format proper has the ability to contain 24 bit bitmaps, but is not used in FF7. The format was created because the PSX does not have direct access to it's VRAM, and must go through the GPU for any graphic access. A TIM file is a clean way to load a texture and color look up table into VRAM.

TEX texture data format for the PC

TEX files are texture files for the PC. The format for these files are located here.


File formats for 3D models

During the development process, 3D models contain a good deal of information needed by the artist every time they save or load the model. When the model is finished, it is often exported and broken up into smaller files with many unneeded attributes stripped from them. When the models for FF7 were created, they were exported into Psy-Q's 3D library formats. These include resource data (.RSD), polygon data (.PLY), polygon groups (.GRP), materials (.MAT), textures (.TIM), skeletal hierarchy (.HRC), and animation (.ANM).

The models are handled differently between modules. The models in the "battle" modules have a different animation system than the field models. When the models were converted to the PC version, they were taken from the Psy-Q formats to a more PC-friendly one. Some are even the original, uncompiled, Psy-Q files.

Model formats for PSX

The Playstation models are stored in the following directories, \ENEMY1 \ENEMY2 \ENEMY3 \ENEMY4 \ENEMY5 \ENEMY6 (battle models), \FIELD (field models and field character models), \MAGIC (Summon magic), and \STAGE1 \STAGE2 (battle scenes). Battle model names for special characters and party characters are stored in \ENEMY6, all models of this type end in an .LZS extension. The same goes with summon magic used they are stored with there animations etc. in \MAGIC with a .LZS extension. The only exception to this extension is the FIELD models, which use the extensions BSX and BCX for scene models and character models respectively. The Playstation battle model format, is different than the Playstation field model format, also the FF7/Playstation battle scene format, is similiar but not identical to the Playstation battle model format. The Playstation magic model format is a work in progress.

Model Formats for PC

The PC models are stored in the LGP files in the /DATA directory. The names for the models were obfuscated a little. The data can be found in the Hierarchy files (.HRC), Resource data files (.RSD), and Polygon files (.P).