

2023-11-05 22:15 BugFix & Update: jSAVF 1.80 An user was interested in opening a large multi-save magnetic tape image file with jSAVF, so I have made a few changes to support that use-case. It also enabled me to add limited support for optical disk image files (ISO9660 only for now, not UDF) since these may also contain multiple save files. The tape imaging tool he was using produces a tape image file in the AWSTAPE format (see Description, Hercules-390 Emulator), which inserts 6 byte block headers before each tape block. This tape image format is not ideal given it describes the lengths of the previous and current block with 16-bit words, when modern tapes appear to have 256KB blocks or more. This will lead to overflows, so for example a block size described as 0xF000 in the block header can mean either 0x0F000, 0x1F000, 0x2F000 or 0x3F000 for a tape block size of 0x40000 (256KB) described in the ANSI X3.27-1978 tape file header labels. jSAVF works around the issue by looking at these possible offsets for an AWSTAPE block header with a previous block length corresponding to the block length of the current header modulo 65536, and checking that the header flags look correct. If unlucky there may be data at one of these alternative offsets which looks enough like the real block header, so in that case jSAVF will bail with an error. If you're in that case I may improve upon the current strategy to distinguish the real header from data (by looking a the next headers), but since it seems to work well enough for now I'll leave it as it is unless someone has a tape which can't be read this way. As scanning all the alternative block headers of a multi-gigabyte tape image file to find all the save files written on it can be a lengthy process, jSAVF will generate a compressed tape file index next to it named "xxx.jsavf_awstape_index.bin" where "xxx" is the original tape image file name when that takes more than 3s. The index contains the tape labels for all files on it, and offset tables to be able to quickly convert a save file offset to a tape file image position. This lets jSAVF open the file nearly instantly the next time, and doesn't take too much space (~1MB index for a ~30GB file). When there are multiple saves in the file, jSAVF will now prompt the user to select the one to open. For batch scripts this is handled using a new jsavf:openMultiSaveFile(path, selector) API which calls a user-provided selector function with the list of save files found in the tape or optical image file and expects the selected one to be returned. This allows a script to select it based on the save size, name, or order. The JEXL dependency was updated to v3.3 to make this possible, so if you find incompatibilities in your batch scripts please tell me about them so I can see if it's something I can fix. I've exposed a few more classes to the batch API which help converting text to int or long, and updated the batch API documentation accordingly. This version also fixes two bugs: Some extractions failed to determine the saved item structure because of an error in the way jSAVF was computing the position of the first section when the item header length was exactly 4096 bytes. Some large CSV extractions with CHAR VARYING or CLOB or BLOB fields couldn't be performed when the total amount of data in these fields was above some limit, which happened on tables with many records or with enough data in them. jSAVF now correctly handles tables with many records, but may still have issues with BLOB / CLOB whose values are very large. Don't hesitate to reach out to me if you're confronted with such an issue. The CSV extraction of BLOB fields was also modified to extract their values as a hexadecimal string, because putting raw binary inside a UTF-8 CSV file is not very useful. I've also updated the jSAVF dependencies and the embedded JRE bundled in the installable version for Windows. So jSAVF now needs a Java 21 version or better. If you encounter any issue with this version don't hesitate to reach out to me.

2023-11-05 22:15

BugFix & Update: jSAVF 1.80

An user was interested in opening a large multi-save magnetic tape image file with jSAVF, so I have made a few changes to support that use-case. It also enabled me to add limited support for optical disk image files (ISO9660 only for now, not UDF) since these may also contain multiple save files.

The tape imaging tool he was using produces a tape image file in the AWSTAPE format (see Description, Hercules-390 Emulator), which inserts 6 byte block headers before each tape block.

This tape image format is not ideal given it describes the lengths of the previous and current block with 16-bit words, when modern tapes appear to have 256KB blocks or more. This will lead to overflows, so for example a block size described as 0xF000 in the block header can mean either 0x0F000, 0x1F000, 0x2F000 or 0x3F000 for a tape block size of 0x40000 (256KB) described in the ANSI X3.27-1978 tape file header labels.

jSAVF works around the issue by looking at these possible offsets for an AWSTAPE block header with a previous block length corresponding to the block length of the current header modulo 65536, and checking that the header flags look correct. If unlucky there may be data at one of these alternative offsets which looks enough like the real block header, so in that case jSAVF will bail with an error. If you're in that case I may improve upon the current strategy to distinguish the real header from data (by looking a the next headers), but since it seems to work well enough for now I'll leave it as it is unless someone has a tape which can't be read this way.

As scanning all the alternative block headers of a multi-gigabyte tape image file to find all the save files written on it can be a lengthy process, jSAVF will generate a compressed tape file index next to it named "xxx.jsavf_awstape_index.bin" where "xxx" is the original tape image file name when that takes more than 3s. The index contains the tape labels for all files on it, and offset tables to be able to quickly convert a save file offset to a tape file image position. This lets jSAVF open the file nearly instantly the next time, and doesn't take too much space (~1MB index for a ~30GB file).

When there are multiple saves in the file, jSAVF will now prompt the user to select the one to open. For batch scripts this is handled using a new jsavf:openMultiSaveFile(path, selector) API which calls a user-provided selector function with the list of save files found in the tape or optical image file and expects the selected one to be returned. This allows a script to select it based on the save size, name, or order. The JEXL dependency was updated to v3.3 to make this possible, so if you find incompatibilities in your batch scripts please tell me about them so I can see if it's something I can fix. I've exposed a few more classes to the batch API which help converting text to int or long, and updated the batch API documentation accordingly.

This version also fixes two bugs:

Some extractions failed to determine the saved item structure because of an error in the way jSAVF was computing the position of the first section when the item header length was exactly 4096 bytes.
Some large CSV extractions with CHAR VARYING or CLOB or BLOB fields couldn't be performed when the total amount of data in these fields was above some limit, which happened on tables with many records or with enough data in them. jSAVF now correctly handles tables with many records, but may still have issues with BLOB / CLOB whose values are very large. Don't hesitate to reach out to me if you're confronted with such an issue.

The CSV extraction of BLOB fields was also modified to extract their values as a hexadecimal string, because putting raw binary inside a UTF-8 CSV file is not very useful.

I've also updated the jSAVF dependencies and the embedded JRE bundled in the installable version for Windows. So jSAVF now needs a Java 21 version or better.

If you encounter any issue with this version don't hesitate to reach out to me.