Detect path encoding for compatibility

[tang3w]

Fixes #373

Changes proposed in this PR

• This PR leverages the function NSString.stringencoding to detect and convert the path data into a valid Swift String, to fix the problem when dealing with ZIP files which contains GB-18030 encoded entries. This improvement only takes effective when function FileManager.unzipItem is called without specifying the pathEncoding, and centralDirectoryStructure.usesUTF8PathEncoding is false.

Tests performed

All existing unit tests are passed

Further info for the reviewer

N/A

Open Issues

N/A

[tang3w]

@weichsel Any chance you can look at this pull request?

[tang3w]

@weichsel I pushed a commit to fix the CI issues, could you take another look?

[weichsel]

Hi tang3w,
Thanks for looking into additional path decoding support. However, the heuristic approach implemented in this PR doesn’t align with the ZIP spec . In “APPENDIX D - Language Encoding (EFS)”, it’s mentioned that paths can either be CP437 or UTF-8 if the general purpose bit “11” is set. This is the current implementation. Additionally, the “0x0008 Extended Language Encoding Extra Field” can provide support for other encodings. However, we currently don’t allow setting this field or reading its content when decoding paths. Therefore, the spec-aligned approach would be to:

• Check if the encoding extra field contains any information.
• If it does, decode the path accordingly.

[tang3w]

Attaching the hex view of the sample test.zip if it helps:

@weichsel Does it mean we have to add support for "Extended Language Encoding Extra Field" first? Is the content 08 at the cursor above the target value to read out? Could you help outline the code-wise solution with the above hex view? Any chance that you can help implement this part?

Additionally, the “0x0008 Extended Language Encoding Extra Field” can provide support for other encodings. However, we currently don’t allow setting this field or reading its content when decoding paths.

[weichsel]

I just had another look at "Appendix D". As far as I understand it now, this can only be used to provide an alternate, UTF-8 encoded version of the path (vs. providing info about the encoding - as I initially thought). So there is no spec-compliant way to encode paths in GB-18030 . One can only provide a path in "original" form (sequence of bytes) in the path field and a UTF-8 encoded version of that in the "Extended Language Encoding Extra Field" (+ a CRC32 of the original bytes).
You could have a look at relevant fields using the "ZIP" template of HexFiend (https://github.com/HexFiend/HexFiend) to check if the app/library that creates your archive properly sets the general purpose bit 11 and adds a UTF-8 version of the path.

[tang3w]

Hey @weichsel, I extracted one of the entries using HexFriend:

01A3DBF325E4BFA1E993B6E8A784E7ABA0E3809432303235E38095323734E58FB7EFBC88E99984E4BBB6EFBC895F546573742E646F6378

It does seem to include the utf-8 encoded file name as I can decode the file name with the test script below:

# the Extra Field bytes of a ZIP file entry
extra_fields = "01A3DBF325E4BFA1E993B6E8A784E7ABA0E3809432303235E38095323734E58FB7EFBC88E99984E4BBB6EFBC895F546573742E646F6378"
data = bytes.fromhex(extra_fields)
utf8_bytes = data[5:] # Skip CRC32 bytes for testing
print(utf8_bytes.decode("utf-8")) # Output "信银规章〔2025〕274号（附件）_Test.docx" as expected

Then seems like it's a missing feature for ZIPFoundation to utilize the extra field to get the expected path name?

[weichsel]

Thanks for looking into this! Could it be that your archive uses some Info-ZIP extensions, like those mentioned in section 4.6.9 of the App Note? Or perhaps another proprietary ZIP extension? Also, do you have the ability to control how the archive is created?

If the path can be decoded in a way that matches the spec, I’m happy to add that. I’m a bit hesitant about using heuristics, as they might affect existing users or not work the same way on all platforms.

[tang3w]

Could it be that your archive uses some Info-ZIP extensions, like those mentioned in section 4.6.9 of the App Note?

It seems to be - the identifier is 0x7075:

do you have the ability to control how the archive is created?

Unfortunately no, we don't have the ability to control how the zip file is created - it's totally up to the users to choose their own software to create the archive.

I’m a bit hesitant about using heuristics, as they might affect existing users or not work the same way on all platforms.

I understand. How about add something to let users opt in the heuristics then (e.g., allow users to pass in a list of encodings for detect) so existing users won't be impacted by the change?

If the path can be decoded in a way that matches the spec, I’m happy to add that.

@weichsel That would be great and appreciated! Is there a rough timeline?

[weichsel]

@weichsel That would be great and appreciated! Is there a rough timeline?

I don't have the capacity to add that myself at the moment. But I can have a look at a PR that considers the Info-ZIP header when decoding paths.

[tang3w]

@weichsel - I drafted a new pull request #375 which adds Info-ZIP support.

[weichsel]

@weichsel - I drafted a new pull request #375 which adds Info-ZIP support.

Already had a look - the new PR is the way we should handle this. Thanks!