Skip to content

Support for concatenated snappy-in-java files #23

Description

@gwittel

When dealing with some legacy format files, I noticed that snzip will fail to read snappy-in-java format files that are concatenated together. The issue is when it encounters the 2nd file, it reads the 's' (0x73) from the header and aborts since its not a recognized id.

The simple workaround is to skip the next 6 bytes (nappy\0 ) similar to how the framing2 format implicitly skips the header (this is due to it reading 0xff 0x06 0x00 0x00 as 6, then skipping those 6 bytes (sNaPpY) with the fseek.

Before I sent a real PR I wanted to get some feedback. My quick and dirty workaround does not validate the 2nd header is actually a valid snappy header. However, framing2 doesn't do this either (it relies on the implicit skipping defined by the header format itself).

Creating test file:

$ echo 'hello' | ./snzip -t snappy-in-java > one.snappy
$ echo 'world' | ./snzip -t snappy-in-java > two.snappy
$ cat one.snappy two.snappy > three.snappy

Original version:

$ ./snzip -d -c three.snappy
hello
Unknown compressed flag 0x73

Patched:

$ ./snzip -d -c three.snappy
hello
world

Thoughts/preferences on patch approach?

Hacky version diff:

diff --git a/snappy-in-java-format.c b/snappy-in-java-format.c
index 0f95e1a..2b2579a 100644
--- a/snappy-in-java-format.c
+++ b/snappy-in-java-format.c
@@ -195,6 +195,16 @@ static int snappy_in_java_uncompress(FILE *infp, FILE *outfp, int skip_magic)
     case UNCOMPRESSED_FLAG:
       /* pass */
       break;
+       case 's':
+         /* s== 0x73 Possible concatenated block.
+          * Note that other framing formats like frame2 see 0xff and just skip
+          * the rest of the header due to the header being: 0xff 0x06 0x00 0x00 snappy
+          * (it reads the 3-byte chunk header length resulting in a block length of
+          * 6 bytes, and skips 6 bytes which happens to be == snappy)
+          */
+         /* Likely concatenated snappy file.  We read first byte, skip rest */
+         fseek(infp, SNAPPY_IN_JAVA_MAGIC_LEN - 1, SEEK_CUR); /* TODO strict check? */
+         continue;
     default:
       print_error("Unknown compressed flag 0x%02x\n", compressed_flag);
       goto cleanup;

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions