360 video/photo zoom out/gyro control/view angles/lens corrections

[d3xt3r01]

Feature request

Which Nextcloud Version are you currently using: (see administration page)

30.0

Is your feature request related to a problem? Please describe.
Using insta360 X5 videos, the image is grainy and low quality. That is because it's fairly zoomed in and with a low viewing angle.
Also, this should be automatically be used when the file has this metadata:

 Side data:
      stereo3d: 2D
      spherical: equirectangular (0.000000/0.000000/0.000000)

This is also how youtube allows videos to be shown as 360.

It would also be great if this would work on the mobile version of the app also! I don't have an option to view it in 360 there.

Another nice option would be to have the ultra wide/asteroide/crystal ball/dewarp viewing on the photos.
Gyro control on the phone maybe?

Describe the solution you'd like
Ability to set the default zoom and/or angle.

Describe alternatives you've considered
Other viewers ( like bivrost ) allow it seems a little bit more zoom out.

[R0Wi]

Hi @d3xt3r01! First of all sorry for the delayed answer.

I think your ideas make sense from a usability perspective. Even though (as you might know) this app is just an integration of photo-sphere-viewer.js (for showing images) and marzipano (for showing 360 videos). So if you're facing issues regarding the quality of 360 videos, it might be worth creating a new issue for marzipano instead?

Regarding the Gyro control on mobile: this should already work since we're including PhotoSphereViewers modules for that. Doesn't this work on your device?

Sorry for not being able to provide you with some good news but from my perspective we need to stick to some open source components which are doing the visualization job for us. I know that there are always alternatives out there with a slightly different feature set but the two main compontents I mentioned were the ones we choose and we probably won't change them in the near future unless there's a serious reason for doing so.

[d3xt3r01]

Hi,

I imagine it's a busy project.

photo-sphere-viewer is just fine. The viewer itself is nice.

I looked at the demos on https://photo-sphere-viewer.js.org/demos/ . Is there a place I could embed some settings I saw there?

For photos:

• ability to change the default pitch in a configuration is useful because the default one now makes everything blurry and weird by default
• ability to change pitch it available in the navbar is easier for some people ( me included ) to zoom in/out.
• on the phone app, the viewer doesn't show up. I have to share the folder/image and view it in the browser.
  The above are available in the currently selected components (photo-sphere-viewer) they just need better integration with the nextcloud app and tweaks for the browser viewer.
• gyro doesn't seem to work for me in the mobile browser, would also be useful to be able to toggle it on/off
• no controls in the viewer in the mobile browser

For videos:

• There is no zoom in/out in the browser or the mobile app for videos. I'll open a feature request in their GH repo
• there is dedicated metadata in the file to detect it's a 360 video beforehand so there shouldn't be needed a dedicated button to view the 360 videos ( maybe the other way around )
• ability to change the default pitch
• no gyro controls or options

I understand that this was the chosen option. I don't know when it was done and how, but it doesn't matter. I agree it should be an open source component rather than rebuilding something already available.
Don't get me wrong, I'm just trying to ask for this app better for alot more uses!

The insta360 X series is getting a lot of traction. Gopro and DJI are also launching for a while now devices in this section!

Again, I'm not asking for the components to be changed with others ( the suggestions for alternatives were just in case anybody was open and needed the suggestions, they can be safely ignored ). All I'm saying is that there are cameras out there accessible to a lot of people that do 360 and the current components need a little bit of tweaking to make them more pleasant to use by default for these situations.

Thank you!

[R0Wi]

@d3xt3r01 thanks for your feedback. No worries, of course I'm open to discuss your points, even though I cannot promise that we can solve all of them 😄

So for the photo part we can certainly try to integrate the full functionality which is offered by PhotoSphereViewer. As mentioned, for example the gyro function should already work on mobile. If this is not the case I need to check - I remember it was working before but t.b.h. I'm also not using NC on mobile phone via browser.
I guess what we couldn't change (at least I don't know how to) is the behavior that the iOS and Android Nextcloud apps will probably just use their internal viewer for all the files. This would be something we'd need to ask the mobile App team.

there is dedicated metadata in the file to detect it's a 360 video beforehand so there shouldn't be needed a dedicated button to view the 360 videos ( maybe the other way around )

This would be interesting. That's exactly the approach we're taking for 360 images but I didn't know that this also exists for 360 videos. So maybe you could share some info where to find this metadata and how to read it from the file? The problem is (and we had the same for images), that those videos are not using a dedicated mediatype, which makes a file based distinction pretty hard since Nextcloud relies on MIME types rather than particular file contents.

If you think there's a better option for a 360 video player, feel free to make suggestions and we can discuss. To be honest the 360 video capability was always just the cherry on top. I wrote this app mainly to be able to view my Google Photospheres 😃

[d3xt3r01]

Openness is all we need!

Gyro didn't work for me on phone, I like it that way that way.

For the videos, here's how youtube recognizes the video as being 360. Seems to me to be a good enough requirement to be auto-discovered.

1. ffprobe reports this side data.

  Stream #0:0[0x1](und): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, bt709), 7680x3840 [SAR 1:1 DAR 2:1], 217839 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
      Side data:
        stereo3d: 2D, view: packed, primary eye: none
        spherical: equirectangular

2. the google spatial media spherical video rfc uuid ffcc8263-f855-4a93-8814-587a02521fdd as per rfc . I use this script to inject it.

#!/usr/bin/env python3
"""Minimal 360° video metadata injector"""
import struct, sys, os, argparse, tempfile, shutil

def inject_360_metadata(input_file, output_file=None, inplace=False):

    # https://github.com/google/spatial-media/blob/master/docs/spherical-video-rfc.md
    # Spherical metadata
    uuid_id = b"\xff\cc\x82\x63\xf8\x55\x4a\x93\x88\x14\x58\x7a\x02\x52\x1f\xdd"
    xml = b'<?xml version="1.0"?><rdf:SphericalVideo xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:GSpherical="http://ns.google.com/videos/1.0/spherical/"><GSpherical:Spherical>true</GSpherical:Spherical><GSpherical:Stitched>true</GSpherical:Stitched><GSpherical:StitchingSoftware>Spherical Metadata Tool</GSpherical:StitchingSoftware><GSpherical:ProjectionType>equirectangular</GSpherical:ProjectionType></rdf:SphericalVideo>'

    with open(input_file, 'rb') as f:
        data = bytearray(f.read())

    # Find moov box
    i = 0
    while i < len(data) - 8:
        size, box_type = struct.unpack('>I4s', data[i:i+8])
        if box_type == b'moov':
            moov_start, moov_size = i, size
            break
        i += size
    else:
        return print("Error: moov box not found")

    # Check if spherical metadata already exists
    if uuid_id in data:
        return print("360° metadata already present, skipping injection")

    # Find video trak box
    i = moov_start + 8
    while i < moov_start + moov_size - 8:
        size, box_type = struct.unpack('>I4s', data[i:i+8])
        if box_type == b'trak' and b'vide' in data[i:i+size]:
            trak_end = i + size
            # Create uuid box
            uuid_data = uuid_id + xml
            uuid_box = struct.pack('>I4s', 8 + len(uuid_data), b'uuid') + uuid_data
            # Insert uuid box and update sizes
            data[i:i+4] = struct.pack('>I', size + len(uuid_box))
            data[moov_start:moov_start+4] = struct.pack('>I', moov_size + len(uuid_box))
            data[trak_end:trak_end] = uuid_box
            break
        i += size
    else:
        return print("Error: video track not found")

    if inplace:
        # Use temporary file for inplace operation
        with tempfile.NamedTemporaryFile(mode='wb', delete=False, dir=os.path.dirname(input_file)) as temp_f:
            temp_file = temp_f.name
            temp_f.write(data)

        # Rename temporary file to original
        shutil.move(temp_file, input_file)
        print(f"360° metadata injected inplace: {input_file}")
    else:
        with open(output_file, 'wb') as f:
            f.write(data)
        print(f"360° metadata injected: {output_file}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Inject 360° metadata into MP4 files")
    parser.add_argument("input", help="Input MP4 file")
    parser.add_argument("output", nargs="?", help="Output MP4 file (required unless --inplace is used)")
    parser.add_argument("-i", "--inplace", action="store_true", help="Modify file inplace")

    args = parser.parse_args()

    if not os.path.exists(args.input):
        sys.exit(f"Error: {args.input} not found")

    if args.inplace:
        if args.output:
            sys.exit("Error: Cannot specify output file with --inplace")
        inject_360_metadata(args.input, inplace=True)
    else:
        if not args.output:
            sys.exit("Error: Output file required unless --inplace is used")
        inject_360_metadata(args.input, args.output)

[R0Wi]

Thanks for your feedback! So I need to check the Gyro feature - I'm sure this worked before 😄

Regarding auto-recognizing 360° videos, your post clearly shows that it's definitely possible to detect those on the NC server backend (btw. we're using a similar approach to identify Photosphere images). The spec from the link you shared seems to be slightly outdated, though:

Note: This metadata scheme is superseded by the Spherical Video V2 metadata specification.

So if we would implement some feature like this, we should cover both the old and new specs, which makes it even more complicated. Right now I don't have the capacity to take over the implementation part. Nevertheless I would be happy to see your PR if you're interested! Will keep this issue open for further discussions.