diff --git a/docs/mrtk3-overview/api-reference.md b/docs/mrtk3-overview/api-reference.md new file mode 100644 index 000000000..f885590a3 --- /dev/null +++ b/docs/mrtk3-overview/api-reference.md @@ -0,0 +1,15 @@ +# API reference — MRTK3 + +MRTK3 is now shipping as [a set of individually versioned packages](index.md#versioning). As a result, each MRTK3 package ships its API reference individually. Please find the links to the API references for MRTK3 packages below: + +- [MRTK Core Definitions](https://aka.ms/mrtk3coreapi) +- [MRTK Accessibility](https://aka.ms/mrtk3accessibilityapi) +- [MRTK Audio Effects](https://aka.ms/mrtk3audioapi) +- [MRTK Data Binding and Theming](https://aka.ms/mrtk3dataapi) +- [MRTK Diagnostics](https://aka.ms/mrtk3diagnosticsapi) +- [MRTK Graphics Tools](https://learn.microsoft.com/dotnet/api/Microsoft.MixedReality.GraphicsTools) +- [MRTK Input](https://aka.ms/mrtk3inputapi) +- [MRTK Spatial Manipulation](https://aka.ms/mrtk3spmanipapi) +- [MRTK UX Components](https://aka.ms/mrtk3uxcompapi) +- [MRTK UX Core](https://aka.ms/mrtk3uxcoreapi) +- [MRTK Windows Speech](https://aka.ms/mrtk3winspeechapi) diff --git a/docs/mrtk3-overview/architecture/architecture.md b/docs/mrtk3-overview/architecture/architecture.md new file mode 100644 index 000000000..b2b906987 --- /dev/null +++ b/docs/mrtk3-overview/architecture/architecture.md @@ -0,0 +1,40 @@ +# Architecture overview — MRTK3 + +![Architecture MRTK3](../images/MRTK_v3_Architecture.png) + +One of the goals with MRTK3 was to take everything we've learned from the start of MRTK2 back in early 2018, combine it with the work that's been done by our industry partners across OpenXR and Unity since then, and come out the other side with a strong, extensible foundation that allows MRTK to focus more on providing differentiators and an overall improved user (and developer!) experience. + +## Input and interactions + +The overall architecture of the input stack of MRTK3 is built on four foundational components: + +1. OpenXR + 1. [Khronos Specification](https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html) + 1. [Unity OpenXR Plugin documentation](https://docs.unity3d.com/Packages/com.unity.xr.openxr@latest) +1. [Unity subsystems](https://docs.unity3d.com/ScriptReference/UnityEngine.SubsystemsModule.html) +1. [Unity's Input System](https://docs.unity3d.com/Packages/com.unity.inputsystem@latest) +1. [Unity's XR Interaction Toolkit](https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@latest) + +along with a layer of MRTK-defined [interactors](interactors.md) and [subsystems](subsystems.md), providing features like poke and speech. + +### OpenXR + +OpenXR is the interface between an application and an XR runtime system, allowing for a common set of features to be called generically and allow the hardware-specific implementation to be handled by the XR runtime. Adopting this in MRTK3, along with Unity's Input System, lets Unity handle more of the cross-platform and extensible input story while allowing MRTK3 to focus on helping you build rich experiences on top. + +### Subsystems + +[Subsystems](subsystems.md) and Unity's [SubsystemManager](https://docs.unity3d.com/ScriptReference/SubsystemManager.html) should be conceptually familiar to MRTK2 users, as they're the new "data providers". The idea is that different platforms or services can provide an implementation of a specific type of MRTK subsystem and have that run when relevant, providing data to MRTK3 and the app overall, just like data providers did to the various systems in MRTK2. Since we're focusing on OpenXR, the goal is that many features are covered by a cross-vendor EXT extension in OpenXR and multiple subsystems aren't needed, but vendor-specific extensions can represent early tech advancements that we want to support. + +### Unity Input System + +Conceptually, Unity's Input System will also feel familiar to users of the MRTK2 controller mapping profile. It provides a central place for mapping the buttons and other input axes on a controller or hand to a set of actions. These actions are then consumed by Unity's XR Interaction Toolkit (XRI) and MRTK3, so the object being interacted with doesn't care as much about _what_ is manipulating it, just that it _is_. + +### XR Interaction Toolkit + +XRI provides a foundation of interactors and interactables. MRTK builds upon this with its own set of [interactors](interactors.md) and [interactables](interactables.md), allowing for additional features like articulated hand tracking, gaze, and pinch. + +## See also + +- [Subsystems](subsystems.md) +- [Interactors](interactors.md) +- [Interactables](interactables.md) diff --git a/docs/mrtk3-overview/architecture/images/UGUI.svg b/docs/mrtk3-overview/architecture/images/UGUI.svg new file mode 100644 index 000000000..6fb37a1e6 --- /dev/null +++ b/docs/mrtk3-overview/architecture/images/UGUI.svg @@ -0,0 +1,15 @@ + + + + + + + + + + + + + + + diff --git a/docs/mrtk3-overview/architecture/images/configuration.png b/docs/mrtk3-overview/architecture/images/configuration.png new file mode 100644 index 000000000..fc4b2d377 Binary files /dev/null and b/docs/mrtk3-overview/architecture/images/configuration.png differ diff --git a/docs/mrtk3-overview/architecture/images/create-mrtk-asset.png b/docs/mrtk3-overview/architecture/images/create-mrtk-asset.png new file mode 100644 index 000000000..1113432d4 Binary files /dev/null and b/docs/mrtk3-overview/architecture/images/create-mrtk-asset.png differ diff --git a/docs/mrtk3-overview/architecture/images/create-new-asset.png b/docs/mrtk3-overview/architecture/images/create-new-asset.png new file mode 100644 index 000000000..b88a68d41 Binary files /dev/null and b/docs/mrtk3-overview/architecture/images/create-new-asset.png differ diff --git a/docs/mrtk3-overview/architecture/images/interactable_classes.svg b/docs/mrtk3-overview/architecture/images/interactable_classes.svg new file mode 100644 index 000000000..4d65b8240 --- /dev/null +++ b/docs/mrtk3-overview/architecture/images/interactable_classes.svg @@ -0,0 +1,19 @@ + + + + + + + + + + + + + + + + + + + diff --git a/docs/mrtk3-overview/architecture/images/pressable.png b/docs/mrtk3-overview/architecture/images/pressable.png new file mode 100644 index 000000000..883ede73d Binary files /dev/null and b/docs/mrtk3-overview/architecture/images/pressable.png differ diff --git a/docs/mrtk3-overview/architecture/images/profiles.png b/docs/mrtk3-overview/architecture/images/profiles.png new file mode 100644 index 000000000..87e5f05f0 Binary files /dev/null and b/docs/mrtk3-overview/architecture/images/profiles.png differ diff --git a/docs/mrtk3-overview/architecture/images/selectedness.svg b/docs/mrtk3-overview/architecture/images/selectedness.svg new file mode 100644 index 000000000..4dbb4bd5b --- /dev/null +++ b/docs/mrtk3-overview/architecture/images/selectedness.svg @@ -0,0 +1,16 @@ + + + + + + + + + + + + + + + + diff --git a/docs/mrtk3-overview/architecture/interactables.md b/docs/mrtk3-overview/architecture/interactables.md new file mode 100644 index 000000000..762937a3e --- /dev/null +++ b/docs/mrtk3-overview/architecture/interactables.md @@ -0,0 +1,55 @@ +# Interactables — MRTK3 + +MRTK builds on the `XRBaseInteractable` provided by Unity's XR Interaction Toolkit. The existing interactable behavior and API is fully supported in MRTK, and all of our custom interactables obey the existing XRI interactable API. + +For developers new to XRI, we _strongly_ recommend that you first review Unity's [XRI architecture documentation](https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@2.0/manual/architecture.html). + +To expand upon the interactable mechanisms included in XRI, MRTK offers two base classes upon which advanced interactions can be built, one extending the other. + +![Interactables inheritance diagram](images/interactable_classes.svg) + +- `MRTKBaseInteractable : XRBaseInteractable` + - This class offers filtering and flagging for different types of interactors. While the base XRI `XRBaseInteractable` doesn't discriminate between interactor types, `MRTKBaseInteractable` provides convenience functions for checking whether common types of interactions are occurring. Convenience properties like `IsGazeHovered` or `IsGrabSelected` are shortcuts to querying whether a participating interactor implements a given interface (correspondingly, `IGazeInteractor` or `IGrabInteractor`). These flags are more performant than iterating through the list of `interactorsHovering` or `interactorsSelecting`. In addition, `MRTKBaseInteractable` can filter/reject certain types of interactors in the case that the developer wishes to exclude certain input modalities. +- `StatefulInteractable : MRTKBaseInteractable` + - While `MRTKBaseInteractable` adds flags and filters, and avoids adding any additional state to the interactable, `StatefulInteractable` introduces useful stateful features like toggling and variable selection. + +## Strict separation of state and visuals + +In MRTK 2.x, interactables were often responsible for driving their own visual effects, be it the compressing of a 3D button, a hover effect, or even just changing color on a click. The limitation of this approach is that the interaction logic is tightly bound to the visuals. If you were to redesign the visuals or use a different size/shape/displacement/etc. of button, the interaction script itself would need to change. + +In MRTK3, **interactables are pure state and interaction.** The interactable doesn't render any visual changes or effects based on its internal state. It's purely a collection of state and interaction logic that's highly portable between visual presentation setups. + +![Strict isolation of state and visuals](images/pressable.png) + +The same `PressableButton` script can be used to build a squishy ball, a pressable "trackpad"-like plane, or an abstract pressable that issues network events on press. The `PressableButton` script doesn't even care "where" it is; it could be inside a Canvas, or on a rigidbody. + +To drive visuals, a separate "visual driver" is used to poll the state from the interactable and render the appropriate feedback. `StateVisualizer` is the recommended low-code method for driving common visual feedback effects from interactable state, but developers are free to write their own custom visual drivers. For example, our button components generally use `StateVisualizer` for their advanced 3D + shader-based feedback effects, but we also provide an example `BasicPressableButtonVisuals` that shows how a simple visual driver can be authored in code. + +## Variable selection + +`StatefulInteractable`'s most useful additional feature over the base XRI functionality is support for variable `Selectedness`. While base XRI interactables are either selected or not selected, MRTK's `StatefulInteractable`s can be any floating-point fraction of selected. + +This concept is useful when working in XR, since nearly all forms of input are no longer binary states. Motion controllers often have analog triggers (or analog grips!), hand interactions can provide a variable "pinchedness", and volumetric press interactions can depress a button or pushable surface by a varying amount. You see these variable, analog interactions everywhere in XR, and MRTK is equipped to help developers build delightful interactions on top of these analog inputs. + +A wide range of different interactors and types of interactions can all contribute together to the overall Selectedness of an interactable. Notably, all interactors that implement `IVariableSelectInteractor` contribute their analog selection amount, typically through a `max()` of all participating interactors. This variable amount is combined with the binary, non-variable selections coming from vanilla-style interactors. + +For derived classes like `PressableButton`, the `Selectedness()` function is overridden to add an additional "ingredient" to the selectedness computation. Interactors that implement `IPokeInteractor` can contribute Selectedness based on their physical location and how they're physically pressing down on the interactable. Other derived classes can introduce other, arbitrary forms of selection. + +![Variable selectedness](images/selectedness.svg) + +For the interactables MRTK provides, `Selectedness()` and `isSelected` will always "agree"--in other words, you'll never observe a `Selectedness()` greater than the `SelectThreshold` without a corresponding XRI `isSelected` and an accompanying interactor in `interactorsSelecting`. + +> [!IMPORTANT] +> Your custom interactable subclasses can obviously override `Selectedness` to some other value that's completely disconnected from the XRI `isSelected`. However, our interactables don't do this, and we strongly discourage it. **In general, never write _interactions_ that do not have a corresponding _interactor_.** XRI selection will, in the vast majority of cases, be sufficient, and any custom interactions you build should be written as interactors. + +When you're creating a custom interactable that supports a new method of determining `Selectedness()`, simply override the method and combine your new selectedness with the existing selection amount. If you're using `StateVisualizer` or any other visual layer that listens to variable selection, it will respond accordingly to your new selection type. + +## Map UGUI events to XRI + +In some cases, it's desirable to have interactables respond to UGUI events, such as mouse, gamepad, or touchscreen input. The `UGUIInputAdapter`, which is a UGUI `Selectable`, receives UGUI events and forwards them to a `CanvasProxyInteractor`, if one is present. + +![UGUI adapter flow](images/UGUI.svg) + +When the `CanvasProxyInteractor` is notified of the UGUI events by the `UGUIInputAdapter`, it issues _equivalent_ XRI actions on the relevant interactable. The mapping between UGUI input and XRI actions is somewhat lossy and is an area of active development. + +With this system, existing XRI interactables that are built for immersive platforms, hands, motion controllers, and 3D input can react equally well to accessible 2D controls like mouse and gamepad. diff --git a/docs/mrtk3-overview/architecture/interactors.md b/docs/mrtk3-overview/architecture/interactors.md new file mode 100644 index 000000000..e132550b1 --- /dev/null +++ b/docs/mrtk3-overview/architecture/interactors.md @@ -0,0 +1,96 @@ +# Interactor Architecture — MRTK3 + +MRTK builds upon the set of interactors offered by Unity's XR Interaction Toolkit. Mixed reality features like articulated hand tracking, gaze, and pinch require more elaborate interactors than the set provided with XRI by default. MRTK defines new interactor interfaces, categorized generally by the input modality, and corresponding implementations. + +## Summary and Review + +For developers new to XRI, we recommend that you first review Unity's [XRI architecture documentation](https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@2.0/manual/architecture.html). MRTK interactors are subclasses of existing XRI interactors or implementations of the XRI interactor interfaces. See Unity's documentation on their interactor architecture which also applies to MRTK. + +### Good Citizens of XRI + +The custom MRTK interactors are well-behaved with respect to the default XRI interactor interfaces; from the perspective of XRI systems, they're indistinguishable from "vanilla" interactors. The inverse is also true; when building advanced interactables in MRTK, the default XRI interactors will still work for basic hover and select. It's part of the MRTK effort to be fully compatible with existing XRI projects. If you have an XRI application, MRTK interactables and UI controls will work with your existing "vanilla" XRI setup. + +### Abstraction of Input Modality + +The input device, the interactor performing the interaction, and the interaction events they generate are all architecturally isolated in XRI. This isolation is critical to the input abstraction strategy in MRTK3, and enables us to write cross-platform and cross-device interactions that function well in all contexts. + +From MRTK v2, there's a common instinct to code interactions specific to a particular input type or device. Many developers are accustomed to writing interactions that react specifically to a near grab, a far ray, or some other specific input type. + +While MRTK3 still allows for the disambiguation and detection of individual input modes, hard-coding interactions to specific individual input types is artificially limiting and reduces the flexibility of your interactions. More on this can be found in the [interactable architecture documentation](interactables.md), but the key for interactors is that they generally don't have to map 1:1 with input devices. + +### AttachTransform and Inversion of Control + +Much of what MRTK v2 did in "move logics" as part of `ObjectManipulator`, `Slider`, and so forth, is now the responsibility of the interactor itself. The interactor now controls its attachTransform to define how a specific type of manipulation behaves. One no longer needs to write complex interaction logic on the interactable that differs between input modalities; instead, your unified manipulation logic can listen to the `attachTransform`'s pose regardless of the input modality or the device driving it. + +For example, a `GrabInteractor`'s `attachTransform` is located at the grabbing point on the hand/controller. An `XRRayInteractor`'s `attachTransform` is located at the hit point at the ray's end. The `CanvasProxyInteractor`'s `attachTransform` is located wherever the mouse has clicked. For all of these different interactors, the interactable **_doesn't have to care about the type of interactor in order to respond appropriately to manipulations._** + +The interactable queries the `attachTransform` and can treat every `attachTransform` the same regardless of the interactor type. + +This approach is critical for compatibility with existing XRI interactors as well as future-proofing your interactions for input modalities that haven't yet been developed. If a new input method is introduced, you don't need to alter existing interactables if the new interactor generates a valid and well-behaved `attachTransform`. + +Thus, philosophically, the `attachTransform` _is_ the interaction logic. For any custom interactions, always give preference to writing a new interactor with new `attachTransform` logic rather than rewriting or extending interactables to be customized for your new interaction. In this way, all existing interactables can enjoy the benefits of your new interaction instead of only the ones you've rewritten or extended. + +### XRControllers and Input Binding + +Most interactors don't bind directly to input actions. Most derive from `XRBaseControllerInteractor`, which requires an `XRController` above the interactor in the hierarchy. The `XRController` binds to input actions and then propagates the relevant actions (select, and so forth) down to all attached interactors. + +Nonetheless, some interactors may need special input bindings or additional input that the `XRController` doesn't provide. In these cases, interactors have the option to bind directly to their own unique input actions or even use other non-Input-System sources for interaction logic. The XRI base classes prefer to listen to the `XRController`'s bindings, but these behaviors can be overridden to use external or alternative input sources. + +## Interfaces + +XRI defines the basic `IXRInteractor`, `IXRHoverInteractor`, `IXRSelectInteractor`, and `IXRActivateInteractor`. MRTK defines additional interfaces for interactors. Some expose additional information about MRTK-specific interactions, and others are simply for categorization and identification. These interfaces are all located within the **Core** package, while the implementations reside in other packages, including **Input**. + +> [!IMPORTANT] +> While these interfaces are helpful if you need to filter for a specific type of interaction, we recommend that you do _not_ hard-code your interactions to listen for these interfaces specifically. _In every situation, always give preference to the generic XRI **isSelected** and **isHovered**, rather than any interaction-specific interface_.

+Unless necessary, you shouldn't reference the concrete MRTK implementations of these interfaces in interactables unless it's absolutely necessary. In all cases, it's better to reference the interfaces. Explicitly referencing the concrete types will restrict your interactables to only work with the current, existing types. By referencing only the interfaces, you ensure compatibility with future implementations that may not subclass the existing implementations. + +### IVariableSelectInteractor + +Interactors implementing this interface can issue variable (that is, analog) selectedness to interactables. The variable select amount can be queried with the `SelectProgress` property. MRTK interactors that implement this interface include the `MRTKRayInteractor` and the `GazePinchInteractor`. Base interactables (the default XRI interactables, and `MRTKBaseInteractable`) won't be affected by the variable selection amount; `StatefulInteractable`, however, listens to this value and computes its `Selectedness` based on the `max()` of all participating variable and non-variable interactors. + +### IGazeInteractor + +Interactors that implement this interface represent the user's passive gaze, separate from any manipulation or intent. The MRTK implementation is `FuzzyGazeInteractor`, which inherits from the XRI `XRRayInteractor`, and adds fuzzy cone-casting logic. `XRBaseInteractable` will flag `IsGazeHovered` when an `IGazeInteractor` is hovering. + +### IGrabInteractor + +Interactors that implement this interface represent a physical near-field grabbing interaction. The `attachTransform` is defined as the grabbing point. The MRTK implementation is `GrabInteractor`, which subclasses XRI's `XRDirectInteractor`. + +### IPokeInteractor + +Interactors that implement this interface represent a poking interaction. Note that this doesn't necessarily imply a finger! Arbitrary interactors can implement this interface and offer poking interactions from non-finger sources. In one of the few instances where checking interactor interfaces is a good idea, interactables like `PressableButton` listen for `IPokeInteractor`s, specifically, to drive volumetric press. Any interactor that implements `IPokeInteractor` will induce 3D presses on buttons. + +`IPokeInteractor` exposes the `PokeRadius` property, which defines the characteristics of the poking object. The poke is considered to be centered on the `attachTransform` and extends outwards from the `attachTransform` by the `PokeRadius`. Interactables like `PressableButton` offset their 3D push distance by this radius, which can be driven by the user's physical finger thickness in the case of finger-based presses. + +The MRTK implementation of this interface is `PokeInteractor`. In our template project, we also provide another example of `IPokeInteractor` that's not finger-driven; `PenInteractor` provides poke interactions rooted on the tip of a virtual 3D stylus. + +### IRayInteractor + +Interactors that implement this interface represent a ray-based pointing interaction. The `attachTransform` represents the hit location of the ray on the surface of the targeted object during a selection. + +The MRTK implementation of this interface is `MRTKRayInteractor`, inheriting directly from the XRI `XRRayInteractor`. + +> [!NOTE] +> The XRI `XRRayInteractor` doesn't implement this MRTK interface. + +### ISpeechInteractor + +Interactors that implement this interface represent speech-driven interactions. The MRTK implementation is `SpeechInteractor`. + +The MRTK `SpeechInteractor`, internally, uses `PhraseRecognitionSubsystem` and subscribes to interactable registration events from the XRI `XRInteractionManager`. However, interactables need not be concerned about what subsystem is performing speech processing; `ISpeechInteractor`s generate the same XRI events (select, and so forth) that any other interactor does. + +### IGazePinchInteractor + +This interface is simply a specialization of the `IVariableSelectInteractor` interface. Interactors that implement this interface are, implicitly, variable-select interactors. `IGazePinchInteractor`s expressly represent an indirectly targeted remote manipulation. A separate gaze-based interactor drives the target of the interaction, and the manipulation is by a hand or controller. `attachTransform` behaves the same way `IRayInteractor`'s `attachTransform` does; it snaps to the hit point on the target when a select is initiated. + +When multiple `IGazePinchInteractor`s participate in a single interaction, their `attachTransform`s are offset by their displacement from the median point between all participating pinch-points. Thus, interactables can interpret these `attachTransform`s in the same way they would for any other multi-handed interaction, like the `attachTransforms` from grab interactions, or ray interactions. + +The MRTK implementation is the `GazePinchInteractor`. + +### IHandedInteractor + +Some interactors can choose to implement `IHandedInteractor` interface to explicitly specify that they're associated with a particular hand on a user. Some interactors aren't associated with handedness and thus don't implement this. The most obvious examples would be ones like `SpeechInteractor` or `FuzzyGazeInteractor`. + +The MRTK interactors that implement this interface are the `HandJointInteractor`, a generic, abstract `XRDirectInteractor` driven by an arbitrary hand joint, the `GazePinchInteractor`, and the `MRTKRayInteractor`. + +Interactables currently use this interface to fire certain effects when selected that must disambiguate between a left or right hand. The most notable example of this is the pulse effect in the UX components library. diff --git a/docs/mrtk3-overview/architecture/subsystems.md b/docs/mrtk3-overview/architecture/subsystems.md new file mode 100644 index 000000000..78d2e9871 --- /dev/null +++ b/docs/mrtk3-overview/architecture/subsystems.md @@ -0,0 +1,87 @@ +# Subsystems — MRTK3 + +MRTK3 leverages the Unity XR Subsystem Management infrastructure for writing extensible modules that can help provide cross-platform support for features like speech and hand tracking. These subsystems are initialized and loaded by Unity alongside the existing Unity-native subsystems like `XRMeshSubsystem` and `XRInputSubsystem`. See [the documentation for how Unity subsystems work](https://docs.unity3d.com/ScriptReference/UnityEngine.SubsystemsModule.html). + +## Philosophy + +In MRTK v2, "services" provided much of the functionality in the scene itself. They would instantiate objects, move objects around, update the scene hierarchy, etc. In MRTK3, the subsystems don't explicitly modify the scene. The MRTK3 subsystems are modular providers of data, information, or events or perform computation for end-users. If something in the scene should change or be acted upon based on data input, there must be a separate scene-based visualizer component to act on the data. This split ensures that the subsystems are non-destructive regarding scene changes and don't cause scene-related side effects. + +While MRTK v2 used systems and services liberally for processing input, MRTK3 generally uses OpenXR and the Unity Input System for cross-platform input. However, some types of data are not yet wrapped by the Input System. In these cases, we provide cross-platform interfaces through our subsystems. + +## MRTK subsystem lifecycle + +The subsystem definitions that are included with Unity's infrastructure offer simple lifecycle methods like `Start`, `Stop`, and `Destroy`. We extend this definition to include helpful "tick" methods, such as `Update`, `LateUpdate`, and `FixedUpdate`. Our `MRTKLifecycleManager` manages subsystems that implement our lifecycle interface. This lifecycle manager is the only MonoBehaviour involved in the subsystem architecture; this can be placed anywhere in the scene, but we tend to leave it somewhere on the Rig. + +## Querying + +Querying for a subsystem implementation is straightforward and performant. + +```c# +// Gets the first valid implementation of T that is started and running. +T mySubsystem = XRSubsystemHelpers.GetFirstRunningSubsystem(); + +// Gets the first valid implementation of T, even if it hasn't been started. +T mySubsystem = XRSubsystemHelpers.GetFirstSubsystem(); + +// If multiple implementations of T are loaded, get all of them. +List allOfThem = new List(); +GetAllRunningSubsystemsNonAlloc(allOfThem); +``` + +## Descriptors + +Different implementations of a subsystem can have different capabilities. For example, the different implementations of the `HandsSubsystem` can specify their capability for reporting physical data or synthesized data. This capability information is stored in the subsystem descriptor, and can be queried for any given implementation. + +```c# +// Get the first running hands subsystem. +var handsSubsystem = XRSubsystemHelpers.GetFirstRunningSubsystem(); + +// If we found one... +if (handsSubsystem != null) +{ + // Read the capability information off the implementation's descriptor. + bool isPhysicalData = handsSubsystem.subsystemDescriptor.IsPhysicalData; +} +``` + +## Profiles + +Not to be confused with MRTK 2.x's profiles, MRTK3 subsystem profiles are a per-deployment-platform asset that defines which subsystems are created and started. + +![Subsystem profiles, as shown in the MRTK project settings view.](images/profiles.png) + +Subsystems that have their corresponding checkbox checked will be created and started by the `MRTKLifecycleManager` and have their lifecycle methods called. Different profiles can be assigned to different deployment targets. + +The subsystems shown here are determined by which packages you've installed. If a package isn't installed, the subsystems associated with that package won't be shown here, and the list auto-refreshes. + +There's a pre-made `MRTKProfile` supplied as part of the MRTK v3 package. It's an immutable asset. However, if you'd like to create a custom selection of subsystems to run, you should create your `MRTKProfile` asset within your project. + +![Create your own MRTK subsystems](images/create-mrtk-asset.png) + +## Configuration + +Subsystems can be assigned configuration objects to customize their behavior. + +![Configuring a subsystem](images/configuration.png) + +These configuration objects are accessible from anywhere through the `XRSubsystemHelpers` API. + +```c# +XRSubsystemHelpers.GetConfiguration() +``` + +Subsystems define which config type is relevant to them in their `MRTKSubsystemAttribute`. Along with this, the attribute also defines several pieces of metadata, along with the concrete types of the implemented provider. For example, this is the attribute that the MRTK Hands Aggregator Subsystem uses. + +```c# +[MRTKSubsystem( + Name = "com.microsoft.mixedreality.hands", + DisplayName = "MRTK Hands Aggregator Subsystem", + Author = "Microsoft", + ProviderType = typeof(MRTKAggregator), + SubsystemTypeOverride = typeof(MRTKHandsAggregatorSubsystem), + ConfigType = typeof(MRTKHandsAggregatorConfig))] +``` + +As with Profiles, default Configuration assets are provided. They're immutable and must be duplicated to your project to be edited. You can also create a new asset through the asset creation menu. + +![New asset creation menu](images/create-new-asset.png) diff --git a/docs/mrtk3-overview/contributing.md b/docs/mrtk3-overview/contributing.md new file mode 100644 index 000000000..67deb604f --- /dev/null +++ b/docs/mrtk3-overview/contributing.md @@ -0,0 +1,61 @@ +# Contributing to MRTK3 + +MRTK3 is an open-source project under the MIT license. Community contributions are welcome and appreciated, both for new features and bug fixes. + +Contributing to MRTK3 is easy. We recommend using the `MRTKDevTemplate` Unity project as a convenient development testbed, as it already includes all the MRTK3 packages as local on-disk dependencies. [For more information, see the documentation on the MRTKDevTemplate project for more details on sample scenes and local on-disk dependencies.](getting-started/exploring-features/mrtk3-sample-scenes.md) + +## Contribution guide + +1. Fork the MRTK repository to your GitHub account. + +2. Clone your forked MRTK repository by following our guide on [starting from a template project](getting-started/setting-up/setup-new-project.md) Ensure you have the required tooling, especially the correct Unity version. To ensure you are on the right branch, clone using the command: + + ```pwsh + git clone --branch mrtk3 YOUR_GIT_URL + ``` + +3. Create a new branch for your changes or fixes. + + ```pwsh + git checkout -b yourchange_fix + ``` + +4. Open the `MRTKDevTemplate` template project located in `UnityProjects/MRTKDevTemplate`. You can add the project to your Unity Hub for easy access. + +5. Make your desired changes and create unit tests that ensure your changes work as expected. Make sure to test across in-editor and deployed to device. Commit your changes to your branch. Publish your branch to your fork upstream. + +6. Open a pull request on the MRTK repo, targeting the `mrtk3` branch. Make sure to accurately describe the changes you've made and apply relevant labels to your pull request for better categorization and triage. If you're a new contributor to MRTK, you may need to sign our contribution agreement. + +7. Address any fixes requested by the community or the maintenance team and merge your PR after approval. + +## Writing tests + +Tests are a critical part of ensuring MRTK is a reliable foundation for high-quality mixed reality applications. Any new features that are added should have unit tests to ensure their functionality remains correct as other changes are made to the codebase in the future. + +To write unit tests, we recommend that you first look at the existing unit tests and learn how the MRTK test utilities and simulator are used to mock XR input. You can mock hand input, gaze, HMD position, and other basic input-related features. Here's some general advice for writing good unit tests: + +- Try to write more granular tests that evaluate smaller pieces of functionality, rather than larger monolithic tests. More granular unit tests allow maintainers to see which specific feature has been broken. More general end-to-end functionality tests are also appreciated but ensure that each smaller part of your feature is well tested to begin with. +- Make sure your test (and your feature) doesn't make any assumptions about the orientation or location of the user. Your tests and features should work at any arbitrary offset or rotation from the world origin. +- If your tests mock user input, make sure to subclass our `BaseRuntimeInputTests`, which ensures that the proper test harness is set up and torn down. +- Use NUnit parameterization to easily increase the variety and flexibility of your test. [See the documentation for parameterized NUnit tests here.](https://docs.nunit.org/articles/nunit/technical-notes/usage/Parameterized-Tests.html) +- Some inputs or interactions may take multiple frames to register. You can use `yield return RuntimeTestUtilities.WaitForUpdates()` to add extra frames of delay to your test if your interactions aren't registering. +- Try to write unit tests that execute as quickly as possible to avoid slow CI iteration times. +- Make sure you add the relevant test dependencies to the `package.json`, and the correct references to the test folder's assembly definition file. + +## Continuous integration + +Every pull request is subject to automated tests before being able to be merged. Other continuous integration (CI) jobs are also run on the resulting commit on the main development branch to ensure broken packages aren't deployed to the feed. + +If your tests are passing in-editor but fail in the CI run, you should run your tests locally in batch mode. Some types of tests may unexpectedly fail when running in no-graphics batch mode due to timing differences or other Unity quirks. Running your tests locally in batch mode helps identify these inconsistent tests before the CI does. + +Use the `Tooling/Tests/run_playmode_tests.ps1` script to run tests locally in batch mode. You'll need to close your Unity editor to do so. + +```pwsh +./Tooling/Tests/run_playmode_tests.ps1 +``` + +The script will generate output files in the `/out` folder, including both `.log` files and the test results `.xml`. You can filter which tests are run by passing a regular expression to the script. Custom Unity versions and project folder locations can also be provided as arguments. + +```pwsh +./Tooling/Tests/run_playmode_tests.ps1 -unityVersion 2021.3.5f1 -projectPath ../my/project/path +``` diff --git a/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-sample-scenes.md b/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-sample-scenes.md new file mode 100644 index 000000000..9d2512412 --- /dev/null +++ b/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-sample-scenes.md @@ -0,0 +1,79 @@ +# Exploring MRTK3 sample scenes + +Unlike MRTK2, MRTK3 isn't distributed as a Unity project. Instead, MRTK3 consists of a loosely coupled collection of individual UPM packages distributed through the [Mixed Reality Feature Tool](https://learn.microsoft.com/windows/mixed-reality/develop/unity/welcome-to-mr-feature-tool), as well as through our official GitHub repository. + +As a result, we no longer ship our sample scenes inside the MRTK library/package itself. Instead, we maintain the `UnityProjects` folder at the top level of the [GitHub repository](https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity), which contains any Unity projects we want to ship. Currently, this folder includes the `MRTKDevTemplate` project, which contains all of our example scenes and is configured to align with our recommended best settings. + +We also recommend using the `MRTKDevTemplate` project for local development when submitting fixes or changes. All of the packages are specified as local on-disk dependencies, making editing and submitting changes easy. Clone the repo and ensure you're on the `mrtk3` branch, and open the `MRTKDevTemplate` Unity project. + +Within `MRTKDevTemplate`, you can find all of our sample scenes. Most of the sample scenes are in `UnityProjects/MRTKDevTemplate/Assets/Scenes`, while some experimental or early-preview sample scenes are located in `UnityProjects/MRTKDevTemplate/Assets/Data Binding Example`. + +## Included sample scenes + +We list below just a few of them. + +### HandInteractionExamples + +This sample scene offers a wide variety of interaction examples. Despite the name, this scene is a good example of cross-platform input, including hand tracking, controller input, and mouse input. Examples of several different UI controls and interactables are present, including the volumetric UI systems. + +![Hand Menu](../../images/hand-interaction-examples.png) + +### BoundsControlExamples + +Various configurations of BoundsControl, showing both flattened and 3D bounds. + +### CanvasExample + +Shows a collection of UX components built with UnityUI. These UX components are built with a combination of XRI interactables and traditional UGUI event handlers. This combination enables flexibility and responsive design across a wide variety of input methods and contexts. + +### CanvasUITearsheet + +This scene showcases all available UI building blocks and their permutations in MRTK. All controls are based on the new Mixed Reality Design Language. + +### DialogExample + +This scene demonstrates use of the Dialog control. + +### EyeGazeExample + +Example of using the Gaze Interactor to highlight objects within a scene. + +### HandMenuExamples + +Demonstrates using a menu appearing beside the hand. + +### InteractableButtonExamples + +An example of different styles of interactable buttons. + +### NearMenuExamples + +Near interaction menu examples. + +### NonCanvasObjectBarExample + +Demonstrates the Object Bar component, which enables horizontal or vertical arrangement of arbitrary 3D objects. + +### NonCanvasUIBackplateExample + +The scene demonstrates `UIBackplate.prefab`, which you can use to construct various types of UI panels and menus. + +### SampleEmptyMRTKScene + +The sample empty MRTK scene only contains the core MRTK prefab (**MRTK XR Rig**) and the input simulator prefab (**MRTKInputSimulator**). It's intended to give developers an empty scene with only the MRTK essentials necessary to get started. + +### SlateDrawingExample + +A demonstration of using MRTK3 to create a basic drawing application. + +### SpatialMappingExample + +The spatial mapping example scene demonstrates using `ARMeshManager` (**MRTK XR Rig > ARSpatialMeshManager**) in MRTK3 to visualize the spatial mesh. + +### TabViewExample + +Shows a collection of toggles that control the visibility of associated game objects. + +### ToggleCollectionExample + +Demonstrates the `ToggleCollection` script, which allows multiple toggle interactables to be grouped. Only one toggle can be toggled at any given time. diff --git a/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-tutorials.md b/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-tutorials.md new file mode 100644 index 000000000..84583e0ab --- /dev/null +++ b/docs/mrtk3-overview/getting-started/exploring-features/mrtk3-tutorials.md @@ -0,0 +1,13 @@ +# Exploring MRTK3 tutorials + +Several tutorials have been created to help developers learn about MRTK3's various features and capabilities. + +## [MRTK3 Aquarium](https://learn.microsoft.com/windows/mixed-reality/develop/unity/mrtk3-aquarium): the in-editor tutorial + +The [MRTK3 Aquarium project](https://learn.microsoft.com/windows/mixed-reality/develop/unity/mrtk3-aquarium) provides a Unity in-editor tutorial that explores various MRTK3 features. Set in an underwater scene, you'll be introduced to the creatures of the aquarium and the objects that make up their habitat. Using MRTK3 features, you'll add interactivity to the aquarium, enabling you to create an aquarium of your very own! + +## [Zappy's Playground](https://learn.microsoft.comhttps://learn.microsoft.com/windows/mixed-reality/develop/unity/playground-tutorial) + +[Zappy's Playground](https://learn.microsoft.com/windows/mixed-reality/develop/unity/playground-tutorial) is a cross-platform developer sample project that showcases how to develop intuitive and comprehensive end-to-end experiences for mixed reality. It makes use of many advanced features present in MRTK3, such as Gaze Interaction, Hand Menus, and Spatial Audio. + +This sample is currently out of date and will be brought up to the current MRTK3 developments in the near future. diff --git a/docs/mrtk3-overview/getting-started/overview.md b/docs/mrtk3-overview/getting-started/overview.md new file mode 100644 index 000000000..5a7a3d589 --- /dev/null +++ b/docs/mrtk3-overview/getting-started/overview.md @@ -0,0 +1,22 @@ +# Getting started with MRTK3 + +Welcome to the MRTK3! This guide serves as a starting point for using MRTK to build and experience your app in AR/VR. It includes resources for becoming acquainted with the building blocks of MRTK, and guides users through setting up their project to deploying on device. + +## Where should I go next? + +[Exploring MRTK features](exploring-features/mrtk3-tutorials.md): Learn about MRTK's features through our tutorial project and explore our example scenes. + +[Set up an MRTK Project](setting-up/setup-dev-env.md): Learn how to create an AR/VR ready Unity project using MRTK + +[Test and Deploy your app](../test-and-deploy/overview.md): Learn how to test and deploy your application on a device. + +## Software Requirements + +To acquire and use MRTK3, the following software tools are required. + +| Software | Version | Notes | +| --- | --- | --- | +| [Microsoft Visual Studio](https://visualstudio.microsoft.com/) | 2019 Community edition or greater | Recommend Visual Studio 2022 | +| Unity | 2022.3 LTS or newer | Recommend using an LTS release | +| [Mixed Reality Feature Tool for Unity](https://aka.ms/mrfeaturetool) | | Used to acquire MRTK3 packages | +| Mixed Reality OpenXR Plugin | | Install via Mixed Reality Feature Tool | diff --git a/docs/mrtk3-overview/getting-started/setting-up/setup-dev-env.md b/docs/mrtk3-overview/getting-started/setting-up/setup-dev-env.md new file mode 100644 index 000000000..33255d224 --- /dev/null +++ b/docs/mrtk3-overview/getting-started/setting-up/setup-dev-env.md @@ -0,0 +1,23 @@ +# Setting up your development environment + +Before setting up a Unity Project with MRTK3, make sure you have the following prerequisites. + +- A Windows 10 or 11 PC +- Visual Studio 2022 with the required workloads (as noted in the [Installation Checklist](https://learn.microsoft.com/windows/mixed-reality/develop/install-the-tools)) +- Windows 10 SDK 10.0.18362.0 or later +- Unity Hub with Unity 2022.3 LTS or Unity 2021.3 LTS installed + +If your target platform is a HoloLens device, your Unity installation needs to include the Universal Windows Platform Support Module. + +![UWP Module Installation](../../images/setting-up/MRTK-Development-Setup-UWPModule.png) + +If your target platform is a Quest device, your Unity installation needs to include the Android Build Support Module and its submodules. More more information specifics, see the [Oculus Developer documentation](https://developer.oculus.com/documentation/unity/book-unity-gsg/#install-unity-editor). + +![Android Module Installation](../../images/setting-up/MRTK-Development-Setup-AndroidModule.png) + +## Next steps + +After setting up the development environment, there are few options for creating a Unity Project with MRTK3. + +- [Starting from a Template Project](setup-template.md): This guide walks you through cloning a template project, which is pre-configured to consume all MRTK3 packages. This template project is set up with Unity project settings for running your application on a device. +- [Starting from a New Project](setup-new-project.md): This guide walks you through adding vital MRTK3 packages to a new Unity project. The guide also helps you set up the Unity project settings for running your application on a device. diff --git a/docs/mrtk3-overview/getting-started/setting-up/setup-new-project.md b/docs/mrtk3-overview/getting-started/setting-up/setup-new-project.md new file mode 100644 index 000000000..0159f16ac --- /dev/null +++ b/docs/mrtk3-overview/getting-started/setting-up/setup-new-project.md @@ -0,0 +1,106 @@ +# Starting from a new project + +Since MRTK3 is a collection of loosely coupled packages, consuming MRTK3 is done differently than the way you consume MRTK 2.x. We don't ship MRTK as a Unity project, so you have to manually add MRTK3 packages to your project in order to consume them. + +You're not expected to consume every MRTK package. See [which features are useful to you](../../packages/packages-overview.md) and add only the dependencies that matter. + +## Setting up a new Unity project with MRTK3 + +### 1. Create a new Unity project + +Create a new Unity project with Unity 2021.3.21f1 or newer. Close the Unity project before proceeding to the next step. + +### 2. Import required dependencies and MRTK3 packages with Mixed Reality Feature Tool + +There are a handful of packages that MRTK3 uses that aren't part of this toolkit. To obtain these packages, use the [`Mixed Reality Feature Tool`](https://learn.microsoft.com/windows/mixed-reality/develop/unity/welcome-to-mr-feature-tool) and select the latest versions of the following in the **Discover Features** step. + +- **Platform Support → Mixed Reality OpenXR Plugin** +- **Spatial Audio → Microsoft Spatializer** (Optional) + +For MRTK3 packages, we highly recommend the following two packages to help you get started quickly: + +- **MRTK3 → MRTK Input** (Required for this setup) +- **MRTK3 → MRTK UX Components** + +These two packages, along with their dependencies (automatically added by the Feature Tool), will enable you to explore most of our UX offerings and create projects ready to be deployed to various XR devices. You can always come back to the Feature Tool and add more packages to your project later. + +Be sure to select the `org.mixedrealitytoolkit.*` packages, and not the deprecated packages. The `com.microsoft.mrtk.*` packages have been deprecated, and are no longer supported. + +![Selecting the default MRTK3 packages in Microsoft's Mixed Reality Feature Tool](../../images/mrtk3-featuretool-setup-packages.png) + +> [!NOTE] +> For more information on MRTK3 packages, see the [package overview page](../../packages/packages-overview.md). + +When you're finished selecting packages, click **Get features**, and then follow the instructions in the Mixed Reality Feature Tool to import the selected packages into your Unity project. + +### 3. Open the Unity project + +Open the Unity project and wait for Unity to finish importing the newly added packages. There may be two pop-up messages in this process: + +1. The first message asks whether you want to enable the new input backend. Select **yes**. +1. The second message asks whether you want to update XR InteractionLayerMask. Select **No Thanks**. + +Unity might restart a few times during this process--wait for it to finish before proceeding. + +### 4. Configure MRTK profile after import + +Once imported, MRTK3 requires a profile to be set for the standalone target platform and each additional target platform. + +1. Navigate to **Edit > Project Settings**. +1. Under **Project Settings**, navigate to **MRTK3** and then switch to the standalone tab. Note that the profile is initially unspecified. +1. Populate the field with the default MRTK profile that ships with the core package. You can type in the keyword "MRTKprofile" in the search bar of the project window; make sure you search in `All`. Alternatively, you can find the profile under `Packages/org.mixedrealitytoolkit.core/Configuration/Default Profiles/MRTKProfile.asset`. + > [!NOTE] + > Not all of the MRTK subsystems are shown in the screenshot below. The MRTK subsystems that you see may be different depending on the MRTK3 packages you've added to your project. + + ![assign the default MRTK profile](../../images/mrtk-profile.png) +1. Switch to the tabs of other build target(s) you want to use (for example, UWP, Android) and check to see if the profile is assigned. If not, repeat the previous step on the current tab. + +### 5. Configure OpenXR-related settings + +Once imported, MRTK3 requires some configuration on OpenXR if you're targeting an XR device such as HoloLens 2 or Quest. + +> [!NOTE] +> The following instructions apply to HoloLens 2 or WMR headsets. If you're targeting Quest, refer to the instructions on the [Quest deployment page](../../test-and-deploy/quest-deployment.md#deployment-prerequisites). + +1. Navigate to **Edit > Project Settings**. + +1. Under **Project Settings**, navigate to **XR Plug-in Management** and enable **OpenXR** under both the Standalone and UWP tabs. Under each tab, ensure that **Initialize XR on Startup** is selected and that the **Windows Mixed Reality feature group under Standalone** and the **Microsoft HoloLens feature group under UWP** are enabled. + + > [!NOTE] + > A yellow warning icon may appear after checking the **OpenXR** option. Click that icon to open the **OpenXR Project Validation** tool. Click **Fix all** and ignore the interaction profile issue that can't be auto-fixed. The profiles will be added in the step below. + + For standalone: + + [![Standalone XR Plug-in Management window](../../images/standalone-xr-plug-in-management.png)](../../images/standalone-xr-plug-in-management.png) + + For UWP: + + [![UWP XR Plug-in Management window](../../images/uwp-xr-plug-in-management.png)](../../images/uwp-xr-plug-in-management.png) + +1. Under **Project Settings**, navigate to **XR Plug-in Management > OpenXR > Interaction Profiles** and add the following three profiles for UWP and Standalone: + + - **Eye Gaze Interaction Profile** + - **Microsoft Hand Interaction Profile** + - **Microsoft Motion Controller Profile** + + > [!NOTE] + > You might need to use the **OpenXR Project Validation** tool to eliminate the yellow triangle. Some of the warnings may be resolved manually:
1. Under **Project Settings**, navigate to **Player > Resolution and Presentation**. Ensure that **Run in Background** is unchecked.
2. For UWP, under **Player > Publishing Settings > Capabilities**, ensure that **WebCam**, **Microphone**, **SpatialPerception**, and **GazeInput** are checked if these features are needed by the application. For more information about Window's App Capabilities see [App capability declarations](https://learn.microsoft.com/windows/uwp/packaging/app-capability-declarations). + + For standalone: + + [![Standalone OpenXR](../../images/standalone-openxr.png)](../../images/standalone-openxr.png) + + For UWP: + + [![UWP OpenXR](../../images/uwp-openxr.png)](../../images/uwp-openxr.png) + +1. For HoloLens 2, we recommend that you set **Depth Submission Mode** to 16-bit in the settings above. +1. For immersive headsets, you can use 24-bit depth submission. See the [Microsoft development docs for Unity](https://learn.microsoft.com/windows/mixed-reality/develop/unity/recommended-settings-for-unity?tabs=openxr#enable-depth-buffer-sharing) for more info. + +### 6. Congratulations, the project setup is now finished + +Proceed to [creating a new MRTK3 scene](../setting-up/setup-new-scene.md). + +## Next steps + +Once you've finished setting up your Unity project, learn how to [experience your application on a device](../../test-and-deploy/overview.md) diff --git a/docs/mrtk3-overview/getting-started/setting-up/setup-new-scene.md b/docs/mrtk3-overview/getting-started/setting-up/setup-new-scene.md new file mode 100644 index 000000000..6f105c97e --- /dev/null +++ b/docs/mrtk3-overview/getting-started/setting-up/setup-new-scene.md @@ -0,0 +1,16 @@ +# Creating a new scene with MRTK3 + +The following will walk through through creating a AR/VR ready scene using MRTK3. + +1. Create a new Unity scene. +1. Add the **MRTK XR Rig** prefab. +1. Remove the **Main Camera** Game Object because **MRTK XR Rig** already contains a camera. + + ![MRTK XR rig screenshot](../../images/mrtk-xr-rig.png) + +1. Add the MRTK Input Simulator prefab to your scene. + + > [!NOTE] + > This step is optional, but required by in-editor simulations. + + ![MRTK input simulator hierarchy pane](../../images/mrtk-input-simulator.png) diff --git a/docs/mrtk3-overview/getting-started/setting-up/setup-template.md b/docs/mrtk3-overview/getting-started/setting-up/setup-template.md new file mode 100644 index 000000000..8d5abe533 --- /dev/null +++ b/docs/mrtk3-overview/getting-started/setting-up/setup-template.md @@ -0,0 +1,13 @@ +# Starting from the MRTK3 template project + +The easiest way to acquire and try out MRTK3 is to explore our pre-configured project. This project contains references to all of the current MRTK3 packages, and comes pre-configured with the project settings required to deploy to device. Clone the project from [the MRTK3 GitHub repo](https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity). After that, you can simply launch Unity (2021.3.21f1 or newer) on the `MRTKDevTemplate` project under `UnityProjects` and start playing with the sample scenes in the Editor by using remoting or deploying them to devices. + +If you work with Git using the command line, you can clone the repo while specifying the `main` branch: + + `git clone --branch main https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity.git` + +For information on the sample scenes included in the preview, see [Exploring MRTK3 Sample Scenes](../exploring-features/mrtk3-sample-scenes.md). + +## Next steps + +Once you've finished setting up your Unity project, learn how to [experience your application on a device](../overview.md) diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-04-GettingStartedMRTK3-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-04-GettingStartedMRTK3-1920x1080_w800.png new file mode 100644 index 000000000..1ff3f16f6 Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-04-GettingStartedMRTK3-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-07-MRTK3BuildingBlocks-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-07-MRTK3BuildingBlocks-1920x1080_w800.png new file mode 100644 index 000000000..a1b38f993 Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-07-MRTK3BuildingBlocks-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-10-BuildingRichUI-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-10-BuildingRichUI-1920x1080_w800.png new file mode 100644 index 000000000..ec97a0004 Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-10-BuildingRichUI-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-12-WorkingWithDynamicData-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-12-WorkingWithDynamicData-1920x1080_w800.png new file mode 100644 index 000000000..6b204b393 Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-12-WorkingWithDynamicData-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-15-HashOpenDeploy-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-15-HashOpenDeploy-1920x1080_w800.png new file mode 100644 index 000000000..d618f064c Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-15-HashOpenDeploy-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRDevDays/MRDD-June8-04-IntroducingMRTK3-1920x1080_w800.png b/docs/mrtk3-overview/images/MRDevDays/MRDD-June8-04-IntroducingMRTK3-1920x1080_w800.png new file mode 100644 index 000000000..c503b32a7 Binary files /dev/null and b/docs/mrtk3-overview/images/MRDevDays/MRDD-June8-04-IntroducingMRTK3-1920x1080_w800.png differ diff --git a/docs/mrtk3-overview/images/MRTK3_Packages.png b/docs/mrtk3-overview/images/MRTK3_Packages.png new file mode 100644 index 000000000..4873ecdd1 Binary files /dev/null and b/docs/mrtk3-overview/images/MRTK3_Packages.png differ diff --git a/docs/mrtk3-overview/images/MRTK_UX_v3_Cover.png b/docs/mrtk3-overview/images/MRTK_UX_v3_Cover.png new file mode 100644 index 000000000..e37862db6 Binary files /dev/null and b/docs/mrtk3-overview/images/MRTK_UX_v3_Cover.png differ diff --git a/docs/mrtk3-overview/images/MRTK_v3_Architecture.png b/docs/mrtk3-overview/images/MRTK_v3_Architecture.png new file mode 100644 index 000000000..2667fb8fb Binary files /dev/null and b/docs/mrtk3-overview/images/MRTK_v3_Architecture.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_Solver_Main.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_Solver_Main.png new file mode 100644 index 000000000..c9d18c397 Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_Solver_Main.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_BoundsControl.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_BoundsControl.png new file mode 100644 index 000000000..763588da2 Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_BoundsControl.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Button.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Button.png new file mode 100644 index 000000000..7be235604 Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Button.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Dialog.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Dialog.png new file mode 100644 index 000000000..fbed3faab Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Dialog.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_HandMenu.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_HandMenu.png new file mode 100644 index 000000000..75d13057c Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_HandMenu.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_NearMenu.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_NearMenu.png new file mode 100644 index 000000000..539ff2acc Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_NearMenu.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_ObjectManipulator.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_ObjectManipulator.png new file mode 100644 index 000000000..77bf23a62 Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_ObjectManipulator.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slate.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slate.png new file mode 100644 index 000000000..be6fedf78 Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slate.png differ diff --git a/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slider.png b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slider.png new file mode 100644 index 000000000..d2928521d Binary files /dev/null and b/docs/mrtk3-overview/images/UXBuildingBlocks/MRTK_UX_v3_Slider.png differ diff --git a/docs/mrtk3-overview/images/hand-interaction-examples.png b/docs/mrtk3-overview/images/hand-interaction-examples.png new file mode 100644 index 000000000..ca1dbf035 Binary files /dev/null and b/docs/mrtk3-overview/images/hand-interaction-examples.png differ diff --git a/docs/mrtk3-overview/images/mrtk-input-simulator.png b/docs/mrtk3-overview/images/mrtk-input-simulator.png new file mode 100644 index 000000000..2848e512e Binary files /dev/null and b/docs/mrtk3-overview/images/mrtk-input-simulator.png differ diff --git a/docs/mrtk3-overview/images/mrtk-profile.png b/docs/mrtk3-overview/images/mrtk-profile.png new file mode 100644 index 000000000..c8befd7fb Binary files /dev/null and b/docs/mrtk3-overview/images/mrtk-profile.png differ diff --git a/docs/mrtk3-overview/images/mrtk-xr-rig.png b/docs/mrtk3-overview/images/mrtk-xr-rig.png new file mode 100644 index 000000000..05bd1c797 Binary files /dev/null and b/docs/mrtk3-overview/images/mrtk-xr-rig.png differ diff --git a/docs/mrtk3-overview/images/mrtk3-featuretool-setup-packages.png b/docs/mrtk3-overview/images/mrtk3-featuretool-setup-packages.png new file mode 100644 index 000000000..be01b3b55 Binary files /dev/null and b/docs/mrtk3-overview/images/mrtk3-featuretool-setup-packages.png differ diff --git a/docs/mrtk3-overview/images/oculus-openxr.png b/docs/mrtk3-overview/images/oculus-openxr.png new file mode 100644 index 000000000..1c86d6a4c Binary files /dev/null and b/docs/mrtk3-overview/images/oculus-openxr.png differ diff --git a/docs/mrtk3-overview/images/oculus-xr-plug-in-management.png b/docs/mrtk3-overview/images/oculus-xr-plug-in-management.png new file mode 100644 index 000000000..67f0b05e5 Binary files /dev/null and b/docs/mrtk3-overview/images/oculus-xr-plug-in-management.png differ diff --git a/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-AndroidModule.png b/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-AndroidModule.png new file mode 100644 index 000000000..91f788323 Binary files /dev/null and b/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-AndroidModule.png differ diff --git a/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-UWPModule.png b/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-UWPModule.png new file mode 100644 index 000000000..c26d91411 Binary files /dev/null and b/docs/mrtk3-overview/images/setting-up/MRTK-Development-Setup-UWPModule.png differ diff --git a/docs/mrtk3-overview/images/standalone-openxr.png b/docs/mrtk3-overview/images/standalone-openxr.png new file mode 100644 index 000000000..770f8ba32 Binary files /dev/null and b/docs/mrtk3-overview/images/standalone-openxr.png differ diff --git a/docs/mrtk3-overview/images/standalone-xr-plug-in-management.png b/docs/mrtk3-overview/images/standalone-xr-plug-in-management.png new file mode 100644 index 000000000..6c3b2f784 Binary files /dev/null and b/docs/mrtk3-overview/images/standalone-xr-plug-in-management.png differ diff --git a/docs/mrtk3-overview/images/uwp-openxr.png b/docs/mrtk3-overview/images/uwp-openxr.png new file mode 100644 index 000000000..ba76cc8f3 Binary files /dev/null and b/docs/mrtk3-overview/images/uwp-openxr.png differ diff --git a/docs/mrtk3-overview/images/uwp-xr-plug-in-management.png b/docs/mrtk3-overview/images/uwp-xr-plug-in-management.png new file mode 100644 index 000000000..dda84e20c Binary files /dev/null and b/docs/mrtk3-overview/images/uwp-xr-plug-in-management.png differ diff --git a/docs/mrtk3-overview/index.md b/docs/mrtk3-overview/index.md new file mode 100644 index 000000000..fa8838391 --- /dev/null +++ b/docs/mrtk3-overview/index.md @@ -0,0 +1,106 @@ +# Mixed Reality Toolkit 3 + +![MRTK3 banner](images/MRTK_UX_v3_Cover.png) + +Mixed Reality Toolkit 3 is an open-source project to accelerate cross-platform mixed reality development in Unity. MRTK3 is built on top of Unity's XR Management system and XR Interaction Toolkit. Here are some of its functions: + +* Provides the **cross-platform input system and building blocks for spatial interactions and UI**. +* Enables **rapid prototyping** via in-editor simulation that allows you to see changes immediately. +* Operates as an **extensible framework** that allows developers the ability to swap out core components. +* **Supports a wide range of platforms:** + +| Platform | Supported Devices | +|---|---| +| OpenXR devices | [Android XR](https://www.android.com/xr/)
Microsoft HoloLens 2
Meta Quest
Magic Leap 2
Lenovo ThinkReality A3 (with [Qualcomm Snapdragon Spaces](https://docs.spaces.qualcomm.com/unity/samples/preview/mrtk3-setup-guide))
Windows Mixed Reality (experimental)
SteamVR (experimental) | +| Windows | Traditional flat-screen desktop (experimental) | +| And more! | Coming soon! | + +## Welcome to MRTK3 + +[Mixed Reality Toolkit Organization](https://github.com/MixedRealityToolkit) currently maintains MRTK3, and released MRTK3 for general availability (GA). + +### Key improvements + +#### Architecture + +* Built on Unity XR Interaction Toolkit and the Unity Input System. +* OpenXR focused. +* Open-ended and flexible interaction paradigms. + +#### Performance + +* Rewrote and redesigned most features and systems, from UX to input to subsystems. +* Zero per-frame memory allocation. +* Tuned for maximum performance on mixed reality devices and other resource-constrained mobile platforms. + +#### User Interface + +* New interaction models (gaze-pinch indirect manipulation). +* Updated Mixed Reality Design Language. +* Unity Canvas + 3D UX: production-grade dynamic auto-layout. +* Unified 2D & 3D input for gamepad, mouse, and accessibility support. +* Data binding for branding, theming, dynamic data, and complex lists. + +#### Accessibility (Early Preview) + +* Low vision aids. +* Input assistance. + +#### Long Term Support + +* Minimum requirements: OpenXR, Unity 2022.3 LTS, Unity’s XR Interaction Toolkit 3.0. + +## Versioning + +In previous versions of MRTK (HoloToolkit and MRTK v2), all packages were released as a complete set, marked with the same version number (ex: 2.8.0). Starting with MRTK3, each package is individually versioned, following the [Semantic Versioning 2.0.0 specification](https://semver.org/spec/v2.0.0.html). + +> [!NOTE] +> The '3' in MRTK3 is not a version number. It's an indicator of the generation of the underlying architecture, with HoloToolkit being generation one and MRTK v2.x being generation two. + +Individual versioning will enable faster servicing while providing improved developer understanding of the magnitude of changes and reducing the number of packages needing to be updated to acquire the desired fix(es). + +For example, if a non-breaking new feature is added to the UX core package that contains the logic for user interface behavior, the minor version number will increase (from 3.0.x to 3.1.0). Since the change is non-breaking, the UX components package, which depends upon UX core, is not required to be updated. + +As a result of this change, there isn't a unified MRTK3 product version. + +To help identify specific packages and their versions, MRTK3 provides an "about" dialog that lists the relevant packages included in the project. To access this dialog, in Unity on the menu bar, select `Mixed Reality` > `MRTK3` > `About MRTK`. + +## Branch Status + +[Mixed Reality Toolkit Organization](https://github.com/MixedRealityToolkit) currently maintains and updates MRTK3. We appreciate your feedback, and you can open bugs and feature request at the [Mixed Reality Toolkit for Unity](https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity) GitHub project. + +### Early preview packages + +Some parts of MRTK3 are at earlier stages of the development process than others. Early preview packages can be identified in the Mixed Reality Feature Tool and Unity Package Manager by the `Early Preview` designation in their names. + +As of November 2025, the following components are considered to be in early preview. + +| Name | Package Name | +| --- | --- | +| Accessibility | org.mixedrealitytoolkit.accessibility | +| Data Binding and Theming | org.mixedrealitytoolkit.data | + +We encourage you to provide any and all feedback to help shape the final form of these early preview features. + +## UX building blocks + +| | | | +|---|---|---| +| ![Button](images/UXBuildingBlocks/MRTK_UX_v3_Button.png)
**Button**
A volumetric button optimized for a wide range of input modalities, including poking, gaze-pinch, ray interactions, mouse click, and gamepad. | ![Bounds Control](images/UXBuildingBlocks/MRTK_UX_v3_BoundsControl.png)
**Bounds Control**
Intent feedback and precision manipulation affordances. | ![Object Manipulator](images/UXBuildingBlocks/MRTK_UX_v3_ObjectManipulator.png)
**Object Manipulator**
Move and manipulate objects with one or two hands with a wide variety of input modalities. | +| ![Hand Menu](images/UXBuildingBlocks/MRTK_UX_v3_HandMenu.png)
**Hand Menu**
A hand-anchored collection of UX controls for easy access to quick actions. | ![Near Menu](images/UXBuildingBlocks/MRTK_UX_v3_NearMenu.png)
**Near Menu**
Collection of UX controls that can be manipulated, pinned, and set to follow the user. | ![Slider](images/UXBuildingBlocks/MRTK_UX_v3_Slider.png)
**Slider**
Adjust a value along a one-dimensional axis. | +| ![Solver](images/UXBuildingBlocks/MRTK_Solver_Main.png)
**Solver**
Various object positioning behaviors such as tag-along, body-lock, constant view size and surface magnetism. | ![Dialog](images/UXBuildingBlocks/MRTK_UX_v3_Dialog.png)
**Dialog**
Prompt for user action. | ![Slate](images/UXBuildingBlocks/MRTK_UX_v3_Slate.png)
**Slate**
A flat panel for displaying large-format interfaces and content. | + +### Figma Toolkit for MRTK3 Preview + +The [prerelease of Figma Toolkit for MRTK3](https://www.figma.com/community/file/1145959192595816999) includes UI components based on Microsoft's new Mixed Reality Design Language introduced in MRTK3. You can use the 2D representations of the components in the design process for creating UI layouts and storyboards. + +## Session videos from Microsoft Mixed Reality Dev Days 2022 + +| | | | +|---|---|---| +| [![Introducing MRTK3](images/MRDevDays/MRDD-June8-04-IntroducingMRTK3-1920x1080_w800.png)](https://youtu.be/fjQFkeF-ZOM?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[Introducing MRTK3 – Shaping the future of the MR Developer Experience](https://youtu.be/fjQFkeF-ZOM?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | [![Getting started with your first MRTK3 project](images/MRDevDays/MRDD-04-GettingStartedMRTK3-1920x1080_w800.png)](https://youtu.be/aVnwIq4VUcY?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[Getting started with your first MRTK3 project](https://youtu.be/aVnwIq4VUcY?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | [![MRTK3 Interaction building blocks](images/MRDevDays/MRDD-07-MRTK3BuildingBlocks-1920x1080_w800.png)](https://youtu.be/naVziEJ-yDg?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[MRTK3 Interaction building blocks](https://youtu.be/naVziEJ-yDg?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | +| [![Building Rich UI for MR in MRTK3](images/MRDevDays/MRDD-10-BuildingRichUI-1920x1080_w800.png)](https://youtu.be/g2HF5HMy-2c?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[Building Rich UI for MR in MRTK3](https://youtu.be/g2HF5HMy-2c?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | [![Working with Dynamic Data and Theming in MRTK3](images/MRDevDays/MRDD-12-WorkingWithDynamicData-1920x1080_w800.png)](https://youtu.be/IiTpZ2ojyno?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[Working with Dynamic Data and Theming in MRTK3](https://youtu.be/IiTpZ2ojyno?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | [![#Open - Deploy Everywhere with OpenXR and MRTK3](images/MRDevDays/MRDD-15-HashOpenDeploy-1920x1080_w800.png)](https://youtu.be/LI6lyW9TG9o?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)
**[#Open - Deploy Everywhere with OpenXR and MRTK3](https://youtu.be/LI6lyW9TG9o?list=PLlrxD0HtieHhkPlibqfQf1pGvM0vLNPpL)** | + +## Roadmap + +[Mixed Reality Toolkit Organization](https://github.com/MixedRealityToolkit) will announce future releases. diff --git a/docs/mrtk3-overview/known-issues.md b/docs/mrtk3-overview/known-issues.md new file mode 100644 index 000000000..d326b9ac2 --- /dev/null +++ b/docs/mrtk3-overview/known-issues.md @@ -0,0 +1,3 @@ +# Known issues — MRTK3 + +For known issues in MRTK3, refer to [the issues page](https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity/issues?q=is%3Aopen+is%3Aissue) of our GitHub repo. diff --git a/docs/mrtk3-overview/packages/packages-overview.md b/docs/mrtk3-overview/packages/packages-overview.md new file mode 100644 index 000000000..be8f738fc --- /dev/null +++ b/docs/mrtk3-overview/packages/packages-overview.md @@ -0,0 +1,59 @@ +# Using MRTK3 packages + +Microsoft MRTK3 is distributed as a set of packages that are imported into Unity using the Mixed Reality Feature Tool for Unity and the Unity Package Manager (UPM). These packages enable developers to customize the MRTK within their projects. + +## Dependencies + +Some MRTK3 packages require additional packages, provided by Microsoft and/or Unity, in order to correctly function. Some of these packages are optional and will enable additional functionality. + +The following diagram illustrates the relationship between MRTK packages and some of the Unity dependencies. + +![MRTK3 Package Graph](../images/MRTK3_Packages.png) + +> [!NOTE] +> When importing packages using the Mixed Reality Feature Tool, dependency management is automatically performed. + +The following table describes the Mixed Reality Toolkit package dependencies. + +| Display name | Package name | Description | Required | Optional | +| ----------- | ----------- | --------- | -------- | ---------- | +| MRTK Core Definitions | org.mixedrealitytoolkit.core | Shared definitions, utilities and components. |
  • com.unity.xr.interaction.toolkit
  • com.unity.xr.management
| | +| MRTK Accessibility | org.mixedrealitytoolkit.accessibility | Definitions, features and subsystem for building accessible mixed reality experiences. |
  • org.mixedrealitytoolkit.core
  • org.mixedrealitytoolkit.graphicstools.unity
  • com.unity.textmeshpro
| | +| MRTK Audio Effects | org.mixedrealitytoolkit.audio | Effects and features that enhance the audio in mixed reality experiences. |
  • org.mixedrealitytoolkit.core
| | +| MRTK Data Binding and Theming | org.mixedrealitytoolkit.data | Support for data binding and UI element theming. |
  • org.mixedrealitytoolkit.core
  • com.unity.nuget.newtonsoft-json
  • com.unity.textmeshpro
| | +| MRTK Diagnostics | org.mixedrealitytoolkit.diagnostics | Diagnostics and performance monitoring subsystems and tools. |
  • org.mixedrealitytoolkit.core
  • com.unity.xr.management
| | +| MRTK Environment | org.mixedrealitytoolkit.environment | Environmental features and subsystems, such as Spatial Awareness and boundaries. |
  • org.mixedrealitytoolkit.core
  • com.unity.xr.management
| | +| MRTK Extended Assets | org.mixedrealitytoolkit.extendedassets | Additional audio, font, texture and other assets for use in applications. |
  • org.mixedrealitytoolkit.standardassets
  • org.mixedrealitytoolkit.graphicstools.unity
| | +| MRTK Graphics Tools | org.mixedrealitytoolkit.graphicstools.unity | Shaders, textures, materials and models. | |
  • com.unity.render-pipelines.universal
| +| MRTK Input | org.mixedrealitytoolkit.input | Input components including support for articulated hands, offline speech recognition and in-editor input simulation. |
  • org.mixedrealitytoolkit.core
  • org.mixedrealitytoolkit.graphicstools.unity
  • com.unity.xr.interaction.toolkit
  • com.unity.inputsystem
  • com.unity.xr.management
  • com.unity.xr.openxr
  • com.unity.xr.arfoundation
| | +| MRTK Spatial Manipulation | org.mixedrealitytoolkit.spatialmanipulation | Spatial positioning and manipulation components and utilities, including solvers. |
  • org.mixedrealitytoolkit.core
  • org.mixedrealitytoolkit.uxcore
  • com.unity.inputsystem
  • com.unity.xr.interaction.toolkit
|
  • org.mixedrealitytoolkit.input
| +| MRTK Standard Assets | org.mixedrealitytoolkit.standardassets | Standard assets, including materials and textures, for use by applications. |
  • org.mixedrealitytoolkit.graphicstools.unity
| | +| MRTK Tools | org.mixedrealitytoolkit.tools | Collection of Unity Editor tools used to extend and optimize MRTK3 applications. |
  • org.mixedrealitytoolkit.core
| | +| MRTK UX Components | org.mixedrealitytoolkit.uxcomponents | MRTK UX component library, containing prefabs, visuals, pre-made controls, and everything to get started building 3D user interfaces for mixed reality. |
  • org.mixedrealitytoolkit.uxcore
  • org.mixedrealitytoolkit.spatialmanipulation
  • com.microsoft.standardassets
| | +| MRTK UX Components (Non-Canvas) | org.mixedrealitytoolkit.uxcomponents.noncanvas | MRTK non-Canvas UX component library, for building 3D UX without Canvas layout. For most production-grade UI, we recommend the dynamic hybrid Canvas-based UX systems, located in org.mixedrealitytoolkit.uxcomponents. However, in some circumstances, static/non-Canvas UI may offer improved performance and batching, and may be desirable in resource-constrained scenarios. |
  • org.mixedrealitytoolkit.uxcore
  • org.mixedrealitytoolkit.spatialmanipulation
  • com.microsoft.standardassets
| | +| MRTK UX Core | org.mixedrealitytoolkit.uxcore | Core interaction and visualization scripts for building MR user interface components.\n\nNote: this is intended to be consumed in order to build UX libraries. To build MR interfaces with a pre-existing library of components, see org.mixedrealitytoolkit.uxcomponents. |
  • org.mixedrealitytoolkit.core
  • org.mixedrealitytoolkit.graphicstools.unity
  • com.unity.inputsystem
  • com.unity.textmeshpro
  • com.unity.xr.interaction.toolkit
|
  • org.mixedrealitytoolkit.data
| +| MRTK Windows Speech | org.mixedrealitytoolkit.windowsspeech | Speech subsystem implementation for native Windows speech APIs. Allows for the use of native Windows speech recognition to fire events and drive XRI interactions. |
  • org.mixedrealitytoolkit.core
| | + +## Running package tests + +Some MRTK packages contain tests used to validate the included components. In some cases, these tests require additional MRTK packages not asserted as dependencies. + +> [!NOTE] +> When importing packages into Unity, test assemblies aren't compiled by default. To enable compilation of tests, please use the `testables` element of the project's `manifest.json` file. + +In order to place minimal overhead on applications importing the Mixed Reality Toolkit, dependencies are asserted only for runtime requirements. The following table describes the additional packages required to enable compiling and running the included test assemblies. + +| Display name | Package name | Test requirements | +| ------------ | ------------ | ----------------- | +| MRTK Core Definitions | org.mixedrealitytoolkit.core | | +| MRTK Accessibility | org.mixedrealitytoolkit.accessibility | | +| MRTK Data Binding and Theming | org.mixedrealitytoolkit.data | | +| MRTK Diagnostics | org.mixedrealitytoolkit.diagnostics | | +| MRTK Environment | org.mixedrealitytoolkit.environment | | +| MRTK Extended Assets | org.mixedrealitytoolkit.extendedassets | | +| MRTK Input | org.mixedrealitytoolkit.input | | +| MRTK Spatial Manipulation | org.mixedrealitytoolkit.spatialmanipulation |
  • org.mixedrealitytoolkit.input
| +| MRTK Standard Assets | org.mixedrealitytoolkit.standardassets | | +| MRTK UX Components | org.mixedrealitytoolkit.uxcomponents |
  • org.mixedrealitytoolkit.input
| +| MRTK UX Core | org.mixedrealitytoolkit.uxcore |
  • org.mixedrealitytoolkit.input
| +| MRTK Windows Speech | org.mixedrealitytoolkit.windowsspeech | | diff --git a/docs/mrtk3-overview/release-notes.md b/docs/mrtk3-overview/release-notes.md new file mode 100644 index 000000000..156fe5b98 --- /dev/null +++ b/docs/mrtk3-overview/release-notes.md @@ -0,0 +1,3 @@ +# Release Notes — MRTK3 + +MRTK3 is now shipping as [a set of individually versioned packages](index.md#versioning). You can find release notes on the corresponding page for each package. diff --git a/docs/mrtk3-overview/test-and-deploy/hololens2-deployment.md b/docs/mrtk3-overview/test-and-deploy/hololens2-deployment.md new file mode 100644 index 000000000..c1b321a54 --- /dev/null +++ b/docs/mrtk3-overview/test-and-deploy/hololens2-deployment.md @@ -0,0 +1,19 @@ +# Deploy an MRTK3 project to HoloLens 2 + +This page describes how to deploy your Unity Project with MRTK3 onto a HoloLens 2. + +> [!NOTE] +> We strongly recommend using [Holographic remoting](streaming.md) for rapid iteration and testing on HoloLens 2, which allows for instant testing on the device without the need for compile + deploy. + +## Deployment Pre-requisites + +- Add MRTK to your project and ensure your [project settings](../getting-started/setting-up/setup-new-project.md#5-configure-openxr-related-settings) are configured correctly to use the OpenXR pipeline and MRTK's feature set. **These features are required to deploy your project onto your HoloLens**. + +> [!NOTE] +> If starting from our [template project](../getting-started/setting-up/setup-template.md), these project settings should already be configured for you. + +## Deploying to Device + +1. After you have the project configured, proceed to [Build the Unity Project](/windows/mixed-reality/develop/unity/build-and-deploy-to-hololens#build-the-unity-project). + +1. Once built, you'll need to deploy the project through [Visual Studio](/windows/mixed-reality/develop/advanced-concepts/using-visual-studio?tabs=hl2). diff --git a/docs/mrtk3-overview/test-and-deploy/overview.md b/docs/mrtk3-overview/test-and-deploy/overview.md new file mode 100644 index 000000000..888b49c35 --- /dev/null +++ b/docs/mrtk3-overview/test-and-deploy/overview.md @@ -0,0 +1,22 @@ +# Testing and experiencing overview + +Now that you have a Unity project with MRTK3, there are several means to test and experience your application. + +## Preview your application + +Compiling and deploying your app can take a significant amount of time, so we recommend using the following instant iteration/preview solutions while developing your application. + +- [In-editor input simulation](../../mrtk3-input/packages/input/input-simulation.md) + - Easily preview your app without any XR device attached. Control the user's head, hands, and hand gestures with traditional WASD controls. + +- [Stream to a device](streaming.md) + - These solutions allow you to run the app locally in the Unity editor in Play Mode and stream the experience to your device. All inputs from your device are sent to the PC, where the content is then rendered in a virtual immersive view. We highly recommend this solution for instant iteration and for showcasing prototypes + +## Build and deploy + +The following guides will walk you through building and running your application on a device. + +- [HoloLens2](hololens2-deployment.md) +- [Quest Devices](quest-deployment.md) + +If you've deployed a build to your target device of choice, you can debug the build as it runs on device with [Managed debugging](https://learn.microsoft.com/windows/mixed-reality/develop/unity/managed-debugging-with-unity-il2cpp). diff --git a/docs/mrtk3-overview/test-and-deploy/quest-deployment.md b/docs/mrtk3-overview/test-and-deploy/quest-deployment.md new file mode 100644 index 000000000..7806a8637 --- /dev/null +++ b/docs/mrtk3-overview/test-and-deploy/quest-deployment.md @@ -0,0 +1,59 @@ +# Deploy an MRTK3 project to a Quest device + +This page describes how to deploy your Unity Project with MRTK3 onto a Quest device. + +> [!NOTE] +> We strongly recommend using [Meta Quest Link](https://www.meta.com/quest/) for rapid iteration and testing on Quest Devices, which allows for instant testing on the device without the need for compile + deploy. + +## Deployment Prerequisites + +These steps are based around OpenXR as your runtime (i.e. XR plugin provider) as we don't recommend using OculusXR due to underlying compatibility issues. + +1. Ensure that your project is ready to deploy on the Quest Device by following [these steps](https://developer.oculus.com/documentation/unity/book-unity-gsg/). + +1. Ensure that [developer mode](https://developer.oculus.com/documentation/native/android/mobile-device-setup/#enable-developer-mode) is enabled on your device (you may need to [join a developer organization](https://developer.oculus.com/documentation/native/android/mobile-device-setup/#joining-or-creating-an-organization) first). Installing the Oculus ADB Drivers is optional. + +1. Add MRTK to your project and ensure that your [project settings](../getting-started/setting-up/setup-new-project.md#5-configure-openxr-related-settings) are configured correctly to use the OpenXR pipeline and MRTK's feature set. **These features are required to deploy your project onto your Quest device**. You may ignore project settings instructions regarding the UWP platform. + +> [!NOTE] +> If starting from our [template project](../getting-started/setting-up/setup-template.md), these project settings should already be configured for you. + +1. Navigate to **File > Build Settings**. + +1. Under **Platform**, select **Android**. Switch the platform to **Android**, and wait for the operation to finish. + +1. Navigate to **Edit > Project Settings**. + +1. Under **Project Settings**, navigate to **XR Plug-in Management** and enable **OpenXR** under the **Android** tab. Ensure that **Initialize XR on Startup** is selected and that no feature groups are enabled, and wait for the operation to finish. + + ![Quest XR Plug-in Management window](../images/oculus-xr-plug-in-management.png) + +1. Under **Project Settings**, navigate to **XR Plug-in Management > OpenXR > Interaction Profiles** and change it so only **Oculus Touch Controller Profile** is present. + +1. Under **Project Settings**, navigate to **XR Plug-in Management > OpenXR > OpenXR Feature Groups** and ensure the following are checked under **All Features**. + + > [!NOTE] + > If you don't see **Hand Tracking** or **Motion Controller Model** under the **OpenXR Feature Groups** panel, please refer to Configure OpenXR-related settings section of [project settings](../getting-started/setting-up/setup-new-project.md#5-configure-openxr-related-settings). + + ![Meta Quest OpenXR](../images/oculus-openxr.png) + +1. Navigate to Project Validation, and fix any Red or yellow error/warning icons might appear during this process. Click the icon to open the **OpenXR Project Validation** tool and select **Fix All** to address the issues. There may be several items to address. + +1. If you plan on using the native keyboard, please refer to the [keyboard documentation](../../mrtk3-input/packages/input/System-keyboard.md#meta-quest-specific-setup) for a required `AndroidManifest.xml` modification. + +## Using platform controller models + +> [!NOTE] +> **Controller models** are stored in a format that is not natively supported by Unity. To use MRTK Controller Visualization on Quest you will need to make sure you have the following packages in your project: +> +> - [glTF importer](https://github.com/atteneder/glTFast) which enables the use of glTF asset files in Unity and allows the use of MRTK Controller Visualization on Quest +> - [KTX Package](https://github.com/atteneder/KtxUnity) which allows users to load KTX or Basis Universal texture files +> +> If you started with the MRTK3 template project, these packages have already been included in the project. + +## Deploying to Device + +> [!NOTE] +> **Do not** follow the Configure Settings instructions on Oculus's documentation page. Their instructions require the use of the Oculus Integration SDK, and uses the Oculus XR plugin rather than the OpenXR plugin. + +After you have the project configured, proceed to [Generate Build](https://developer.oculus.com/documentation/unity/unity-build/#generate-build). We recommend that you select **Build and Run**. This option lets Unity deploy your project directly to your Quest device. diff --git a/docs/mrtk3-overview/test-and-deploy/streaming.md b/docs/mrtk3-overview/test-and-deploy/streaming.md new file mode 100644 index 000000000..447ded149 --- /dev/null +++ b/docs/mrtk3-overview/test-and-deploy/streaming.md @@ -0,0 +1,14 @@ +# Streaming your application to a device + +These options detail how to stream your application to the device of your choice. Streaming your application allows for rapid iteration and development, as the application runs locally on your machine, without the need to compile and install onto your device. It also allows you to use Unity's plethora of in-editor debugging tools and features. + +- **Recommended:** [Holographic remoting (on HoloLens 2)](/windows/mixed-reality/develop/unity/preview-and-debug-your-app) + - For development on HoloLens 2 and related platforms (including other OpenXR targets that include hand tracking), we strongly recommend the use of holographic remoting to accelerate your iteration time. Advanced features like hand tracking, eye tracking, and scene reconstruction are available through remoting, and behave the same as if the app were deployed to a device. +- Play-mode testing with the desktop's active OpenXR runtime + - Many popular PC VR platforms now support OpenXR, including [Windows Mixed Reality](/windows/mixed-reality/develop/native/openxr-getting-started), [SteamVR](https://www.steamvr.com/), and [Oculus Rift on PC](https://developer.oculus.com/documentation/native/pc/dg-openxr/). +- **Experimental**: [Meta Quest Link](https://www.meta.com/quest/) + - Some aspects of hand interactions are still being developed for Quest, and your results may vary. + - Controller interactions should be full parity over Link. + - In **Player Settings** > **OpenXR**, the following must be assigned for the **Windows, Mac, Linux Settings** tab: + - Set **Play Mode OpenXR Runtime** to **Oculus OpenXR**. + - Add the **Oculus Touch Controller Profile** to the list of **Interaction Profiles**.