Flutter EffectsSDK integration

Tested for flutter-webrtc v0.14.0

Instructions

Clone flutter WebRTC repository

git clone https://github.com/flutter-webrtc/flutter-webrtc.git

Go to the cloned dir.

cd flutter-webrtc

[Optional] Switch to the required version

git switch -d v0.14.0

For other versions, conflicts may occur.

Apply the patch to the flutter-webrtc repo

curl https://raw.githubusercontent.com/EffectsSDK/flutter-webrtc-integration/refs/heads/main/flutter-webrtc-effects-sdk.patch | git apply

You can mannually download the patch and apply it with git apply path/to/patch

Add flutter-webrtc as dependency to your project by path

pubspec.yaml

dependencies:
  flutter_webrtc:
    path: path/to/flutter-webrtc

Android

For android platform additionally you need:

1. Download the Video Effects SDK release for Android
2. Add the SDK as a WebRTC dependency by placing it in: path/to/flutter-webrtc/android/libs/

How to use

1. Add effectsSdkRequired flag to your getUserMedia request

final mediaConstraints = <String, dynamic>{
  'audio': false,
  'video': {
    'mandatory': {
      'minWidth': '640',
      'minHeight': '480',
      'minFrameRate': '30',
    },
    'facingMode': 'user',
    'effectsSdkRequired': true,
    'optional': [],
  }
};
var stream = await navigator.mediaDevices.getUserMedia(mediaConstraints);

2. Call EffectsSDK methods by using WebRTC Helper

var status = await Helper.auth(stream.getVideoTracks().first, 'YOUR_CUSTOMER_ID');
switch (status) {
    case AuthStatus.active:
    await Helper.setEffectsSdkPipelineMode(stream.getVideoTracks().first, PipelineMode.blur);
    await Helper.setEffectsSdkBlurPower(stream.getVideoTracks().first, 0.6);
    case AuthStatus.expired:
    // TODO: Handle this case.
    case AuthStatus.inactive:
    // TODO: Handle this case.
    case AuthStatus.unavailable:
    // TODO: Handle this case.
}

EffectsSDK methods

Check platform specifications:

1. iOS
2. android

Effects SDK Image

Image proxy for iOS/android compatibility.

Can be created from:

1. raw data
2. file
3. encoded data
4. rgb color

Color correction options

4 different methods are supported:

1. Color correction - use ML model for color correction
2. Color grading - made color filter by using reference image color scheme
3. Low light - use ML model for color correction (ust it only in dark places)
4. Preset - predefined cube lut

Helper methods

Name	Summary
auth	Future<rtc.AuthStatus> auth(String customerId, {String? apiUrl}) Auth sdk by using customer ID
localAuth	Future<rtc.AuthStatus> localAuth(String localKey) Auth SDK by using local key
getEffectsSdkPipelineMode	Future<rtc.PipelineMode> getPipelineMode() Return current background processing mode
setEffectsSdkPipelineMode	void setPipelineMode(rtc.PipelineMode mode) Set background processing mode
setEffectsSdkBlurPower	void setBlurPower(double blurPower) Set blur power
setEffectsSdkBackgroundImage	void setBackgroundImage(rtc.EffectsSdkImage data) Set background image
enableEffectsSdkBeautification	void enableBeautification(bool enable) Enable beautification feature
isEffectsSdkBeautificationEnabled	Future isBeautificationEnabled() Return true if beautification enabled
setEffectsSdkBeautificationPower	void setBeautificationPower(double power) Set beautification power
getEffectsSdkZoomLevel	Future getZoomLevel() Get current zoom level
setEffectsSdkZoomLevel	void setZoomLevel(double zoomLevel) Set zoom level
enableEffectsSdkSharpening	void enableSharpening(bool enable) Enable sharpening mode
getEffectsSdkSharpeningStrength	Future getSharpeningStrength() Return current sharpening power
setEffectsSdkSharpeningStrength	void setSharpeningStrength(double strength) Set sharpening power
setEffectsSdkColorCorrectionMode	void setColorCorrectionMode(rtc.ColorCorrectionMode colorCorrectionMode) Set color correction mode
setEffectsSdkColorFilterStrength	void setColorFilterStrength(double strength) Set color filter strength
setEffectsSdkColorGradingReferenceImage	void setColorGradingReferenceImage(rtc.EffectsSdkImage data) Set reference image for color correction

Technical details

Effects SDK included by using custom CameraVideoCapturer instance for Effects SDK camera pipeline( android). You can modify our solution as you need or try another way for integration (for example with custom VideoProcessor). Also you can replace CameraPipeline to lite version of it.

Additional links

1. Platform documentation (iOS, android)
2. Effects SDK site