Real-time transcription

[lucabeetz]

Hey, awesome package!

I wanted to ask how one could use this for on-device realtime description with microphone audio, similar to the objc example from the whisper.cpp package

[cerupcat]

I'd also like to see an SFSpeechRecognizer-like API for easy replacement of SFSpeechRecognizer.

[libratiger]

+1 for this feature

[fakerybakery]

Yes, this would be a great feature.

[fakerybakery]

Right now, I'm setting a timer to start + stop the transcription every 2 seconds, however it's not that accurate because if a word is cut off, then whisper tries to improvise, and the text often has hallucinations.

[jacobjiangwei]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

[cerupcat]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

The Whisper cpp repo has examples of how to implement realtime.

[fakerybakery]

I think that the whisper.cpp library stores some of the previous recording history and uses that to fix the cutoff issue, but I'm not sure.

[jacobjiangwei]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

The Whisper cpp repo has examples of how to implement realtime.

thanks for point that out. Just curious, why it's in Obj-C but not in swift version...

[fakerybakery]

I don't know why, but if someone could port the example to Swift, I would really appreciate that (I'm really bad at Obj-C).

[brytonsf]

I think that the whisper.cpp library stores some of the previous recording history and uses that to fix the cutoff issue, but I'm not sure.

Yep, I believe it does too – see this line (and line 245)

[barkb]

Don't have a great understanding, but to me it looks like whisper.objc is storing the contents of a buffer when it fills up, then calling it's transcribe function against what it just stored, while clearing the buffer and re-enqueuing it. I don't know a ton about AVFAudio, but does anyone know if you could use AVAudioEngine and AVAudioPCMBuffer to create similar functionality? I'm thinking you could call Whisper.transcribe here with the buffer data if you can get that buffer data back from AVAudioEngine. Does anyone know if that would work?

[ldenoue]

@barkb have you ever found a solution to this real-time idea?

[moaljazaery]

+1

[aehlke]

I found this Swift implementation of streaming: https://github.com/leetcode-mafia/cheetah/blob/b7e301c0ae16df5c597b564b2126e10e532871b2/LibWhisper/stream.cpp with a swift file inside a swit project. It's CC0 licensed.

I couldn't tell if it's uses the right config to benefit from the latest Metal/OpenML performance-oriented config and it uses some tool that requires a brew install so I don't know how sandbox friendly it is.

[fakerybakery]

I found this Swift implementation of streaming: leetcode-mafia/cheetah@b7e301c/LibWhisper/stream.cpp with a swift file inside a swit project. It's CC0 licensed.

I couldn't tell if it's uses the right config to benefit from the latest Metal/OpenML performance-oriented config and it uses some tool that requires a brew install so I don't know how sandbox friendly it is.

@aehlke

The linked app is an AI interview ... er assistant? and it listens to your audio and tries to respond with GPT-4 (it doesn't use SwiftWhisper). It uses the sdl12 library, which, according to their website:

... provide low level access to audio, keyboard, mouse, joystick, and graphics hardware via OpenGL and Direct3D ...

I haven't extensively researched this subject, but my interpretation is that this allows the app to listen to your system audio and transcribe it, so you don't have to install external software such as BlackHole. This leads me to believe that the library may not be necessary if the object is to listen from the microphone, which may mean that it can be run on other devices, such as iOS.