memory usage

[ratzycon]

Hey,

trying to run this program, but says it's going OOM, and that paging file is too small

I have 64gb ram and paging file is 96gb (set to automatic)
the test input mp3 file is 26kb

i'm unsure if manually increasing the page file is going to do much

how come this program needs this kind of memory to run?

(venv) A:\repos\audio2vmd\audio2vmd>python audio2vmd.py test.mp3
Traceback (most recent call last):
  File "A:\repos\audio2vmd\audio2vmd\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 62, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The paging file is too small for this operation to complete.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "A:\repos\audio2vmd\audio2vmd\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 62, in <module>
    from tensorflow.python._pywrap_tensorflow_internal import *
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The paging file is too small for this operation to complete.

im also not sure if this tool can use cuda, but it seems to want to run on cuda 11, which needs some manual setup with PATHS if cuda 12 is the default on the system

I have multiple cuda installed and it can be set for the shell session with:

set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
set PATH=%CUDA_PATH%\bin;%CUDA_PATH%\libnvvp;%PATH%

it no longer complains it can't find cuda v11, but it still fails due to page file, so not sure if it changes anything

[ratzycon]

yay, I got one output to work, after setting the page file manually to minimum 128gb and max 192gb xD
the output looks fine, nice work :)

however i can't run it again after

it only worked once after a reboot, the same input afterwards failed with a ton of errors, mainly starting with oom
in task manager i can see it barely uses 8gb ram when it runs, why this memory paging oom error?

when i ran it again it turned my screen black :S
not even gpu driver reset with win+ctrl+alt+b helped, had to do full system restart
something is not right. maybe it was a cuda mismatch or something, who knows
think it only worked when it ran on the cpu tho

a suggestion for the app:

option to not do the voice extraction
sometimes we already have just the voice or did the extraction in another program,
doing to twice really degrades the output

cheers, ty for making this :)
hope it can be fixed

[EliseWindbloom]

Hi ratzycon, thanks for the detailed report and sorry for your trouble. I'm aware of the memory usage issue on some pcs (I've discovered and talked about a little on a different forum). Despite the CUDA error messages, it shouldn't actually need CUDA, as it seems to run 100% on the cpu and runs relatively fast on a lower-end pc.

Just for reference, i have 32GB RAM, and the paging file set to 32GB as well, and the app runs without issue on my lower-end pc, but I've notice others are having memory issues(even with higher pc specs) that are keeping the app from outputting vmds.
I've tried numerous times before but wasn't able to majorly fix the memory issues. However now it seems the main cause of the memory issues is tied directly to the usage of spleeter/tensorflow in the app.

I don't really have the free time now to research and edit this(as resource optimization is the biggest issue with this app now and I don't know much about how to fix this in python), but I plan to return to github to update this in about a month or so, where i plan to replace spleeter with something far more efficient (and add an option to turn off audio separation), which should significantly improve the apps performance for all users and get rid of the memory problems, in theory.
Once I'm able to (hopefully) get it working properly for other users, i'll try to make a video tutorial as well.

[ratzycon]

Hey Elise :)

good to hear you will look into it at a later stage.
I would have liked to test it out more, but i have a lot of other things running (discord ai chatbots and MMD work), a bit of afraid if it randomly crashes/blackout the system again

I can try run it again at some point and send the full error logs
from what i could skim, unfortunately it looked like tensorflow that fails,
spleeter (if thats the one doing the audio split) does run and outputs results

if it doesn't run cuda at all, maybe it could just be set to go for cpu directly, tensorflow just has cpu fallback

the page file size having an effect is kind of mysterious, haven't seen this behaviour before with tensorflow
since it doesn't work the second time i run it, it could be indicative of something either holding on to memory and doesn't get released, it does seem to be running a very large amount of threads
i'm on a 3950x, with 16 cores + hyperthreads, so 32 cores - could it be something like it tries reserving 8gb for each thread?
that would be 256gb for me

looking at the memory usage in taskman tho, it only builds up 8gb system ram,
i don't even see the committed pagefile size rise when it runs/fails, it just complains about oom

is it using an ai model to do the phenome output btw?
in any case, glad you would look into it, it might be very useful tool to run :)

cheers!

[EliseWindbloom]

Sorry for the late reply and thank you for all the feedback, I'm in the process of doing tests/research now.

From what i can tell, spleeter does indeed use tensorflow directly, and tensorflow is causing the extreme memory usage issues (apparently there are ways that tensorflow's memory usage can really balloon according to Claude AI). But yes it is kind of mysterious why it's only happening on some users pcs (i'm not sure why it's doing this, and i still couldn't get it when asking AI like Claude for help). However, I'm now in the process of swapping out spleeter. And yes, spleeter does use an trained AI model i think.

From my research/test, Open-Unmix looks like it could be the best alternate.
Open-Unmix is way better for our use, as it is apparently way better optimized to use CPU-only and uses pytorch instead of tensorflow(tensorflow apparently tends to use much more resources), and it is faster as well with output quality that at least seems to be on par with what we currently have using spleeter.
I haven't directly compared them yet, but from what i see, openunmix is indeed faster than spleeter, with both using only cpu.

I'm still in the process of things, but i will keep you updated. Thanks again for all the nice detailed feedback, i appreciate it.

[EliseWindbloom]

Hi ratzycon, I've had great results and did a ton of coding stuff, memory problems should hopefully be completely solved now.
audio2vmd updated to version 16! Please let me know if you have any more problems with it's memory or such.

Here are the main new features of audio2vmd v16:

• Optimizations, huge speed improvement and lowering of system requirements
• Replaced spleeter(which uses tensorflow) with openunmix to reduce memory usage/increase cpu optimization
• Huge speed boost, was over twice as fast when testing it (an audio file that took 1min6secs to convert, now takes about 31seconds to convert)
• Significant code optimization to reduce memory usage without messing up the vmd output
• Should no longer require a paging file or a lot of memory to use
• "Separate Vocals" setting added to audio2vmd.py along with GUI and config file:
  • Can be set to "automatic", "always", or "never" to choose how audio2vmd separates vocal files.
  • 'automatic' = detect and separate if needed
  • 'always' = skip detection and always separate all audio files
  • 'never' = skip detection and assume file is already vocals-only).
  • "automatic" is recommended if any of your files have music/background noise, otherwise "never" is recommended for a additional big speed boost.

[ratzycon]

Hey Elise,

That sounds awesome, will try it soon and see how it works :)

Been missing a tool like this, I make some vocaloid songs with Suno AI and then make MMD videos for them, so ability to animate mouth for the vocals would be excellent ^-^

Good to hear you have added various options for the 'separate vocals', because for example suno also has a Stem option to get back only the vocals for the track, so this will probably come in handy (depending on which audio separation does the best job, will have to see)

Nice work!

[EliseWindbloom]

Thanks ratzycon, i hope it works well for you. I've tested it with more files and everything seems to be working properly on my pc.

Nice, i've been playing with Suno AI a little too.
If Suno AI natively has the vocals-only(i mean like, it creates the vocals before any music and then merges them together), then Suna AI would always be better than any AI separator since AI separators basically just make educated guesses (and even if Suna AI vocals-only is separated in post-creation, i imagine it would be a very high quality stem), but openunmix's audio separation notably good too and the vocals seem to sound better and clearer with it than with spleeter. So either way should work well with your audio separation, in theory.

Anyways, thanks alot for using audio2vmd, i'll try to see if i can make a video tutorial for it sometime in the near future too.

[ratzycon]

Suno also use some form of extraction, it's far from perfect,
they might even use something similar to what you use, but i haven't compared the results yet
just call me ratzy, name was taken so had to pick ratzycon on twitter/github xD
I'll get back to you when i tried it for a project, cheers!