[macos] "Too many open files" I/O Error

[mitchmindtree]

Originally reported by @cbfry

Hmmm it looks like both errors are related to this "too many open files" I/O error. This error sometimes happens on macos if there is some sort of file handle leak, e.g. if files are being repeatedly open for some reason but never closed.

Does the program crash immediately on startup? Or does it run for a little while? If it does run for a little while, the following would help a lot to identify the cause:

1. While the program is running, open the mac Activity Monitor application.
2. Select the audio_server process so that it is highlighted.
3. Press the small i info icon near the top left of the screen.
4. Switch to the "Open Files and Ports" tab which should show a list of all open files and ports in raw text.
5. Copy all of the text in this box and post it here.

If you could follow these steps I can have a look through that text and see if there are any potential culprit files being repeatedly opened.

[abathman94]

The program will run for a while before crashing, usually around an hour or so.
Thank you for helping solve this error.

I am running a Mac Mini with MacOS V10.14.1
Tex requested below:

/Users/administrator
txt
/Content/FINAL SERVER/audio_server
txt
/Library/Preferences/Logging/.plist-cache.xpTeI0hs
txt
/System/Library/Extensions/AppleHDA.kext/Contents/PlugIns/AppleHDAHALPlugIn.bundle/Contents/MacOS/AppleHDAHALPlugIn
txt
/private/var/db/timezone/tz/2018g.1.0/icutz/icutz44l.dat
txt
/usr/lib/libobjc-trampolines.dylib
txt
/usr/share/icu/icudt62l.dat
txt
/System/Library/Components/CoreAudio.component/Contents/MacOS/CoreAudio
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/VibrantLightAppearance.car
txt
/Library/MessageTracer/SubmitDiagInfo.default.domains.searchtree
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/SystemAppearance.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkAquaAppearance.car
txt
/System/Library/CoreServices/SystemAppearance.bundle/Contents/Resources/DarkAppearance.car
txt
/System/Library/Keyboard Layouts/AppleKeyboardLayouts.bundle/Contents/Resources/AppleKeyboardLayouts-L.dat
txt
/System/Library/Fonts/SFNSText.ttf
txt
/private/var/folders/3q/0lr6kc5j6xd_vmfkqt41whfr0000gn/0/com.apple.LaunchServices-231-v2.csstore
txt
/usr/lib/dyld
0
/dev/ttys000
1
/dev/ttys000
2
/dev/ttys000
3
*:etlservicemgr
4
/Library/MessageTracer/SubmitDiagInfo.default.domains.searchtree crashing 

[mitchmindtree]

@abathman94 thanks for sending that through!

Nothing looks out of the ordinary in that text, however if the bug is only occurring after an hour it is unlikely we would see much if we only check the open files when the program has only just started running. Is there any chance you could grab the the text after the program has been running for about an hour? This should help to find if there is a file handle leak that is accumulating over time.

Also I just wanted to let you know that I've attempted to get this building on the macos machine I have available, however the machine is very old and I'm currently running into servo/gleam#180 and as a result I haven't been able to build it to recreate the error yet. @freesig will have a go at building it on his machine when he gets a chance and we'll go from there. We'll report back when we have more info 👍

[freesig]

So when running the binary on my machine I get a slowly climbing amount of memory. I'm not convinced that it's because of too many file handles because I ran file debuggers and it only ever had a handle open to each wav file. There was no additional file handles open even after an hour.

However the memory usage had gone from 120mb to 1.8gig in one hour. I ran some memory leak dedicators and nothing was found (They basically look for memory that isn't free'd but also not referenced).

Based on the above I think that the too many open files error may actually be caused by not having enough free memory to open another file.
I also don't think it's a matter of memory being created and not free'd because the leak detector would have seen that. I think that the WavReader is continuously reading data into a buffer and growing that buffer each time.

[abathman94]

Thanks for the update, Mitch.
I'll run the audio server with the content till it crashes, then check the open files log and past it back into the chat.
That's an interesting lead on the high memory use, I'm using a mac mini with 4GB of ram so there's not much headroom to play with.
If there is anything else I can do to help, please let me know.
thanks, heaps.

[mitchmindtree]

@abathman94 thanks, but don't worry about checking the open files log again - I've been able to confirm what @freesig is seeing by running the exe you sent through that it's much more likely a memory leak of some sort and probably isn't related to leaked open file handles.

@freesig seeing as the memory slowly climbs over time and the climb is fairly consistently, it does seem like there's a good chance this has something to do with the audio WavReader streams like you mention. The place at which wav files are read and streamed to the audio thread is in this module:

• This is where we spawn the multi-threaded WAV reader.
• There is one "parent" thread that receives requests to play WAV files and forwards the request onto the next available child wav reader thread. It's whole job is basically setting up the stream between a child thread reading the WAV audio and the thread that's requesting the audio.
• There is one child thread for each CPU core on the system, each dedicated to reading audio one buffer at a time from WAVs as necessary to keep the audio thread satisfied.

Hmmm although come to think of it, it could also be one of the channels of communication between threads. E.g. maybe one thread is sending data to another over a channel (MsQueue/SegQueue) but the other thread isn't popping it off the queue either at all or at least not as fast as the sender, and as a result the queue grows infinitely long?

Seeing as the beyond perception installation was not having this memory leak (as far as we know), I bet it's related to some sort of data that the beyond perception exhibition used but the moon installation does not. I wonder if maybe it's the detection or OSC threads sending data, but not being received/processed somewhere seeing as there are no computers in this installation that are doing audio visualisation. 🤔

[mitchmindtree]

Yesssss, that was it!

If I:

1. go into the "Installation Editor" within the audio server GUI
2. select the "MoonRing" installation
3. change number of computers from 0 to 1

The leak immediately stops and the audio_server memory usage hovers at ~50mb pretty consistently.

@abathman94 can you try following those three steps above and let me know if that fixes the bug? If you can confirm it does fix the bug you were getting, then I'll do up an issue so I can remember to make a proper fix for this sometime in the future. For now though hopefully those 3 steps above get things running smoothly again!

[abathman94]

MAGIC!! Nice one find mitch! Ill give it a try right away and let you know how it goes, Again thanks so much. Cheers.

[abathman94]

Can confirm changing the number of computers from 0 to 1 fixed the issue.
Thank you so much for your ongoing support.