unserialize(): Error at offset 219243 of 219250 bytes file: /Application/vendor/spatie/fork/src/Task.php line: 114

[remco-pc]

ErrorException

unserialize(): Error at offset 219243 of 219250 bytes
file: /Application/vendor/spatie/fork/src/Task.php
line: 114

ErrorException: unserialize(): Error at offset 219243 of 219250 bytes in /Application/vendor/spatie/fork/src/Task.php:114
Stack trace:
#0 [internal function]: exceptions_error_handler()
#1 /Application/vendor/spatie/fork/src/Task.php(114): unserialize()
#2 /Application/vendor/spatie/fork/src/Fork.php(109): Spatie\Fork\Task->output()
#3 /Application/vendor/spatie/fork/src/Fork.php(85): Spatie\Fork\Fork->finishTask()
#4 /Application/vendor/r3m_io/framework/src/Module/Parallel.php(22): Spatie\Fork\Fork->waitFor()
#5 /Application/vendor/r3m_io/node/src/Trait/Data/NodeList.php(474): R3m\Io\Module\Parallel->execute()

I debugged the code with this snippet:

try {
                $output = unserialize(
                    substr($output, strlen(self::SERIALIZATION_TOKEN))
                );
            } catch (Exception | \ErrorException $exception){
                ddd($output);
            }

And the output showed me that it stopped at

i:1265;O:8:"stdClass":4:{s:4:"uuid";s:36:"605aaac8-12d2-4c56-9522-8fd8ff2f5db8";s:6:"#class";s:18:"Account.Permission";s:4:"name";s:15:"permission:4265";s:6:"#index";i:3765;}i:1266;O:8:"stdClas"

So the stdClas gets cut off, missing data at the end, i have this with threads=4, if i run threads=8 i don't run in this error, is there a max size of data which can be serialized ?

[redburn]

See #22

How much data are you serializing? There's a limit to the payload that can be returned by a worker. As such, Fork is not well-suited for large payloads, at least not without workarounds (e.g. an intermediary cache). The no. of threads shouldn't affect this.

[remco-pc]

This is only 220 kb, i know there was also a setting in debian, which i have set to handle a lot of forks fitting within memory, it might be that setting is too low now. I have also started making forks myself where i sometimes can return a large chunk of data or where i simply write to ramdisk with json files. And then read them in the main thread. Its the serialize deserialize method which fails, not the pipe, it might also be a wrong php setting, but i cannot remember it

[redburn]

Not exactly sure where the "219250 bytes" in the error comes from, but the default buffer size for each fork is 1024 kb. You can manually adjust it here if you dare, but I've personally opted for a similar approach to yours: if the serialized data exceeds 1024 kb, I store it in a cache and then read that back from the parent process. Works like a charm.

[SteenSchutt]

I just bumped into this as well. I think it would be great to at least mention this under the "Returning data" part of the readme. Don't mind the limitation, but I didn't expect there to be one. Now I have to start my run over again, so I would have loved a little warning :)

[n-hor]

I tried to solve this case in my package by adding a packer that sends content in parts regardless of the buffer size https://github.com/n-hor/pcntl-parallel