Problems with ODM, presumably with ResetDatabase

[vkost]

Hi,

I'm experiencing occasional issues with my tests. Here's the error message:

RuntimeException: Error running "doctrine:mongodb:schema:create --dm=default": 1: Operation not permitted

It always happen on a random test, so unfortunately I'm not able to create code to replicate the problem.

Most of my tests use ResetDatabase trait. I have a feeling this is connected. My guess is that I'm getting this error randomly when ResetDatabase trait attempts to create a new blank database for the test about to be run. However, I don't see why this would happen, because the same operation is executed over 300 times before that successfully.

I am running MongoDB in a docker container, if that matters. The version is 7, but until recently I've been using MongoDB 4 and I've experienced the same issues.

Symfony is 7.4.13, Foundry is 2.10.1, PHP is 8.5.7. Everything runs in MacOS 26.5.1, with Docker Desktop 4.74.0. The tests are also run in a container.

I'd appreciate it if you could point me in the right direction - I'd be happy to dig more and fix the problem on my end.

Thank you
Vic

[nikophil]

Hey @vkost
Any chance you share a stack trace of the error, please?

Are you running parrallel tests with paratest or something like that?

[vkost]

Hi @nikophil thank you so much for responding to this.

I am indeed using paratest, but I remember this happening without it too. Perhaps it would be fair to say it's a bit more frequent with paratest, but it's really hard to be sure.

Here's what I receive when testing:

RuntimeException: Error running "doctrine:mongodb:schema:create --dm=default": 1: Operation not permitted
Created indexes for all classes
Created search indexes for all classes

/app/vendor/zenstruck/foundry/src/symfony_console.php:27
/app/vendor/zenstruck/foundry/src/Mongo/MongoSchemaResetter.php:44
/app/vendor/zenstruck/foundry/src/Persistence/ResetDatabase/ResetDatabaseManager.php:83
/app/vendor/zenstruck/foundry/src/Test/ResetDatabase.php:121
/app/vendor/zenstruck/foundry/src/Test/ResetDatabase.php:50

It's always very similar, but looking through the history I spotted one with a small difference:

RuntimeException: Error running "doctrine:mongodb:schema:create --dm=default": Created collections for all classes
Index build failed: a115a8ad-4ecf-4d2a-b329-df562b7d9d0c: Collection reftrack_test7.Encounter ( f06b528f-a9b2-4416-9117-f2debb1f38d1 ) :: caused by :: index build on empty collection failed: a115a8ad-4ecf-4d2a-b329-df562b7d9d0c :: caused by :: 1: Operation not permitted
Created search indexes for all classes
/app/vendor/zenstruck/foundry/src/symfony_console.php:27
/app/vendor/zenstruck/foundry/src/Mongo/MongoSchemaResetter.php:44
/app/vendor/zenstruck/foundry/src/Persistence/ResetDatabase/ResetDatabaseManager.php:83
/app/vendor/zenstruck/foundry/src/Test/ResetDatabase.php:121
/app/vendor/zenstruck/foundry/src/Test/ResetDatabase.php:50

Looking at '_test7' part, this is definitely a paratest run.

To what I read, Mongo comes back with 'Operation not permitted' when it runs into OS file permissions issue, but it may not be the only reason, of course. Just the one that pops out first on an internet search. Also, file permission problems would probably not pop up randomly during testing with about 450 test cases.

I'm more than willing to do any more tests on this - it's very annoying :)

Thank you for your help
Vic

[nikophil]

Looking at \Zenstruck\Foundry\Mongo\MongoSchemaResetter:

    public function resetBeforeEachTest(KernelInterface $kernel): void
    {
        $application = application($kernel);

        foreach ($this->managers as $manager) {
            try {
                runCommand($application, "doctrine:mongodb:schema:drop --dm={$manager}");
            } catch (\Exception) {
            }

            runCommand($application, "doctrine:mongodb:schema:create --dm={$manager}");
        }
    }

This thing is 5 years old, and I don't remember why we're silently catching the error around doctrine:mongodb:schema:drop 🤔 Maybe you could check if a previous error occurs during doctrine:mongodb:schema:drop when your problem occurs?

I also asked Claude about your problem and he it says that maybe some times, the schema:create occurs too quickly, and conflicts with the schema:drop which is not completely commited in Mongo. Its advice would be to add a retry mechanism. Maybe you could decorate the resetter and implement a very simple retry mechanism, just to see how it behaves? If it fixes your problem, I'm wiling to add it in Foundry

[vkost]

Hi @nikophil
I think you're silently catching the error around the drop because Mongo reports an error if you attempt to drop a database that does not exist, IIRC.
I will try checking the previous error and report back. I'll also try decorating the resetter to retry - it's a great idea. I agree it's likely that this is connected to these operations being sent to Mongo server rapidly. 9 out of 10 it works just fine, but...

Thank you for all your suggestions - I'll let you know what happens.

Vic

[nikophil]

I think you're silently catching the error around the drop because Mongo reports an error if you attempt to drop a database that does not exist, IIRC.

yeah that makes sense, we'd need to add a comment which explain this

[nikophil]

hi @vkost

did you have a chance to test the suggestion?

[vkost]

Hi @nikophil
I apologize for the delay, I was pulled into an unexpected new project at work, with a crazy deadline...
I made the modification as you suggested, just a very basic retry mechanism, as follows:

<?php

declare(strict_types=1);

namespace App\TestHelper;

use Symfony\Component\DependencyInjection\Attribute\AsDecorator;
use Symfony\Component\DependencyInjection\Attribute\When;
use Symfony\Component\HttpKernel\KernelInterface;
use Zenstruck\Foundry\Mongo\MongoResetter;

#[When('test')]
#[AsDecorator(MongoResetter::class)]
final readonly class DecorateDatabaseResetter implements MongoResetter
{
    public function __construct(
        private MongoResetter $decorated,
    ) {
    }

    public function resetBeforeEachTest(KernelInterface $kernel): void
    {
        $maxRetries = 3;
        for ($attempt = 1; $attempt <= $maxRetries; ++$attempt) {
            try {
                $this->decorated->resetBeforeEachTest($kernel);
                break;
            } catch (\Exception $e) {
                if ($attempt === $maxRetries) {
                    throw $e;
                }
                usleep(pow(2, $attempt) * 100 * 1000);
            }
        }
    }
}

It seems to do the job, but I would really like to test it some more before I push the PR. The project I'm currently on is not PHP so I'm not using this test suite actively, and the problem was appearing randomly so I have no way to be sure this fixes it.

I would appreciate any comments on the decorator, and if you can spot any flaws.

Thank you for your patience and all your help.
Vic