Arbitrary load balancing not yet activated

[aliemen]

#479 activated load balancing in OPALX with IPPL's OrthogonalRecursiveBisection algorithm. Even though one can define decomposition axes during field initialization, this is not respected by the ORB algorithm. Meaning, an ORB step for load balancing will always result in a full 3D domain decomposition. Or in other words, something like the following:

FS_SC: Fieldsolver, 
    ...
    PARFFTX = false,
    PARFFTY = false,
    PARFFTZ = true,
    ...

...is not currently possible. Instead we have to set everything to true. This extended decomposition functionality will be available once IPPL-framework/ippl#560 is merged. In order to reactivate arbitrary decomposition axes, we have to pass the axes to the ORB inside PartBunch/LoadBalancer.hpp:

bool repartition(ippl::FieldLayout<Dim>* fl, ippl::UniformCartesian<T, Dim>* mesh) {
    ...
    bool res = orb.binaryRepartition(*R, *fl, false, fl->isParallel()); // instead of "orb.binaryRepartition(*R, *fl, false);"
    ...
}

And then remove the consistency check from FieldSolverCmd::setDomainDecomposition:

void FieldSolverCmd::setDomainDecomposition() {
    domainDecomposition_m[0] = Attributes::getBool(itsAttr[FIELDSOLVER::PARFFTX]);
    domainDecomposition_m[1] = Attributes::getBool(itsAttr[FIELDSOLVER::PARFFTY]);
    domainDecomposition_m[2] = Attributes::getBool(itsAttr[FIELDSOLVER::PARFFTZ]);

    /// \todo At the moment, only 3D domain decomposition is supported. This should be extended to
    /// support 1D and 2D domain decomposition in the future, once the changes in the IPPL ORB are
    /// merged. Having parallel in all dimensions is what's currently supported best.
    if (!(domainDecomposition_m[0] && domainDecomposition_m[1] && domainDecomposition_m[2])) {
        throw OpalException(
                "FieldSolverCmd::setDomainDecomposition",
                "Currently only 3D domain decomposition is supported. Please set PARFFTX, PARFFTY "
                "and PARFFTZ to TRUE. Other decompositions will come soon.");
    }
}