restrict finite differences in test_rule

[tpapp]

For functions that are differentiable on a domain D, but undefined outside that domain, testing with finite differences can result in errors.

It would be useful to extend the API of TestUtils.test_rule to allow for this.

But I am unsure what form the generic interface should take, I am opening this to get ideas and then I would happy to make a PR.

[tpapp]

What I thought of: the user can supply a maximum_ε(x, ẋ) function that returns the maximum allowed scale for perturbations in test_[fr]rule_correctness.

[yebai]

That sounds good.

For a slightly bigger scope, I think it is better to do this as part of #581

[willtebbutt]

I like this idea a lot -- while I agree with Hong that that in order to make use of this functionality throughout the existing Mooncake code base, we should probably do #581 (we already have lots of somewhat unintelligible tuples floating around everywhere defining how to test stuff), I think we could actually modify TestUtils.test_rule quite straightforwardly to give you this option just by adding an extra kwarg. Of course, I hope we'll refactor this at some point to just use a TestCase, but this is nice functionality anyway.

The relevant bits of code are here and here . If it's enough to bound the euclidean norm of the perturbation, I think we would just need to:

1. ensure that ẋ is normalised to unit length, and
2. give the user some way to filter out values from ε_list which are greater than a desired bound.

If, rather than bounding the euclidean norm, we wanted to eg. bound the maximum deviation in any particular coordinate, this might be trickier.

If we wanted to do the above, we would need to

1. add a kwarg to test_frule_correctness and test_rrule_correctness which gives the upper bound on the euclidean norm of ε * ẋ
2. also add this kwarg to test_rule, and propagate it through to test_frule_correctness and test_rrule_correctness, and
3. implement the normalisation (using the functions which are guaranteed to work for any tangents, ie.Mooncake._dot and Mooncake._scale rather than dot and *), and filter the ε_list based on the new kwarg.