change optimizer hyperparameters to double

[marcelluethi]

In DimWit we have the optimizer hyper-parameters, such as learning rate as type Tensor0[Float32].
However, as hyper-parameters are not optimized, they don't need to be differentiable. This PR therefore proposes to replace the Tensor0 type with a simple Double, which is the native floating point type in Scala and hence more ergonomic than, Float

[benikm91]

We had this in an earlier version. The switch to Tensor0 was chosen to communicate the precision to the user. I am strictly against a Double (Float64) resulting (implicitly) in a Float32 implementation. I understand this would allow for the more natural GD(0.1) instead of GD(0.1f). However, note that val t = Tensor(0.1) will also result in a Tensor0[Float64].

The logical (next) extension of the original precision motivation would be to implement the algorithms across precision support (V: IsFloating). This would allow Float16 training (which does not work well in practice) or Float64 training (which is overkill in practice). However, I think the library should not remove these options. Explicitly defining the optimization runs in Float32 seems correct to me.

[marcelluethi]

I don't see the argument regarding precision. From a user's perspective, the learning rate or the other hyper-parameters are never part of the computational graph and all computations done with it are internal to the library. The user simply specifies a floating point value as a literal. He/She should explicitly not have to care about the precision. For a floating point literal, Double would feel most natural as it does not require the f. However, the main point of the PR was that the hyper-parameters should not be Tensor, but primitive Scala values. Whether it is Double or Float feels secondary.

[benikm91]

What I imagine would be something like this

// Generalize GD to any precision
case class GradientDescent[V: IsFloating](learningRate: Tensor0[V]) extends GradientOptimizer[V]:

  type State[P, V] = Unit // Stateless optimizer

  def init[Params: TensorTree: FloatTreeFor[V]](params: Params): Unit = ()

  def update[Params: TensorTree: FloatTreeFor[V]](gradients: Grad[Params], params: Params, state: Unit): (Params, Unit) =
    val newParams = params -- gradients.value.scale(learningRate)
    (newParams, ())

// Map Scala precision to DimWit precision
object GradientDescent:
  def apply(learningRate: Float): GradientDescent[Float32] = GradientDescent(Tensor0(learningRate))
  def apply(learningRate: Double): GradientDescent[Float64] = GradientDescent(Tensor0(learningRate))

PR: https://github.com/benikm91/dimwit/pull/new/optimizer-precision

This precision idea is the main reason I would not change the learningRate: Tensor0[V] to learningRate: Float.

BTW, for me, the learning rate is part of the computation graph, so putting it in tensor land seems right to me.

---

EDIT:

Maybe let me elaborate why this is a good idea. It allows for Float16 training (which does not work in practice, but should be possible to express). It allows for Float64 training (not required for DL but again one should be able to express and maybe useful for Physics stuff?).

[marcelluethi]

I see your point with the precision of the computation in the computational graph and why the proposed change would compromise that goal. However, specifying the hyper-parameters as Tensor0[V] raises new questions. Is it correct that the precision of the hyper-parameters determine with which precision the computations are done? Isn't this a bit too implicit and opaque? What happens if the parameters to be optimized are hard-coded as, e.g. Float32 but the learning rate is Float64? Would this lead to a compilation error? In general, which component should determine with which precision the computation is run? Should this maybe be determined by the concrete instances of the parameters (e.g. initial parameters for the optimization)?

If the optimizer determines the precision, we should maybe make it explicit:

class GradientDescentOptimizer[V : VType](precision : VType[V])(lr : Double) 
    val lrComp : VType[V] = Tensor0(lr).asType[V]
    // computation continues with lrComp

[benikm91]

I like this suggestion with VType[V]. This addresses precision more explicitly, improving on my suggestion. Personally, I would still go with;

class GradientDescentOptimizer[V : VType](precision : VType[V])(lr : Tensor0[V])

But I am fine with

class GradientDescentOptimizer[V : VType](precision : VType[V])(lr : Double)

as well.

I can add VType and lr : Double to my PR this evening and also add some tests for different precisions.