Smooth reward calculation for nice gradients

[kim-mskw]

Description

The current reward calculation in EnergyLearningStrategy uses a hard threshold (step function) to determine whether a bid is accepted or not. This creates:

• Discontinuous reward signals that make learning difficult
• Sharp gradients ( or honestly no gardients) at the market clearing price boundary

In another study, I replaced a nasty binary with a sigmoid, and it did miracles.
Replace the discrete acceptance logic with a smooth sigmoid-based acceptance probability that provides:

• Continuous, differentiable reward signals
• Gradual transitions around the market clearing price
• More stable learning dynamics?

Instead of the orange dashed line (old reward) I implemented the sigmoid rewrad to have proper gradients. This will lead to inaccuracies for the reward close to the marginal costs. They get smaller the steeper the curve (chose k higher)

In EnergyLearningStrategy.calculate_reward():

1. Sigmoid acceptance probability:

   k = 2  # controls steepness
   accept_prob = 1 / (1 + exp(k * (bid_price - market_clearing_price)))
   accepted_volume_smooth = accepted_volume * accept_prob

Checklist

• Documentation updated (docstrings, READMEs, user guides, inline comments, doc folder updates etc.)
• New unit/integration tests added (if applicable)
• Changes noted in release notes (if any)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

Testing

Compared it for example_02a (green and grey with sigmoid) and example_02b (oragne and blue with sigmoid)

For an agent that sets the market price as in example_02a, it does not have any effect besides a differently scaled reward. As you can see the profit behavior is identical. However, for many agents who are supposed to bid their marginal costs most of the time, they seem to find it quicker and stay close to their marginal costs.

[kim-mskw]

Curious to hear your thoughts, @gugrimm, @reinecfi, and @mthede.

[reinecfi]

Smoothing the reward sounds like a great idea in general and this might come handy for a lot of cases.

Still I think there might be an issue regarding price setters:
When a supplier would be able to set the price between the prices M0 and M1 the reward can give false incentives:
As at the market price the smoothing function is exactly 1/2, the reward suddenly spikes below the lower boundary (M0 = 10 in the plot below) because the smoothing value rapidly rises towards 1.
The global optimum will be in the price taking region if M1 <= 2*M0 because the smoothing function is halfing the reward while oneself is price setting.

As a result price setters might not choose to do so because the global optimum (max reward) is at a way lower price.

Currently I am not sure how to fix this other than just capping the smoothing function at 1/2.
still, this wouldnt be smooth anymore...
Maybe you have better ideas @kim-mskw or @mthede?

[kim-mskw]

So this smoothing will always lead to an error... Naturally, but the one you @reinecfi are showing seems quite structurally problematic. What we could do is shift the sigmoid function to the right, so that it is in fact the actual reward once we are in the market. This would shift the error toward situations where we are not in the market, which should distort the reachable maximum less.