Exclude time for VUs creation from the ramp-up time

[ralevex]

I am running performance testing using Autopilot with a broad range of load levels - from 256 VUs to 1,024 VUs. When I configure Driver script, I have to specify ramp-up time sufficiently high to allow for creation of the number of virtual users specified by Autopilot.

I noticed that HammerDB creates just over a 100 VU per minute, so I have to specify ramp-up time of at least 10 minutes to allow enough time to create all VUs for the largest number of VUs in Autopilot sequence, which is too much time for the initial number of VUs, which is 256. I also have to specify a very long time (10 + 5 + 1) for "Minutes per test in Autopilot sequence".

It would be helpful for HammerDB users if the ramp-up time calculation started AFTER all requested VUs are created and HammerDB just wait until requested number of users created prior to applying ramp-up time andm "Minutes for thes duration".

This would significantly speed up overall time required for long and diverse Autopilot sequences and make ramp-up time uniform across VUs in the test set.

[ralevex]

Steve, this is Alex Zarenin (https://github.com/Alex-Zarenin) - you assisted me with the issue #541 - this time by mistake I logged in with my alternative account - is there any chance to reassign this issue to my primary account?

[sm-shaw]

If the issue is the time taken to login between user sessions is the primary issue then you can control this with the User Delay(ms) setting as below. This is set in milliseconds with a default of 500 so HammerDB will pause for 1/2 sec between each Virtual User logging on to prevent a login storm on the server. So for example if you change this value to 1000 then you will see the logins waiting for a second between them.

However you can also reduce this and the example below only allows 50 milliseconds between logins.

The correct setting will depend on the system being tested on how quickly it can respond to simultaneous login requests. (and this will have improved a lot since this feature was first added for older systems).

Note that the other setting repeat delay is when the iterations value has been increased and a virtual user is going to run its script more than once. At some point there was a bug so where User delay and repeat delay where reversed, but on the latest versions this is not an issue.

Please advise if setting the User Delay resolves the issue. Otherwise the explanation is benefical, I can see the positives of adding a setting for the monitor virtual user to wait until all of the virtual users have logged in, something very similar is already done for the builds using tsvs - thread shared variables, the monitor holds the build VUs until the schema is created. The same approach could be used but instead to wait for all VUs to login.

However, there could be potential for scenarios such as if one virtual user failed to login and notify then all of the other virtual users would hang waiting for the monitor to release them so it could introduce bugs.

Let me know your feedback on the existing timing settings and we could look into this more.
For now i cannot assign to you in the account list with your main account, However if you comment with this account you want assigning then I think I will be able to.

[sm-shaw]

@Alex-Zarenin, @ralevex please let me know if adjusting the User Delay value provided a satisfactory solution to the issue.

[ralevex]

Unfortunately, this does not address the issue - my concern is not the user login squal, but overall timing of the session - in my tests I do not use user delay between logins, still creating 1,000 VUs takes about 8-9 minutes and this time need to be accounted for in the ramp-up time.. For example, if I want to have 2 min. of ramp-up time and 5 minute test time, the scenario will look like this:

For a test with 250 users it will be 2 minutes to create users and 2 minutes for ramp-up time plus 5 minutes for test - so in the Driver script configuration I would have to specify ramp-up time of 4 minutes ( 2 min to allow all users to be created and 2 minute for actual ramp-up for the last user) and 5 min for test. In Autopilot I would set Test time to 10 minutes (4 minutes ramp-up, 5 minutes test, and 1 minute to destroy VUs). Even in this simple scenario, the first created user will be in ramp-up for 4 minutes, while the last - for 2 minutes.

Now, if my Autopilot sequence will be (250, 500, 1000) VUs, in the ramp-up time I would have to account for the test with the largest amount of VUs - 1,000 in this case. With the same user creation time of just over 100 VU/min, my ramp-up time on the Driver script should be 10 minutes to allow creation of all required VUs for largest experiment plus 2 minutes to allow ramp-up for the last VU created. Thus the ramp-up time, which used across all experiments in Autopilot session, should be set to 12 minutes. Add 5 minutes for actual test, and my Autopilot experiment time will be 12 + 5 + 1 = 18 minutes.

Now, with this configuration, the smaller experiment in Autopilot session, like the one with 250 users, will also take 18 minutes while it requires just 10.

My recommendation for changing accounting for time is as follows:

1. HammerDB does not start measuring time untill all requested VUs are created;
2. Accounting for ramp-up time starts after all VUs are created;
3. Accounting for Autopilot Experiment time also starts after all VUs are created and should be set to "ramp-up" time + "test time" + "1 min to destroyVUs"

This would simplify setting up Autopilot and significantly shorten time for the whole Autopilot run time! As the more powerful machines come into play - for example, I am testing SQL Server performance on a machinewith 128 CPUs, 5 TB of RAM and storage system supporting 350,000 IOPS - I have to create Autopilot sequences with the broad range of VUs from 250 to 2,000. These experiments take much more time than nactually necessary because of rather inflexible time accounting between Driver script and Autopilot.

[sm-shaw]

Many thanks for the detailed clarification. This does make sense now and explains what is required.

This would be the monitor virtual user waiting on a TSV (thread shared variable) like the build until all virtual users have logged on and then starting the rampup. This would need to be a separate option from the default as it would mean a bigger change for most users who are not using such large VU counts.
Then for the autopilot if this option is selected it can be selected to test + ramp up + configurable additional time - this is already done for CLI scripting so means changing the behaviour to be the same. The additional time can sometimes need to be longer than 1 min such as when gathering timing data and using asynchronous users and is set as the keepalive_margin value in the CLI XML configuration.
The requirements are understood and is a valid enhancement development task for a future enhancement. As always pull requests are very welcome to add such functionality and is the best approach for features to be added in a timely manner as possible.