QtRVSim online evaluation

The app is written in Flask and is using a PostgreSQL database.

Public version is running here.

Roadmap:

• Users are able to register
• Users are able to login
• List of tasks is displayed on the homepage
• Tasks are displayed on their separate pages which the homepage links to
• Logged in users are able to submit solutions on the submit page (with the task number)
• The submission form has a textarea for the task solution (library CodeMirror for assembly highlighting)
• After submitting, the file will be saved in a folder, and a new record will be created in the submission table
• A file will be evaluated and the submission closed, results will be saved in results file
• The results will be displayed on the submission page
• Automatic evaluator, which periodically checks for new submissions and evaluates them
• Find a way to compare the output to the expected output
• Evaluator compares the task output to the reference
• Comparison of reference registers and submission registers
• Comparison of reference memory and submission memory
• On each task page, a leaderboard will be shown with the users best submissions (in cycles), made by a request (to the submissions table) for that task
• Each user will only have the latest submission listed
• Basic app functionality Done?
• Users can view their last submissions (before it gets overwritten by new one of the same task)
• User will see their best score and the their latest submission score in the leaderboard
• Delete old, not needed submissions (not the latest and the best for each task and user, other can be deleted)
• Split submission table into pure submissions and results
• Add the starting template file to each task (instead of one template for all tasks)
• Remove the explicit declaration of do_comapare_registers and do_compare_memory, and implicitly set them to True, if the reference registers or memory are set
• Move database info into .env file
• Register confirm email
• Password reset
• Migrate database into PostgreSQL
• Implement database trigger for evaluator
• Write tasks, same as these: b35apo
• Implement cache settings (first line of the submission code)
• Implement uart communication
• Implement c file submission
• Correctly implement c syntax highlighting (fix submission.S appearing instead of submission.c)
• Implement submission viewer
• Implement admin view for the submissions (permission override)
• Add profile settings page
• Add about page
• Move the whole evaluation into try - catch, if something goes (for some reason) horribly wrong
• Implement random testcases
• Add a reevaluate button to invalidate a users result for a certain submission and run the evaluation again
• Implement a time window (deadline and start time) for the tasks - as optional parameters

Wiki page

All necessary information can now be obtained from the wiki page running at:

https://eval.comparch.edu.cvut.cz/wiki

or you can read the documentation itself in markdown at the docs/ folder.

Some of the information below may be outdated. Please check the wiki for up to date info.

Database structure

Users table

Field	Type	Length	Default
id	int	32	AUTO_INCREMENT
username	varchar	128	None
password	varchar	128	None
email	varchar	128	None
salt	varchar	128	None
verification_code	varchar	128	None
user_verified	tinyint	1	0
admin	boolean	1	false
display_name	varchar	64	None
country	varchar	128	None
organization	varchar	256	None
group	varchar	128	None
visibility	int	32	0

Email is a hash of the email adress, so it allows users to send a password to their email adress.

Visibility is either 0 (private) or 1 (visible to the same group and organization members) or 2 (visible to the whole organization) or 3 (visible to everyone).

Submissions table

Field	Type	Length	Default
id	int	64	AUTO_INCREMENT
userid	int	64	None
taskid	int	64	None
file	text		None
evaluated	boolean	1	false
time	datetime	None	current_timestamp()

User submits a task -> a submission is created. An evaluator evaluates the tasks in the order they came in the database. After a task is evaluated, it is marked as evaluated, so it is not evaluated more than one time. An evaluation log is created. The evaluated submissions is deleted from the database, and written into the results table.

Result file

Results

Field	Type	Length	Default
userid	int	64	PRIMARY
taskid	int	64	PRIMARY
result_file	text	64	NULL
last_source	text		NULL
best_source	text		NULL
score_best	int	32
score_last	int	32
time	datetime	None	current_timestamp()
result	smallint	16	-1

User submits a task -> a submission is created. An evaluator evaluates the tasks in the order they came in the database. After a task is evaluated, it is marked as evaluated, so it is not evaluated more than one time. An evaluation log is created.

Tasks table

Field	Type	Length	Default
id	int	64	AUTO_INCREMENT
name	varchar	64	None
path	varchar	256	None
available	tinyint	1	1
sequence	int	64	0

Triggers

Triggers have been added for better database management. The triggers are used to delete old submissions, and to update the results table.

Database is running PostgreSQL 16 on port 5432. Release app is running on a Eval-Comparch VPS https://eval.comparch.edu.cvut.cz.

Task creation

Tasks will be stored in toml format, with structure similar to this one (rewritten it to make it more readable, user friendly).

Basic structure

[task]
name = "Bubble Sort"
template = "S_templates/bubble.S"

description = '''
# Bubble Sort.
**Write a program that sorts an array using bubble sort algorithm.**

The size of the array will be located at the address `array_size`, the integer array (32-bit integer words) will start
at the address `array_start`.

The program should sort the array in ascending order.
Your program will be tested with different array sizes and different values in the array.
'''

[arguments]
run = "--dump-cycles --cycle-limit 5000"

[[inputs]]
data_in = "Size of the array located at address array_size, the array start is located at the address array_start."
data_out = "Sorted array of length array_size"
description = "Cache optimized sorting."

[[testcases]]
name = "5 elements"

[[testcases.starting_mem]]
array_size = [5]
array_start = [1, 3, 4, 5, 2]

[[testcases.reference_mem]]
array_start = [1, 2, 3, 4, 5]

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.starting_mem]]
array_size = [5]
array_start = [16, 8, 4, 2, 1]

[[testcases.reference_mem]]
array_start = [1, 2, 4, 8, 16]

[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

Arguments are passed to the QtRVSim object, which is used to run the simulator.

Note that, in order to use the cycles metric, the simulator needs to be run with the --dump-cycles argument.

The final evaluation (and the score in cycles) is done by referencing the test name in score.testcase section of the task file.

Some other variations of the task file (which are currently implemented) are these:

[task]
cache_max_size = 16

Sets the maximum size and allows the user to set the cache size in the submission file. This is done by adding a line #cache:policy,sets,words_in_blocks,ways,write_method to any line of the submission file. If the setting is done incorrectly (or cache size is set in task file but is not present in the submission file), user will get a cache error.

[make]
Makefile="""
your makefile will be here
"""

If the task cannot be run with the QtRVSim integrated assembler (--asm parameter), it can be compiled with a makefile. The makefile needs to compile the submission.S target, the executable needs to be named submission. Please provide a target clean to clean your temporary files (compiled sources and such).

If you require another files to be present during the make process, you can add them to the files section of the task file. The files will be copied to the user's submission folder at the time of compilation. They will be deleted after the evaluation ends.

[[files]]
name = "crt0local.S"
code = """
"""

If the solucion should be written in C, add a flag c_solution = true to the task file. (this also requires custom makefile, which compiles submission.c to submission)

[task]
c_solution = true

For serial port communication, use:

[[testcases.input_uart]]
uart = "112233\n445566\n"

[[testcases.reference_uart]]
uart = "557799\n"

Task evaluation log

The log is saved as a text blob in the database and is displayed on the task page for logged users that already submitted their task. This file is displayed to the user in this way:

The latest score is highlighted in yellow, and the best score is highlighted in green.

A custom styling for CodeMirror has been written to make the log more readable.

Database config

Database configuration is made in a file .env

SECRET_KEY=
MAIL_SERVER=
MAIL_PORT=
MAIL_USE_TLS=False
MAIL_USE_SSL=True
MAIL_USERNAME=
MAIL_PASSWORD=
MAIL_DEFAULT_SENDER=
DB_USER=
DB_PASSWORD=
DB_HOST=localhost
DB_DATABASE=
DB_PORT=5432

Installation

Install docker and docker compose. After this you can build the application from the folder scripts/docker/ by running:

docker compose build
docker compose up

Set up the necessary variables in the variables.env file provided in the same folder. The default script creates a default user admin with password admin (with email address same as the one in the config file), so you needn't set up an admin account each time you deploy this application.

The docker compose creates three containers - the psql database server, the flask web app server and the evaluator service.

To delete the containers simply run

docker compose down

If you wish to delete the volumes as well, this can be done with

docker volume rm <volume_name>

Creating a new task

To create a new task, cd into the scripts/ folder and run tasks.py --create. The script will automatically create a taskfile, as well as inserting the new task into the database. More options are available, run tasks.py --help for more information.

Create a new task in the folder web/tasks/ and a new template in web/S_templates/. After this login as an admin user and go to the admin console of the webpage located at /admin (eg. localhost:8000/admin). There enter the name of the task without the folder (such as task.toml for a file in web/tasks/task.toml) and click the create task. In the admin console, you can also change the name of the task displayed on the front page, as well as the order of the tasks.

Dumping the database

To dump the database, run scripts/pg_dump.sh. The script will create a dump of the database as a .sql file.

Creating a new database

To create a new database, modify and run the scripts/create_database.sh script. The script will create a new database and the necessary tables, as well as triggers and a new user.

Examples and other info

Slides for Installfest 2024 with Flask examples can be found on GitHub. An example for QtRvSim task confguration is in example 5.

Acknowledgements

• Flask
• PostgreSQL
• CodeMirror
• Bootstrap
• QtRVSim
• University Domains and Names Data List