DSL test suite reference

TIP

This is a reference guide, for advanced users. For recipes for common testing scenarios, see the test suite guide. We also have a set of tutorials for creating certain types of exercises.

In TESTed, a test suite specifies which test cases are executed against a submission. TESTed differs from other test frameworks in that its test suites are independent of any programming language. As a result, a single test suite is sufficient to check submissions for the same exercise in different programming languages.

While TESTed has an advanced format for the test suites, we have also developed a small domain-specific language (DSL), to make creating common exercises much easier. This document is the reference for the DSL test suite format, and contains all options and possibilities.

DSL test suites are written in YAML. A JSON Schema of the format is available in the TESTed repository, which can enable checks and autocompletion in your editor. If you use VS Code, you can also install our extension, which configures this JSON Schema automatically.

Structure

The structure of a DSL test suites follows the general Dodona structure, and consists of three levels:

Tabs
Contexts
Test cases

Below we describe objects of each level. Mandatory attributes are indicated with a star (*). At the end of this document, there is a full example of a test suite.

Root of the test suite

The test suite starts with either a root object, or a list of tabs. The root object contains six attributes:

tabs*: a list of tab objects
namespace: the "namespace" for the code of the submission, such as the class name in Java.
config: the global configuration options
language: the language of the expressions and statements. If this attribute is not set to "tested", all expressions and statements (except for return values) will be programming-language-specific expressions or statements.
files: optional list of files
definitions: reusable definitions

Tabs

A tab object maps onto a tab in the output on Dodona. It has six possible attributes:

tab*: the name of the tab to be displayed in Dodona
contexts*: a list of contexts (if this is given, you cannot use the attribute testcases)
testcases*: a list of test cases (if this is given, you cannot use the attribute contexts)
config: the configuration options for this tab and all children
files: optional list of files for this tab and all children
definitions: reusable definitions

In a lot of exercises, you have precisely one testcase per context. This is exactly what you can do using the testcases attribute: behind the scenes, each testcase will be placed in its own context.

Hint

While there are six possible attributes, each tab object can only have five, since contexts and testcases are mutually exclusive.

Contexts

A context is a group of test cases that depend on each other. For example, test cases that use variables must be in the same context. The context object has four attributes:

testcases*: a list of test cases
config: the configuration options for this context and all children
context: an optional description of the context
files: optional list of files

In most cases, it is fine to leave the description empty.

Each context must have at least one test case. Since each context is executed separately, the following two constraints apply:

Only the first test case may have a "main call", i.e. command line arguments or stdin.
Only the last test case may have a test for the program's exit code.

Do note that the first and last test case may be the same one: if you only have one test case, it may be a main call and have a check for the exit code.

Test cases

Test cases are the building blocks of a test suite, and contain some input and the expected outputs (the tests). Within each context, the following constraints apply to test cases:

Only the first test case may have a "main call", i.e. command line arguments or stdin.
Only the last test case may have a test for the program's exit code.

Do note that the first and last test case may be the same one: if you only have one test case, it may be a main call and have a check for the exit code.

A test case can have the following attributes:

description: an optional description of the test case, which Dodona shows instead of the generated one. Either a string, or an object with the attributes description (the text to show) and format (the format of that text: text by default, html or a programming language).
files: optional list of files. On a test case, this attribute is deprecated: use input_files instead.
input_files: optional list of input files for this test case. Each file is an object with a path (the location of the file in the working directory) and optionally content (the content of the file; tag the value with !path to read the content from a file in the evaluation folder instead).

Additionally, a test case can have all attributes described below, but do note:

A test case can only have one "input", meaning the arguments/stdin, expression and statement attributes are mutually exclusive. The exception is that stdin can be combined with an expression (see stdin).
The attribute return requires the attribute expression.

`stdin`

The data to provide to the standard input.

If this attribute is used, you cannot specify a statement as input. You can combine stdin with an expression; it is not mandatory to use arguments:

yaml

- tab: "example"
  testcases:
  - stdin: "Jan"
    expression: "greet()"
    return: "Hello, Jan."

`arguments`

A list of strings to pass to the program as the command line arguments.

If this attribute is used, you cannot specify expression or statement as input, nor can you use return as tests.

`expression` / `statement`

This attribute can take two values: a string or an object.

A string contains the expression to evaluate or statement to execute during this test case. For a statement, in contrast to for an expression, all return values are ignored if there are any.

Expressions and statements use the Python syntax, with some restrictions, which are detailed here.

If the value is an object, it must be a mapping of programming language to a language-specific expressions or statement.

yaml

expression:
  python: "submission.the_function()"
  java: "Submission.theFunction()"

`stdout` / `stderr`

Specifies the expected output on standard output and standard error respectively.

The attribute is either a string (in which case the string is the expected value), or an object for more advanced cases. The object has the following attributes:

data: the expected data, same as using a string
config: the configuration options

`exception`

Specifies the expected message of an expected exception.

To also check the type of the exception, use an object with two attributes: message (the expected message) and types, a mapping of programming language to the name of the expected exception type, since exception types are programming-language-specific:

yaml

- expression: "divide(5, 0)"
  exception:
    message: "Cannot divide by zero"
    types:
      python: "ZeroDivisionError"
      java: "ArithmeticException"

`return`

Specifies the expected return value.

By default, this attribute is interpreted as a YAML value. For example, a YAML string will result in a literal string value.

If you need more advanced return values, there are two options:

string tagged as !expression use the same Python syntax as for expressions and statements
objects tagged as !oracle denote the return value oracle (a custom check function) (see below)

`output_files`

Specifies the files that the submission should create.

The attribute is a list of expected files, each an object with two attributes:

path: the location where the submission should write the file, relative to the working directory
content: the expected content of the file. Tag the value with !path to read the expected content from a file in the evaluation folder instead of writing it inline.

yaml

- expression: "write_greetings()"
  output_files:
    - path: "hello.txt"
      content: "Hello!"
    - path: "hello2.txt"
      content: !path "expected_hello.txt"

For more advanced cases, the attribute can also be an object with the following attributes:

data: the list of expected files, as described above
config: the configuration options for comparing the file contents. In addition to the options for text, there is the option mode: full (the default) compares the whole file at once, while line compares the file line by line.
the attributes of a custom check function, if one is needed

In older test suites, this attribute is called file and consists of an object with the attributes content (the path to a file in the evaluation folder containing the expected content) and location (the location where the submission should write the file). This form still works, but is deprecated.

`exit_code`

Specifies the expected exit code of the program.

Note that only the last test case of a context can have this attribute, although the last test case can also be the first test case if needed.

Custom check function (oracles)

The following attributes can have a custom check function: return, stdout, stderr and output_files.

An object for a custom check function has the following attributes:

oracle: the type of check function. This can be custom_check or builtin. builtin uses the built-in oracle. For custom_check with return, the return object should be tagged with !oracle
value (with return) or data (with stdout/stderr/output_files): the expected value (for advanced values see !expression above)
file: the name of the file containing the custom check function (relative to the evaluation folder)
name: the name of the check function (in snake case)
arguments: a list of values that are arguments to the check function
languages: optional list of programming languages for which the check function can be used. By default, the check function is used for all programming languages.

For a return value:

yaml

return: !oracle
  value: "27-08-2023"
  oracle: "custom_check"
  file: "test.py"
  name: "evaluate_test"
  arguments: [5, 6]

For stdout and stderr, the same notation is used, but with data instead of value:

yaml

- tab: "Today"
  testcases:
    - stdin: '1 + 1'
      stdout:
        data: "2"
        oracle: "custom_check"
        file: "test.py"
        name: "evaluate_stdout"

The check function must have the following signature:

python

from evaluation_utils import EvaluationResult, ConvertedOracleContext

def check_function(context: ConvertedOracleContext, *) -> EvaluationResult

The first argument of the check function is always a ConvertedOracleContext. This object has a few attributes:

expected: the expected value of the oracle as defined by the key value in the test suite
actual: the value that was actually generated by the submission
execution_directory: path to the folder where the submission was judged
evaluation_directory: path to the evaluation folder of the exercise (that contains the test suite)
programming_language: the programming language of the submission
natural_language: the natural language of the user that submitted this submission

The other arguments are the same as the arguments attribute from the test suite. In this example, the check function would have three arguments: the context and the two numbers from the test suite.

The return value is a class of the type EvaluationResult from the module evaluation_utils. The constructor of this class has the following parameters:

result: A boolean indicating if the generated value is correct or not.
readable_expected, optional: The expected value to show on Dodona.
readable_actual, optional: The generated value to show on Dodona.
messages, optional: A list of messages (Messages or strings). These messages are also shown on Dodona and can be used to give students additional feedback or explanation.
dsl_expected, optional: The expected value as string value. TESTed will convert this to the programming language of the submission before showing it on Dodona.
dsl_actual, optional: The generated value as string value. TESTed will convert this to the programming language of the submission before showing it on Dodona.

In most cases, and especially when preparing programming-language-independent exercises, it is better to use dsl_expected and dsl_actual: otherwise the check function itself is responsible for displaying the expected and actual value in the correct programming language.

The list of messages must be strings or Messages. A Message is a class from the evaluation_utils module and has the following attributes:

description: the message to show.
format: the format of the message, like text, code or html.
permission: who can see the message: staff, student or zeus.

This becomes:

python

from evaluation_utils import EvaluationResult, Message

def evaluate_test(context):
    return EvaluationResult(
      result=True,
      dsl_expected=repr("hallo"),
      dsl_actual=repr("hallo"),
      messages=[Message(
        description="Hallo",
        format="html",
        permission="staff"
      )]
    )

Finally, for return there is a third type of check function: specific_check. Instead of one programming-language-independent check function, you provide a check function per programming language:

functions: a mapping of programming language to an object with the attributes file and name of the check function
arguments: a mapping of programming language to a list of language-specific arguments for the check function

More information about these check functions is available in the reference for the advanced format.

Files

Some parameters or other strings are a name of a file. If you want that parameter to link to the actual file, it needs to be added to the list of files. Each object in this list has two attributes:

name: the name of the file as it appears in the input
url: the location where the link should point to, relative to the exercise folder

The list of files can be specified at the root, tab and context level, and applies to all levels below it.

Definitions

The root object and tab objects can have a definitions attribute. This is a place to define YAML objects that are used multiple times elsewhere in the test suite. TESTed itself ignores the contents of this attribute: combine it with YAML anchors (&name) and aliases (*name) to reuse the defined objects:

yaml

definitions:
  rounding: &rounding
    tryFloatingPoint: true
    applyRounding: true
    roundTo: 2
tabs:
- tab: "Sum"
  config:
    stdout: *rounding
  testcases:
  - arguments: ["2.125", "1.212"]
    stdout: "3.34"

Configuration options

The configuration object can be specified at the root, tab and context level, and applies to all levels below it. For example, specifying the config on the tab level means it will apply to all contexts and, in turn, all test cases within that tab. For a single test, the options can be set in the object form of the stdout/stderr and output_files attributes.

The configuration object has three attributes:

stdout: the configuration options for standard output
stderr: the configuration options for standard error
file: the configuration options for expected files (see output_files), including the mode option

Test options

This object contains a set of configuration options that influence how the test results are checked by TESTed. The following options are available:

applyRounding: apply rounding when comparing values as float point numbers
roundTo: the number of decimals to round to, if applyRounding is true
caseInsensitive: ignore the case of text when comparing strings
ignoreWhitespace: ignore leading and trailing whitespace
tryFloatingPoint: try comparing text as floating point numbers
normalizeTrailingNewlines: enforce the trailing newline convention on the expected text (default true)

Expressions and statements

In the test suite, expressions and statements are written as YAML strings, using the Python syntax. For example, a function call with one argument "hello":

yaml

expression: "a_function_name('hello')"

Since the Python syntax does not have a separate syntax for all features supported by TESTed, there are some conventions:

Function calls whose name begins with a capital are considered constructors, e.g. Constructor(56).
Identifiers that are in all caps are considered global constants, e.g. VERY_LONG_NAME.
Casts are done using the normal Python way. For example, to cast a number to int64: int64(56). However, there is no support for Python constructors. A set must be noted as set([1, 2, 3, 5]), not as set(1, 2, 3, 5).

Additionally, most of the syntax is not supported, since TESTed only has support for limited expressions and statements. The following is supported:

Simple values, such as 5, -9.3 or "Hello world".
Complex values, such as [5, 6, 7], {5, "Hello"} or {"key": "value"}.
Function calls, including named arguments the_function(5, named=6). Do note that named arguments are converted to positional arguments in programming languages that do not support named arguments.
Constructors (using our convention).
Assignments, such as some_variable = 5.
Referencing variables, such as the_function(some_variable).

Notably, absent are any type of function or class definitions and all operators.

Language-specific expressions and statements

If language-specific expressions or statements are used (either by setting the language globally or by using an object to an attribute expression or statement), the string will be used literally in the test code.

This has the advantage that all language features of the programming language can be used. On the other hand, this causes exercises to no longer be programming language independent. You have to use the correct namespace yourself, and it will not work for functions with return type void. More information and discussion at https://github.com/dodona-edu/universal-judge/issues/423.

Since TESTed cannot analyse these strings, it is necessary to use the namespace yourself. This is the name of the submitted solution or class (configurable with the attribute namespace). This name is programming language dependent:

yaml

- tab: "My tab"
  testcases:
  - expression:
      c: "to_string(1+1)"
      cpp: "to_string(1+1)"
      haskell: "Submission.toString (1+1)"
      runhaskell: "Submission.toString (1+1)"
      java: "Submission.toString(1+1)"
      javascript: "submission.toString(1+1)"
      kotlin: "toString(1+1)"
      python: "submission.to_string(1+1)"
      csharp: "Submission.toString(1+1)"
    return: "2"

If you only want to support one programming language, you can also set the language of the expressions and statements globally, with the language attribute of the root object:

yaml

language: "java"
tabs:
- tab: "My tab"
  testcases:
  - expression: "Submission.toString(1+1)"
    return: "2"

Supported tags

TESTed supports the following standard YAML types:

!!set to denote a set.

Finally, all TESTed types can also be used as tags. For example !int64 or !double. Note that custom types use one exclamation mark, while standard types use two.

Newlines for textual results

For the result of stdout and stderr, TESTed follows this convention: either the text should be empty, or the text should end with a newline. TESTed will enforce this convention: if the text in the test plan does not end with a newline, TESTed will add a newline.

This is the same convention as in POSIX, and is also applied in many programming languages. For example, print in Python will add a newline by default.

This behaviour can be disabled with the normalizeTrailingNewlines test option.

YAML cheat sheet

This section contains a very brief overview of the YAML features used in the DSL.

Objects

Objects in YAML are key-value pairs, where the key (the attribute) and value are separated by a colon:

yaml

key: value

Nested objects are created using indentation:

yaml

root:
  child0:
    subchild0: "leaf"
    subchild1: "leaf"
  child1:
    subchild0: "leaf"

Lists

Lists in YAML can be written either on one line (using the JSON syntax) or with one value per line. For example, a list on one line

yaml

["Item 0", "Item 1", "Item 2", "Item 3"]

When using one value per line, each value must be prefixed with a dash (-) and space:

yaml

- "Item 0"
- "Item 1"
- "Item 2"
- "Item 3"

You can also combine lists and objects:

yaml

list:
- name: "Item 0"
  items: 5
- name: "Item 1"
- name: "Item 2"
  items: 3
- name: "Item 3"

Strings

Ordinary strings in YAML are written using double quotes:

yaml

description: "Hello"

However, doing multi-line strings is rather ugly:

yaml

description: "Hello\nWorld"

YAML supports special syntax for multi-line strings. Writing the same string as the last example, we get:

yaml

description: |
  Hello
  World

The reverse is also possible, which are called "folded strings". With this syntax, YAML will remove newlines:

yaml

description: >
  Hello
  World

This is equivalent to writing:

yaml

description: "Hello World"

Full example

yaml

# A tab on Dodona.
- tab: "Name of the tab"
  contexts:
    # The files used in this context.
    - files:
        - name: "file.txt"
          url: "media/workdir/file.txt"
      testcases:
        # An assignment of the variable data.
        - statement: 'data = ["list\nline", "file.txt"]'
          # Function call that uses the variable.
        - expression: 'function(data, 0.5)'
          # Expected return value of the function.
          return: [ 0, 0 ]
    - testcases:
        # A function call where the value is cast to "uint8".
        - expression: 'echo(uint8(5))'
          # The expected return value is also cast to "uint8".
          return: !expression "uint8(5)"

# A second tab in the same test suite.
- tab: "Exception"
  contexts:
    - testcases:
        # Another function call.
        - statement: 'function_error()'
          # The expected text on stdout.
          stdout: "Invalid"
          # The expected text on stderr.
          stderr: "Error"
          # We expect an error or exception with the message "Unknown".
          exception: "Unknown"

# A third tab.
- tab: "Arguments"
  testcases:
    # This program gets input via stdin.
    - stdin: "Alice"
      # There are also command line arguments.
      arguments: [ "stdin" ]
      # The expected text on stdout.
      stdout: "Hello Alice"

# A fourth tab.
- tab: "Config"
  # We configure everything on the tab level.
  config:
    stdout:
      # First try to compare text on stdout as float.
      tryFloatingPoint: true
      # When comparing floats, round to 2 decimals.
      applyRounding: true
      roundTo: 2
    # On stderr we ignore white space and make it case insensitive.
    stderr:
      ignoreWhitespace: true
      caseInsensitive: true
  contexts:
    - config:
        stdout:
          # We override the tab configuration for this context.
          roundTo: 0
      testcases:
        - statement: 'diff(5, 2)'
          stdout: "2"
        - statement: 'diff(5, 2)'
          stdout:
            data: "2.5"
            # We override the context configuration in this test.
            config:
              roundTo: 4

DSL test suite reference ​

Structure ​

Root of the test suite ​

Tabs ​

Contexts ​

Test cases ​

stdin ​

arguments ​

expression / statement ​

stdout / stderr ​

exception ​

return ​

output_files ​

exit_code ​

Custom check function (oracles) ​

Files ​

Definitions ​

Configuration options ​

Test options ​

Expressions and statements ​

Language-specific expressions and statements ​

Supported tags ​

Newlines for textual results ​

YAML cheat sheet ​

Objects ​

Lists ​

Strings ​

Tags ​

Full example ​

DSL test suite reference

Structure

Root of the test suite

Tabs

Contexts

Test cases

`stdin`

`arguments`

`expression` / `statement`

`stdout` / `stderr`

`exception`

`return`

`output_files`

`exit_code`

Custom check function (oracles)

Files

Definitions

Configuration options

Test options

Expressions and statements

Language-specific expressions and statements

Supported tags

Newlines for textual results

YAML cheat sheet

Objects

Lists

Strings

Tags

Full example