Categories

Tip

CodeGrade has a Help Center, with better guides, more videos and updated documentation. The documentation and guides on this website are deprecated and will not be updated in the future. Please click here to go to this page on the Help Center!

Within a level you can create a Category. Each Category has to be connected to a Rubric Category (see here for information how this rubric will be filled in). By doing this, AutoTest is feedback oriented. We encourage teachers to create meaningful Rubric Categories and think about how to group their tests accordingly. By grouping tests into AutoTest Categories that are connected to a Rubric Category, this also means that students know where tests belong and how they got their grade for an assignment.

Setup

Press the Add Category button inside a level to create a new Category within your AutoTest configuration.

Firstly, select the Rubric Category you want to connect it to.

Note

Make sure to create your Rubric Categories before creating your AutoTest Categories.

Next, you can add tests to this category. You have several different types of tests you can add to this.

  • IO Tests for simple input/output tests.

  • Run Program Tests to run a program and check if the program exited successfully.

  • Capture Points Tests to run a custom test program (such as a unit test) which captures a number between 0 and 1 and converts this to an amount of points.

  • Check Points, which check if the percentage of points gotten before this special test is high enough to continue with the tests after the Check Points test. This makes it possible to have certain tests only execute if tests before it were successful.

  • Unit Test for integration with unit testing frameworks that support JUnit compatible XML output.

Advanced options

Each category has a few settings that will be active for all tests within the category.

  • Timeout per step: When a single test in the category is taking longer than this amount of seconds, the process is killed, the status of the test will be set to timeout and no points are given for this test.

    It is preferred to keep this value low – while giving the student’s code enough time to run of course! – so that code that contains an accidental infinite loop will not fill up the AutoTest queue.

  • Network disabled: Disable the network while the tests in this category are running.

  • Submission information environment variables: Include information about the submission being tested in the $CG_INFO enfironment variable. The value is a JSON object with the following keys:

    • deadline The deadline of this assignment.

    • submitted_at The date and time the student submitted their work.

    • result_id An identifier unique to this AutoTest result. This value changes every time the AutoTest is run, even if it is run multiple times for the same submission of the same student.

    • student_id An identifier unique to the student for which the AutoTest is run. This value stays constant between runs of different submissions by the same student.

    See the AutoTest Best Practices guide for some concrete examples of how this can be used.

Running

When AutoTest is executed, each Category starts with a fresh snapshot of the environment setup. This means that Categories run independently of each other. Tests within the Category are executed in order.