Best Practices for AutoTest¶
This is a list of common questions and best practices for CodeGrade AutoTest. This is by no means a complete list, if you have any other questions, please consult the AutoTest User Documenation or the CodeGrade Support Team at firstname.lastname@example.org.
Where do I compile students’ code (and how do I stop when compiling fails?)¶
We recommend two different ways to compile students’ code. Which one to use depends on the application.
- Using the per-student setup script
If you want to use the compiled code in multiple categories, we recommend using the per-student setup script for compiling. Either use a compilation script, which you upload as a fixture, or input the compilation command directly in the input field.
If you want to stop AutoTest when the compilation fails, you can do this in the following way:
- Create a compilation rubric category.
- Create a new AutoTest level and add the compilation category in this level.
- Use a Run Program step to check whether compilation was successful (e.g. by checking if the compiled files exist).
- Save this category and create a new AutoTest level to put all your other test categories.
- Set the Only execute further levels to 100%.
- Using a Run Program step and Checkpoint step
If you only want to use the compiled code in one category (e.g. when every category has a different program), we recommend using a Run Program step combined with a Checkpoint to compile the code.
- Create a Run Program step with the compilation command.
- Create a Checkpoint step right below the Run Program step and set it to 100%.
In this way, the category will stop testing if the Run Program step fails.
Keep in mind that the state of AutoTest is reset to the state after the per-student script at the start of each category. So all your compiled files from method 2 are lost after the category finishes executing.
How to use weights and set rubric points?¶
The final grade of an AutoTest run is not defined by the weights you set in AutoTest, but by the amount of points a rubric level in a category has that is reached by AutoTest.
To start setting the weights, first select the rubric calculation mode. Either minimum, where a rubric category item will be chosen when the lower bound is reached, or maximum, where a rubric category item will be chosen when the upper bound is reached.
You want to use maximum when students need to pass all tests in an AutoTest category, before they should get the maximum item in the rubric category.
Let’s go over an example to make this more clear. This is the rubric category we want to create tests for:
|Item name||Nothing works (0)||Compiling works (1)||Simple tests work (5)||Advanced tests work (10)|
|Percentage range to reach item||0%-50%||50%-75%||75%-100%||100%|
As you can see, the maximum mode is selected, as you only reach the last rubric item (Advanced tests work) with 100% of passed tests.
|Checkpoint||Stop if compilation fails||-|
|IO Test (4 substeps)||Simple tests||4|
|Capture Points||Advanced tests||4|
As you can see here, the compile step actually has the highest weight, but will get the student the least amount of points. This is due to the fact that you need a weight of 8 to get 50% in the rubric category, which in turn will get you the Compiling works item.
Both the simple tests and advanced tests have a weight of 4, which is both 25% of the total amount of points achievable, which will make sure the right rubric item is filled in.
How to manually adjust AutoTest grades?¶
You can override the grade at all times by changing it in the grade input field. If you rerun AutoTest, this overridden grade is preserved. If you only want to adjust the grade down, you can also use a rubric category with negative weights (so one item in the category with 0 points, and all the other items with less than 0 points).
How to install packages and third party software?¶
Installing packages and third-party software can be done easily using the
global setup script. Either upload a bash script with installation commands
which you upload as a fixture, or input it directly in the input field. You can
install Ubuntu packages with
sudo apt-get install -y PACKAGE_NAME.
Always make sure to give the
-y option to
apt-get, otherwise the package
How to assess style and structure?¶
You can assess style and structure by using a linter. Write a wrapper script around the linter you’d wish to use, which will parse the output of the linter and give it a score and then use a Capture Points test to load this into AutoTest.
For some linters, CodeGrade can provide these wrapper scripts for you, contact us for more details.
Sometimes some warnings/errors are more important than others, keep this in mind while writing your wrapper script.
How to use a unit testing framework?¶
You can use a unit testing framework, by writing a wrapper script or by slightly modifying the test code, so that it outputs a score between zero and one at the end of the tests. In this way you can use a Capture Points test in AutoTest to capture the score of the unit tests.
For common unit testing frameworks, we already have written frameworks to easily and seamlessly use them in CodeGrade, please contact us at email@example.com for details.
How to integrate existing grading scripts?¶
Using an existing grading script in CodeGrade is straightforward, just slightly modify the script so that it outputs a value between zero and one at the end, upload it as a fixture and use a Capture Points test to execute the grading script and capture the score.
If you need any help converting your existing grading scripts to CodeGrade grading scripts, feel free to contact us at firstname.lastname@example.org.
It is important to note that rubric calculation and capture points might be a bit difficult to combine sometimes, especially when combining with IO tests too. In some cases it might be better to split the test script into multiple scripts (or use command line arguments), and use multiple ‘run program’ tests instead.
How do I combine AutoTest and manual function testing?¶
This is easily achieved by splitting your rubrics into multiple categories, one category for the automated testing and one category for the manual testing. In this way, AutoTest will fill in the automatic category and you can fill in the manual category. This also has the advantage of a clear separation to your students, making it easier for them to see which part is assessed automatically and which part is assessed manually.
How to hide fixtures?¶
Firstly, you can hide your fixtures in the User Interface. By default, fixtures are hidden when you upload them. You can change the state by clicking the icon.
However, this still means the code of students will be able to access these fixtures on the AutoTest servers. You can limit this by using a special script. You can read more about this here.
If you’re uploading solutions as fixtures you probably want to limit student access.
How to use IO tests with floating point numbers¶
Sometimes students might output numbers in a different format, or use a different
type of rounding. CodeGrade supplies a
normalize_floats program in AutoTest
to solve this issue. You can use this in the following way:
normalize_floats amount_of_decimals program_to_run.
normalize_floats only transforms stdout and does not touch stderr.
How to let IO tests pass when the exit code is not 0¶
IO tests fail by default if the exit code of the program is not 0. Sometimes,
however, you want IO tests to also pass with another exit code than 0. You
can simply fix this by appending
|| true to your command, this will make
sure the exit code is always 0.
The “Input arguments” field of an IO step is appended to the command. This
means that if it is not empty, this technique will likely not produce the
expected results. To work around this case, add the
|| true to the
input arguments instead.
How to view AutoTest generated files¶
It may be desirable to inspect files that are generated during the run of an
AutoTest, such as compiled objects or IPython notebooks. By default generated
files are not saved, but they will be when you write them to the
directory. The files will then be accessible through the “Autotest output”
section of the file browser in the Code Viewer.