What happens behind the scenes when we set `self.title` is that the result is pre-computed on the instructors machine and cached. This means the last test will display the correct result regardless of how `reverse_list` has been implemented by the student. The titles are also shown correctly when the method is run as a unittest.
...
...
@@ -288,7 +305,6 @@ Finally, notice how one of the tests has a return value. This will be automatica
To use `unitgrade` as part of automatic grading, it is recommended you check the students output locally and use hidden tests. Fortunately, this is very easy.
Let's start with the hidden tests. As usual we write a complete report script (`report3_complete.py`), but this time we use the `@hide`-decorator to mark tests as hidden:
Just to check, let's have a quick look at the students report script `report3.py`:
```python
# example_docker/instructor/cs103/report3.py
from unitgrade import UTestCase, Report
...
...
@@ -335,7 +348,6 @@ from unitgrade.utils import hide
from unitgrade import evaluate_report_student
import cs103
class AutomaticPass(UTestCase):
def test_automatic_pass(self):
self.assertEqual(2, 2) # For simplicity, this test will always pass
...
...
@@ -354,15 +366,16 @@ The grade script works as normal, and just to make the example self-contained, l
```
### Setting up and using Docker
We are going to run the students tests in a Docker virtual machine so that we avoid any underhanded stuff, and also because it makes sure we get the same result every time (i.e., we can pass the task on to TAs).
To do that, you first have to install Docker (easy), and then build a Docker image. We are going to use one of the pre-baked images from https://gitlab.compute.dtu.dk/tuhe/unitgrade_private/-/tree/master/docker_images, which simply consists of a lean Linux distribution with python 3.8 and whatever packages are found in `requirements.txt`. If you need more, it is very easy to add.
To build the Docker image simply run:
To do that, you first have to install Docker (easy), and then build a Docker image. We are going to use one of the pre-baked images from https://gitlab.compute.dtu.dk/tuhe/unitgrade_private/-/tree/master/docker_images, which simply consists of a lean Linux distribution with python 3.8 and whatever packages are found in `requirements.txt`. If you need more it is very easy to add.
To download and build the Docker image simply run:
```python
# example_docker/instructor/cs103/deploy.py
# Step 3: Compile the Docker image (obviously you should only do this once).
@@ -386,10 +400,9 @@ Just to show it works we will load both `.token`-files and print the results:
```python
# example_docker/instructor/cs103/deploy.py
# Load the two token files and compare their scores
with open(token, 'rb') as f:
checked_token = pickle.load(f)
with open(student_token_file, 'rb') as f:
results = pickle.load(f)
checked_token, _ = load_token(token)
results, _ = load_token(student_token_file)
print("Student's score was:", results['total'])
print("My independent evaluation of the students score was", checked_token['total'])
```
...
...
@@ -478,12 +491,75 @@ What happens behind the scenes is that a code-coverage tool is run on the instru
to determine which methods are actually used in solving a problem, and then the hint-texts of those methods are
collected and displayed. This feature requires no external configuration; simply write `Hints:` in the source code.
# CMU Autolab support (Experimental)
CMU Autolab is a mature, free and opensource web-based autograder developed at Carnegie Mellon University and used across the world. You can find more information here: https://autolabproject.com/. It offers all features you expect from an online autograder
- Web-based submission of homework
- Class-management
- Build in TA feedback mechanism
- Class monitoring/statistics
- Automatic integration with enrollment data (Autolab supports LDAP and Shibboleth) means Autolab can be `plugged in` to existing IT infrastructure (including DTUs)
- CLI Tools
An important design choice behind CMU Autolab is the grading is entirely based on Makefiles and Docker VMs. I.e., if you can make your autograding scheme work as Makefile that runs code on a Docker image you specify it will work on Autolab. This makes it very easy to let third-party platforms work with an **unmodified** version of Autolab. The following contains all steps needed to compile a Unitgrade test to Autolab
### Step 1: Set up Autolab
Simply follow the guide here: https://docs.autolabproject.com/installation/overview/ to set up Autolab. I used the 'manual' installation, but it should also work with the Docker-compose installation.
### Step 2: Compile a unitgrade test to Autolab lab-assignment format
Autolab calls handins for `lab assignments`, and allow you to import them as `.tar`-files (see the Autolab documentation for more information). We can build these automatically in a few lines as this example demonstrates.
The code for the example can be found in `examples/autolab_example`. It consists of two steps. The first is that you need to build the Docker image for Autolab/Tango used for grading. This is exactly like our earlier example using Docker for Unitgrade, except the image contains a few additional autolab-specific things. You can find the image here:
Next, simply call the framework to compile any `_grade.py`-file into an Autolab-compatible `.tar` file that can be imported from the web interface. The script requires you to specify
both the instructor-directory and the directory with the files the student have been handed out (i.e., the same file-system format we have seen earlier).
```python
# autolab_example/deploy_autolab.py
# Step 2: Create the cs102.tar file from the grade scripts.
This will produce a file `cs102.tar`. Whereas you needed to build the Docker image on the machine where you are running Autolab, you can build the lab assignments on any computer.
### Step 3: Upload the `.tar` lab-assignment file
To install the `cs102.tar`-file, simply open your course in Autolab and click the `INSTALL ASSESSMENT` button. Click `Browse` and upload the `cs102.tar` file:
You will immediately see the page for the assignment where you can begin to upload solutions!
The solutions are (of course!) `.token` files, and they will be automatically unpacked and run on Autolab.
To test it, press the big upload square and select the `.token` file for the second assignment found in `examples/example_framework/instructor/cs102/Report2_handin_18_of_18.token`.
The file will now be automatically evaluated and the score registered as any other Autolab assignment:
and TAs can choose to annotate the students code directly in Autolab -- we are here making use of the fact the code is automatically included in the top of the `.token`-file.
