Unitgrade-private

Do not distribute this repository, or files from this repository, to students This repository contains the secret parts of the unitgrade framework.

At a glance

Homework is broken down into reports. A report is a collection of questions which are individually scored, and each question may in turn involve multiple tests. Each report is therefore given an overall score based on a weighted average of how many tests are passed.

In practice, a report consist of an ordinary python file which they simply run, and which executes a sequence of tests. An example:

python cs101report1.py

The file cs101report1.py is a non-obfuscated file which they can navigate and debug using a debugger. The file may contain the homework, or it may call functions the students have written. Running the file creates console output which tells the students their current score for each test:

Starting on 02/12/2020 14:57:06
Evaluating CS 101 Report 1

Question 1: Reversal of list
================================================================================
*** q1.1) ListReversalItem..................................................PASS
*** q1.2) ListReversalWordsItem.............................................PASS
*** Question q1............................................................. 5/5

Question 2: Linear regression and Boston dataset
================================================================================
*** q2.1) CoefficientsItem..................................................PASS
*** q2.2) RMSEItem..........................................................PASS
*** Question q2........................................................... 13/13

Finished at 14:57:06
Provisional evaluation
-----------  -----
Question q1  5/5
Question q2  13/13
Total        18/18
-----------  -----

Note your results have not yet been registered.
To register your results, please run the file:
>>> cs101report1_grade.py
In the same manner as you ran this file.

Once students are happy with the result, they run an alternative, not-easy-to-tamper-with script called cs101report1_grade.py:

python report1_grade.py

This runs the same tests, and generates a file cs101report1.token which they upload to campusnet. This file contains the results of the report evaluation, the script output, and optionally local files from the users system for plagiarism checking. The .token file itself is in a binary but easily readable format.

How to develop tests

To develop a new test, all you need is a working version of the students homework and the api will automatically created the expected output. This saves about half the work of creating tests and ensures tests are always in sync with the code.

As an examle, suppose the students write the code for the function reverse_list in the cs101courseware_example/homework1.py file. Our test script cs101report1.py, which was the same file the students ran before, contains code such as (we omitted one question for brevity):

class ListReversalQuestion(QuestionGroup):
    title = "Reversal of list"

    class ListReversalItem(QPrintItem):
        l = [1, 3, 5, 1, 610]
        def compute_answer_print(self):
            from cs101courseware_example.homework1 import reverse_list
            return reverse_list(self.l)

    class ListReversalWordsItem(ListReversalItem):
        l = ["hello", "world", "summer", "dog"]

class Report0(Report):
    title = "CS 101 Report 1"
    questions = [(ListReversalQuestion, 5), ] # In this case only a single question
    pack_imports = [homework1] # Include this file in .token file

if __name__ == "__main__":
    evaluate_report_student(Report0())    

This code instantiates a group of questions and run two tests on the students code: one in which a short list of integers is reversed, and another where a list of strings has to be reversed. The API contains quite a lot of flexibility when creating tests such as easy parsing of printed output.

All that remains is to prepare the files which are distributed to the students. This can be done in a single line:

if __name__ == "__main__":
    setup_grade_file_report(Report0)

This code will create two files:

• cs101report1_grade.py: This file contains all tests/expected output rolled into a binary tamper-resistant binary blob. Students run this file to generate their token files.
• Report0_resource_do_not_hand_in.dat: This contains the expected output of the tests (i.e., the students output is compared against what is in this file)

Both of these files, plus the actual tests cs101report1.py, are distributed to the students as is.

Why is there a seperate .dat file and seperate test/grading scripts?

Ideally, tests should be a help in creating correct code, and therefore we want the publically-facing test code to be as easy to work with as possible. For this reason the user test script, homework1.py, is completely transparent and does not include any exec(...) magic or similar. It simply

• Instantiates the Report class and loads the expected output from the .dat file (a simple, pickled dictionary with output)
• Execute the tests one by one in a for loop

Therefore, it will integrate well with a debugger and the homework could in principle be stated within the test script itself. Transparency flies in the face of security, which is why there is an obfuscated homework1_grade.py file which bundles the expected output with the test code.

Security/cheating

Any system that relies on local code execution is vulnerable to tampering. In this case, tampering will involve either changing the tests, or the .tokenfile. Both attempts involve figuring out what the main _grade.py does. The content of this file is a single line which executes a binary blob:

'''WARNING: Modifying, decompiling or otherwise tampering with this script, it's data or the resulting .token file will be investigated as a cheating attempt. 
            Note we perform manual and static analysis of the uploaded results.
            '''
import bz2, base64
exec(bz2.decompress(base64.b64decode('QlpoOTFBWSZTWWIfYn8ABUHfgERQev9/9/v//+7/////YAZc...etc.etc.')))

If this blob is decompressed, it will contain badly obfuscated source code bundled with the result of the tests, and figuring this out involves some work and there are no possibilities for accidential tampering.

If the script is decompiled successfully, and the user manage to generate a tampered .token file, the .token file will still contain the source code. Since the source is collected at runtime, it will be possible to prove definitely that cheating occurred. This also adds an extra layer of security for code-copying (or simply copying someone elses .token file). In light of this I think the most realistic form of cheating is simply copying someone elses code. This option is available for all homework, but a benefit of this system is we are guaranteed to have the code included in an easy-to-read format.