Skip to content
Snippets Groups Projects
Commit a9bc2f95 authored by Tue Herlau's avatar Tue Herlau
Browse files

Readme and framework

parent 3170f564
No related branches found
No related tags found
No related merge requests found
# Unitgrade-private
**Do not distribute this repository, or files from this repository, to students**
This repository contains the secret parts of the unitgrade framework.
## What it looks like to a student
## At a glance
Homework is broken down into **reports**. A report is a collection of questions which are individually scored, and each question may in turn involve multiple tests. Each report is therefore given an overall score based on a weighted average of how many tests are passed.
In practice, a report consist of an ordinary python file which they simply run. It looks like this:
In practice, a report consist of an ordinary python file which they simply run, and which executes a sequence of tests. An example:
```
python cs101report1.py
```
The file `cs101report1.py` is just an ordinary, non-obfuscated file which they can navigate and debug using a debugger. The file may contain the homework, or it may call functions the students have written. Running the file creates console output which tells the students their current score for each test:
The file `cs101report1.py` is a non-obfuscated file which they can navigate and debug using a debugger. The file may contain the homework, or it may call functions the students have written. Running the file creates console output which tells the students their current score for each test:
```
Starting on 02/12/2020 14:57:06
Evaluating CS 101 Report 1
......@@ -49,17 +44,67 @@ Once students are happy with the result, they run an alternative, not-easy-to-ta
```
python report1_grade.py
```
This runs the same tests, and generates a file `cs101report1.token` which they upload to campusnet. This file contains the results of the report evaluation, the script output, and so on.
This runs the same tests, and generates a file `cs101report1.token` which they upload to campusnet. This file contains the results of the report evaluation, the script output, and optionally local files from the users system for plagiarism checking. The `.token` file itself is in a binary but easily readable format.
## How to develop tests
The framework is build around the build-in `unittest` framework in python. Using the framework therefore also familiarizes students with the use of automatic testing.
To develop a new test, all you need is a working version of the students homework and the api will automatically created the expected output. This saves about half the work of creating tests and ensures tests are always in sync with the code.
A unittest normally consist of three things:
- The result of the users code,
- The expected result,
- A comparison operation of the two which may either fail or succeed.
As an examle, suppose the students write the code for the function `reverse_list` in the `cs101courseware_example/homework1.py` file. Our test script `cs101report1.py`, which was the same file the students ran before, contains code such as (we omitted one question for brevity):
```
class ListReversalQuestion(QuestionGroup):
title = "Reversal of list"
class ListReversalItem(QPrintItem):
l = [1, 3, 5, 1, 610]
def compute_answer_print(self):
from cs101courseware_example.homework1 import reverse_list
return reverse_list(self.l)
class ListReversalWordsItem(ListReversalItem):
l = ["hello", "world", "summer", "dog"]
The comparisons are build on top of pythons `unittest` framework to obtain a wide variety of well-documented comparisons, and it is easy to write your own.
class Report0(Report):
title = "CS 101 Report 1"
questions = [(ListReversalQuestion, 5), ] # In this case only a single question
pack_imports = [homework1] # Include this file in .token file
if __name__ == "__main__":
evaluate_report_student(Report0())
```
This code instantiates a group of questions and run two tests on the students code: one in which a short list of integers is reversed, and another where a list of strings has to be reversed. The API contains quite a lot of flexibility when creating tests such as easy parsing of printed output.
All that remains is to prepare the files which are distributed to the students. This can be done in a single line:
```
if __name__ == "__main__":
setup_grade_file_report(Report0)
```
This code will create two files:
- `cs101report1_grade.py`: This file contains all tests/expected output rolled into a binary tamper-resistant binary blob. Students run this file to generate their token files.
- `Report0_resource_do_not_hand_in.dat`: This contains the expected output of the tests (i.e., the students output is compared against what is in this file)
Both of these files, plus the actual tests `cs101report1.py`, are distributed to the students as is.
### Why is there a seperate `.dat` file and seperate test/grading scripts?
Ideally, tests should be a help in creating correct code, and therefore we want the publically-facing test code to be as easy to work with as possible.
For this reason the user test script, `homework1.py`, is completely transparent and does not include any `exec(...)` magic or similar. It simply
- Instantiates the Report class and loads the expected output from the `.dat` file (a simple, pickled dictionary with output)
- Execute the tests one by one in a for loop
Therefore, it will integrate well with a debugger and the homework could in principle be stated within the test script itself.
Transparency flies in the face of security, which is why there is an obfuscated `homework1_grade.py` file which bundles the expected output with the test code.
## Security/cheating
Any system that relies on local code execution is vulnerable to tampering. In this case, tampering will involve either changing the tests, or the `.token`file.
Both attempts involve figuring out what the main `_grade.py` does. The content of this file is a single line which executes a binary blob:
```
'''WARNING: Modifying, decompiling or otherwise tampering with this script, it's data or the resulting .token file will be investigated as a cheating attempt.
Note we perform manual and static analysis of the uploaded results.
'''
import bz2, base64
exec(bz2.decompress(base64.b64decode('QlpoOTFBWSZTWWIfYn8ABUHfgERQev9/9/v//+7/////YAZc...etc.etc.')))
```
If this blob is decompressed, it will contain badly obfuscated source code bundled with the result of the tests, and figuring this out involves some work and there are no possibilities for accidential tampering.
To get the expected result, one option (which is certainly possible) is to specify it yourself, however the recommended (and much easier option) is to maintain a working branch of the code with all the funtionality the students must implement and then **automatically** compute the expected output. In other words, we save the step of specifying the expected result, which also has the benefit we don't need to update expected output when the scripts change. The video contains a full working example of how a test is developed.
If the script is decompiled successfully, and the user manage to generate a tampered `.token` file, the `.token` file will still contain the source code. Since the source is collected at runtime, it will be possible to prove definitely that cheating occurred. This also adds an extra layer of security for code-copying (or simply copying someone elses .token file).
In light of this I think the most realistic form of cheating is simply copying someone elses code. This option is available for all homework, but a benefit of this system is we are guaranteed to have the code included in an easy-to-read format.
from unitgrade.unitgrade import QuestionGroup, Report, QPrintItem
from unitgrade.unitgrade_helpers import evaluate_report_student
from unitgrade_private.hidden_create_files import setup_answers, setup_grade_file_report
from cs101courseware_example import homework1
class ListReversalQuestion(QuestionGroup):
title = "Reversal of list"
......@@ -31,13 +31,10 @@ class LinearRegressionQuestion(QuestionGroup):
def process_output(self, res, txt, numbers):
return numbers[-1]
from cs101courseware_example import homework1
class Report0(Report):
title = "CS 101 Report 1"
questions = [(ListReversalQuestion, 5), (LinearRegressionQuestion, 13)]
pack_imports = [homework1] # Include this file in .token file
if __name__ == "__main__":
# setup_answers(Report0()) # hidden for students
# setup_grade_file_report(Report0,minify=True, bzip=True, obfuscate=True)
evaluate_report_student(Report0())
No preview for this file type
from unitgrade import cache_read, cache_write
import jinja2
import pickle
from tabulate import tabulate
from datetime import datetime
import bz2
# from tabulate import tabulate
# from datetime import datetime
# import bz2
import inspect
import json
# import json
import os
data = """
......@@ -89,9 +89,7 @@ def setup_grade_file_report(ReportClass, execute=True, obfuscate=False, minify=F
time.sleep(0.2)
with open(output, 'r') as f:
sauce = f.read().splitlines()
wa = """WARNING: Modifying, decompiling or otherwise tampering with this script, it's data or the resulting .token file will be investigated as a cheating attempt.
Note we perform manual and static analysis of the uploaded results.
"""
wa = """WARNING: Modifying, decompiling or otherwise tampering with this script, it's data or the resulting .token file will be investigated as a cheating attempt."""
sauce = ["'''" + wa + "'''"] + sauce[:-1]
sauce = "\n".join(sauce)
with open(output, 'w') as f:
......@@ -104,10 +102,6 @@ def setup_grade_file_report(ReportClass, execute=True, obfuscate=False, minify=F
s = os.path.basename(fn)[:-3] + "_grade"
exec("import " + s)
if __name__ == "__main__":
from unitgrade.example.report0 import Report0
setup_grade_file_report(Report0, execute=False)
# if __name__ == "__main__":
# from unitgrade.example.report0 import Report0
# setup_grade_file_report(Report0, execute=False)
......@@ -27,16 +27,12 @@ def gather_upload_to_campusnet(report, output_dir=None):
results['sources'] = {}
json_str = json.dumps(results, indent=4)
now = datetime.now()
# now = datetime.now()
# dname = os.path.dirname(inspect.getfile(report.__class__))
# dname = os.getcwd()
if output_dir is None:
output_dir = os.getcwd()
# raise Exception( dname )
#
# print(dname)
payload_out_base = report.__class__.__name__ + "_handin"
obtain, possible = results['total']
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment