Automatic grading of Java and Scala homework assignments

Many students practice object-oriented or functional programming early on by independently implementing homework assignments. The huge number of participants and diverging approaches to solving problems make it difficult for lecturers to grade homework assignments (often exam requirements) according to a uniform standard.
That is why we developed an extension of JUnit in 2013 (at that time based on Java-1.7, JUnit-4, and Scala-2.12), the source code of which we publish at https://github.com/FAU-Inf2/AuDoscore (Java) and https://github.com/FAU-Inf2/ScExFuSS (Scala). Annotations assign test cases a bonus or penalty score. The results of the test execution are recorded and used to calculate a total score fully automatically. The evaluation is carried out in four stages, each of which provides detailed feedback immediately if necessary.
In 2025, we completely redesigned AuDoscore and ScExFuSS after key components became unusable due to the abrupt evolution of Java, JUnit, and Scala, which could no longer be kept running through constant adjustments. Since Java-25, the SecurityManager has been disabled as a security infrastructure. The severe restrictions imposed on the Java compiler API rendered it unusable for our purposes. Due to syntactic changes to the source and byte code, the previous pattern-based problem detection became non-deterministic. Newer JUnit versions have fundamentally different extension mechanisms (that are incompatible with the old ones).
Among others, this raised the following questions:
- How can we reliably prevent students from (un)intentionally disrupting the assessment system itself (previously through SecurityManager)?
- How can we detect when students use explicitly prohibited API functions (declared @Forbidden/@NotForbidden annotations)?
- How can we take into account in the assessment that students implement functions that build on each other incorrectly (consequential errors)?
- How can we integrate AuDoscore and ScExFuSS into the latest JUnit infrastructure?
To solve those problems in AuDoscore, we now use the "Classfile Package" from the Java-25 API. As a replacement for the SecurityManager and to implement the "@[Not]Forbidden" annotations, we use it to directly examine the bytecode for dangerous or prohibited function calls. To avoid consequential errors, we transplant classes, methods, or fields from the bytecode of the sample solution into the student's solution. This involves dealing with many difficult special cases (e.g. due to "type erasure", "lambdas", and many more), for which we may also transfer parts of the bytecode that are not directly in the code block of the method to be replaced and retranslate the tests appropriately for each test case.
To solve the above questions in ScExFuSS, we currently use the TASTy files (Typed Abstract Syntax Trees) generated by the compiler using the "Scala-3 Tasty Inspector". As a replacement for the SecurityManager and to handle the "@[Not]Forbidden" annotations, we statically check which functions are actually used. The consequential error handling based on the TASTy files is now also available for Scala for the first time.
For the purpose of migrating to JUnit-6, we ported AuDoscore, ScExFuSS, and all tests to JUnit-Jupiter. The "hooking" into the entire test execution process and the logging of evaluation events was implemented from scratch. As a result, we also updated all existing self-tests and added new ones to ensure that all changes and all new language features of Java-25, Scala-3, and JUnit-6 are handled correctly.

Publications