repomate-junit4 0.5.0

Description:

repomatejunit4 0.5.0

repomate-junit4, a JUnit 4.12 test runner plugin for repomate






This is a plugin for repomate that runs
JUnit4 test classes on production classes in a cloned student repo.
Requirements
repomate-junit4 has a few non-Python dependencies.

java must ba available from the command line.
javac must be available from the command line.

In other words, install a JDK version that is compatible with the files
you intend to test!


junit-4.12.jar must be available on the CLASSPATH variable, or configured
(see Added CLI arguments and
Configuration file).
hamcrest-core-1.3.jar must be available on the CLASSPATH variable or
configured in order to make use of hamcrest matchers.

The hamcrest and junit jars ar available from Maven Central:
wget http://central.maven.org/maven2/junit/junit/4.12/junit-4.12.jar
wget http://central.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar

Install
The recommended way to install repomate-junit4 is with pip.
python3 -m pip install --user repomate-junit4

The plugin itself does not actually require repomate, but it is fairly
useless without. If repomate and repomate-junit4 are both installed in the
same environment, then repomate should just find repomate-junit4.
For repomate to actually use repomate-junit4, it must be configured
in the repomate configuration file. Refer to the
repomate docs
for information on the configuration file and its expected location, and the
Configuration file section here for info on what you
need to add to it.
Usage
Terminology and conventions
This is terminology added in excess to that which is defined in the repomate
docs.
For brevity, some conventions expected by repomate-junit4 are baked into
these definitions.

Production class: A Java file/class written in the student repo.
Test file: A file ending in Test.java which contains a test class for
some production class. If the students are supposed to write a file called
LinkedList.java, the corresponding test class must be called
LinkedListTest.java.
Test dir: A directory named after a master repo, containing tests for
student repos based on that master repo. Should contain test files
as defined above.
Reference tests directory (RTD): A local directory containing subdirectories
with reference tests. Each subdirectory should be a test dir as defined above.

Security aspects
There are some inconvenient security implications to running untrusted code on
your own computer. repomate-junit4 tries to limit what a student's code can
do by running with a very strict JVM
Security Policy.
This is enforced by the Java
SecurityManager.
The policy used looks like this:
// empty grant to strip all permissions from all codebases
grant {
};

// the `junit-4.12.jar` needs this permission for introspection
grant codeBase "file:{junit4_jar_path}" {{
permission java.lang.RuntimePermission "accessDeclaredMembers";
}};
"""

This policy disallows student code from doing most illicit things, such as
accessing files outside of the codebases's directory, or accessing the network.
The {junit4_jar_path} is dynamically resolved during runtime, and will lend
the actual junit-4.12.jar archive that is used to run the test classes
sufficient permissions to do so.
This policy seems to work well for introductory courses in Java, but there may
be snags because of how restrictive it is. If you find that some permission
should definitely be added, please
open an issue about it
There are plans to add the ability to specify a custom security policy, but
currently, your only choice is to either use this default policy or disable it
with --disable-security.

Important: The security policy relies on the correctness of the Java
SecurityManager. It is probably not bulletproof, so if you have strict
security requirements, you should probably only run this plugin inside of a
properly secured environment (for example, a virtual machine).

Added CLI arguments
repomate-junit4 adds four new CLI arguments to the repomate clone command.

-rtd|--reference-tests-dir

Path to the RTD.
Required unless specified in the configuration file.


-junit|--junit-path

Path to the junit-4.12.jar library.
Required unless specified on the CLASSPATH variable, or in the
configuration file.


-ham|--hamcrest-path

Path to the hamcrest-core-1.3.jar library.
Required unless specified on the CLASSPATH variable, or in the
configuration file.


-i|--ignore-tests

A whitespace separated list of test files (e.g. LinkedListTest.java) to
ignore. This is useful for example if there are abstract test classes in
the test dir.


--disable-security

Disable the seurity policy.


-v|--verbose

Display more verbose information (currently only concerns test failures).
Long lines are truncated.


-vv|--very-verbose

Same as -v, but without truncation.



Configuration file
First and foremost, junit4 must be added to the plugins option under the
[DEFAULTS] section in order to activate the plugin,
see details here.
The --hamcrest-path, --junit-path and --reference-tests-dir arguments can
be configured in the standard
repomate configuration file
by adding the [junit4] section heading. Example:
[DEFAULTS]
plugins = junit4

[junit4]
reference_tests_dir = /absolute/path/to/rtd
junit_path = /absolute/path/to/junit-4.12.jar
hamcrest_path = /absolute/path/to/hamcrest-core-1.3.jar


Important: All of the paths in the config must be absolute for
repomate-junit4 to behave as expected.