2.1.   Student summary

Clicking on the “Summary” tab in the assignment dashboard menu leads to the “Student summary” page. This page allows you to view information about any Student in the course in regards to the currently selected assignment.

Initially, there will not be much information available here if the assignment is not yet public, or the Student hasn’t yet opened the assignment. Here we can see their name, email address, and section. Three buttons are available.

Partner(s): Allow solo

If the assignment has teams enabled, then there will be a “Partner(s)” table entry and an “Allow solo” button. This “Allow solo” button gives Teachers the ability to override the team membership minimum imposed on the Students, if Students are the ones forming the teams.

For CS 15900…

Typically only “lab” assignments will have teams enabled, whereas homeworks and exams will be individual-only. The lab assignments will have a minimum team membership of two so that Students cannot start on their own without at least one partner.

Grade: Override

The “Override” button in the “Grade” table entry allows the Teacher to set a sticky score for the Student.

Caution!

Once set, this sticky score cannot be changed by either the auto-grader, nor by the teaching assistants.

LTI: Push score

“LTI” stands for “Learning Tools Interoperability”, and is the standardized specification of how external learning tools like Vocareum communicate with learning management systems like Brightspace. The “Push score” button allows the Teacher to manually send a score from Vocareum to Brightspace. You can also push scores for an entire assignment through the export menu.

Once Students have opened their workarea but not yet made a submission, the summary will look something like this. There are several new buttons here.

Partner(s): Add a partner

You can add any Student who has not yet become part of a team to this team. This overrides the team size that you specify in the basic assignment options.

Partner(s): Dissolve team

After asking for confirmation, this unlinks the team and destroys their shared workspace.

Note

This button is only available if the team has not yet made a submission. Once a submission has been made, the team cannot be dissolved, and thus its members cannot be moved to a different team, except by contacting Vocareum support.

Status: Extend submission window

You can override the deadlines and/or allowed number of submissions for each individual, or for each team on team assignments.

Status: Force submit: [workarea]

This acts as if the Student clicks the “Submit” button from their view of the assignment.

For CS 15900…

This feature may cause problems if the Student’s current code fails to compile or is missing their email address, and so on. Avoid using this feature if possible.

Status: [upload]

You can upload files that will act as a Student’s submission.

View: Activity

An audit log of every user’s interaction with the Student or team’s assignment. This will be a drop-down menu if there are multiple parts to the assignment.

View: Workarea

Show the view the Student would see of the workarea for the assignment, essentially allowing you to “impersonate” them. This will be a drop-down menu if there are multiple parts to the assignment.

View: Submissions

Show what a teaching assistant would see for the given assignment.

Grader: Add

Give an additional teaching assistant the ability to grade the assignment. Once a Grader has been added, there will also be an “Add more” button and a “Remove” button for each Grader. The “Remove” button may be missing if the Grader’s access to the course has been disabled in the class enrollment manager.

Grade: Team override

Set a sticky score for the entire team.

Grade: Individual override

Set a sticky score for an individual team member.

Caution!

As before, these sticky scores cannot be changed by either the auto-grader, nor by the teaching assistants once they are set.

Once a submission has been made and the auto-grader has been run, there will be an additional button:

Status: Reset auto-grade flag

Every time the auto-grader runs in Vocareum, it marks the assignment as having been auto-graded already. This button allows an individual assignment to be auto-graded again during the next batch.

Note

If the flag has already been reset, or if the auto-grader has not been run yet, this button will not be visible.

You can also reset auto-grade flags for all submissions of a given assignment at once.

2.2.   Submissions tables

Clicking on the “Submissions” tab in the dashboard sub-menu (with the silver background) lists the status of all submissions made for the assignment.

This tells us information such as:

• whether the Student or team has opened the workarea yet (“Started”)
• if the last submission (if any) was considered late or not
• the section of the Student or team, if applicable
• the assigned Grader (teaching assistant) for the Student or team, if one has been assigned yet
• the final grade for the assignment, if one has been given
• and whether any feedback has been left by a Grader or Teacher
• how many parts of the assignment they have submitted, if there is more than one part
• the number of parts that have been graded, if there is more than one part

The last column (“View”) is a link to open the grading view for the submission.

The view looks slightly different if the assignment has multiple parts.

Namely, the submission lateness column is replaced with a column indicating how many of the assignment’s parts have been submitted, and the feedback column has been replaced with a column (moved to the left after the Grader column) indicating how many of the parts have been graded.

There is also another tab named “Submissions Jobs”. This lists when each execution of the submission script started, ended, and how much time elapsed.

The “Auto-Grade” tab gives similar information for each execution of the grades script.

2.3.   Graders table

The “Graders” tab in the assignment dashboard gives a similar listing of each Grader’s progress in assessing their Students’ submissions. The “Weight” column can be used to calculate a weighted average grade if multiple Graders are assigned for a single Student or team.

2.4.   Assignment controls

The “Controls” tab (with the gear icon) in the assignment dashboard gives access to several possible actions for managing the currently selected assignment.

2.4.1.   Assign Graders

This option lets the Teacher assign a Grader for every submission that has been made to the assignment so far. You can randomly assign Graders based on sections, from a CSV file, reuse the same pairings as the last time this action was taken, or randomly assign them irrespective of sections.

The format of the CSV file is each Student’s email in column one, followed by each Grader’s email in column two.

For CS 15900…

Since only one Grader will be assigned to each section, you should always choose “Random - section based”.

2.4.4.   Auto-grade

This starts a new execution of the grades script. Submissions are included in the batch if:

• they have not yet been auto-graded, or
• if they have had the auto-grade flag reset by a Teacher.

You will be prompted to confirm this is what you want to do. If there are multiple parts to the assignment, you can also select which parts to auto-grade (each part can have a different grading script).

Once you have confirmed, the window will continue to update as the batch is processed, or you can wait for the results to go to your Vocareum messages, which looks like the following:

2.5.   Assignment exports

There are a variety of different reports you can generate to get detailed information about the currently selected assignment.

2.5.1.   Export grades

You can export reports on Students’ grades for the currently selected assignment as a CSV file. Values with spaces are quoted. Values with multiple lines have embedded carriage return and newlines embedded as literals.

With all options turned off, the format is the following:

#	Name	Description
01	User	The Student’s full name in Vocareum.
02	Section	The section name of the Student, if any.
03	Score	The total score on the assignment for the Student. If no score has been given yet, it will be a series of dashes (--------).

Students dropped from the course on Vocareum are not included; neither are Students who have not yet made a submission.

Show email

Inserts the Students’ email addresses as a new column after “User” named “Email”.

Name

Seems to be a duplicate of the “User” column and should be turned off.

Parts

Adds a column for each assignment part, giving the total score for that part.

Details

Adds the following columns:

Name	Description
[part name.]criterion name	The current score for this criterion.
Max ([part name.]criterion name)	The maximum score for this criterion, based on all previous submissions for this Student.
[part name: ]Review Comments	Feedback left by Graders or Teachers.

All assignments

Include the relevant columns for every assignment in the course in each Student’s row. The column names are prepended with the assignment’s name followed by a period, but the total score column for each assignment is named Score: assignment name instead. The overall score for the class is given in the Score column.

2.5.2.   Export submission report

You can export a CSV file that includes information on each Student that has made a submission to the assignment.

Enabling option	Name	Description
Show ID	ID	Vocareum configurable ID for the Student.
Show e-mail	email	Student’s email address.
Show name	name	Student’s name.
Late flag(Y/N)	late	Y if considered late, otherwise N.
Start date-time	start date-time	Whenever the Student first attempted to open the assignment, e.g. Jul-15-2021 7:59:25 pm EDT.
Submission date-time	last submission date-time	e.g. Jul-15-2021 8:15:30 pm EDT.
Always enabled	submission count	Number of submissions the Student has made.

If all options are disabled, then only the submission counts are returned with no identifying Student information.

The drop-down lets you include “all Students”, or filter by “late” or “on time” Students only.

2.5.3.   Export submitted assignments

The latest submission made by each Student for this assignment is collected into a ZIP file. The directory structure is email/partname/submission_N where N represents it being the Nth submission by the Student. All files not starting with a period (.) in the Student’s workspace at the time of submission are included in the submission folder, as well as files saved by the submission or grades scripts, or by Vocareum. This includes:

Name or extension	Description
*.compile	Output captured from the compiler at submission time.
*.elf	Executable created by the compiler at submission time (instead of a.out).
submit.rst	Submission report in a plain-text format. This is essentially the same contents as the submission receipt email that gets sent to Students.
.vocStudentSubmissionReport.txt	Identical to the above, but contains the starting execution time (in Pacific Time) of the submission script.
.vocStudentSubmissionReport.html	Submission report in an HTML format.
grade.rst	Grade report in a plain-text format, as generated by the auto-grader.
.vocStudentGradingReport.txt	Identical to the above, but contains the starting execution time (in Pacific Time) of the grading script.
.vocStudentGradingReport.html	Grade report in an HTML format.

Two (mutually exclusive) options are supported:

All Students

Also create empty folders for the Students who have not submitted.

De-identified folders

Instead of using the Students’ email addresses as folder names, use random strings like f73ce498bb2195d094f142a03f7d9697.

Once submitted, the ZIP file will appear in your Vocareum Messages (mail icon).

Note

There appears to be some bugs with this feature. Namely, some Students are mistakenly included when they should not be (e.g. they may be included if they have not even started, let alone submitted, if “all Students” is enabled); or, they may be excluded when they should be included (it seems if the Teacher is also a Student, submissions from that Student account are not included).

2.5.6.   Push grades through LTI

You will be prompted with a screen like the following which shows the score that will be sent to Brightspace for each Student. Click the orange “Push scores to LTI client” button to send the scores to Brightspace. If any errors occur (such as sending a score for a dropped Student), you will be sent a Vocareum Message.

You can also push a score for an individual Student.

3.1.   Basic class settings

From Vocareum’s documentation on basic course settings:

Title

The name of the course that all the users will use to select a course.

Course image

An optional image displayed on the “Card View” home page for this course.

Timezone

This will be used for all the settings when time is needed like due date for submissions.

Note

This should always be set to Eastern Time to ensure due dates are in alignment with Purdue’s local time. Regardless, some areas of Vocareum may still report times in Pacific Time since that is where Vocareum is based. This is mostly contained to things like batch script execution reports and other things not visible to Students.

End date

This is the date when the course ends. As a Teacher you can not change this date. After this date Students will no longer be able to work on their assignments. As a Teacher you will continue to have access to the course.

Access end date

While the Students can no longer work on their projects after the “End Date”, they can continue to access their data until this date. You can change this date.

3.2.   LTI/LMS settings

LTI (Learning Tools Interoperability) should always be enabled so that Brightspace (the LMS) can communicate with Vocareum and vice versa. For now, the LTI version should always be v1.1, Auto create sections should be disabled (pending a future update that will allow this functionality to work), and Send total score (Canvas) should be disabled since we do not use Canvas.

This leaves Show all assignments which is left to your choice. When it is enabled, Students can see every assignment that is published on Vocareum at any time. If it is disabled, Students can only view one assignment at a time, and only via direct links or through links from Brightspace. In the latter configuration one would only need worry about the visibility of assignments on Brightspace, but the former configuration makes it easier for Students to view their past work and feedback without needing to return to Brightspace.

3.3.   Skilljar settings

3.4.   Course parameters

Sections should be enabled. You can add more sections here or rename existing ones. Note that once sections are enabled, they cannot be disabled. The enrollment manager allows you to add Students to specific sections as well as assign Graders to each section.

3.5.   Other class settings

4.1.   Adding Students

Typically you will not need to add more Students manually since their account is automatically created when they click the assignment links in Brightspace. However, this page allows you to add more Students, and view or edit the existing ones.

At the top of the screen you can add new Students manually. You are required to enter at least an email address and a section. The name, while optional, should be entered as it is the only visible identifier for the Student in some areas. The ID column may be auto-populated if the Student was enrolled through Brightspace, but it is not used by any of the CS 15900-specific configuration.

You can also upload a CSV file to enroll multiple Students at once. Include a header in row 1 that has at least the following values as columns:

• Email
• Section

You can optionally add one or both of these columns as well:

• Name
• ID

The columns can be in any order that you specify.

If you are using the CSV upload feature, click the big blue “Enroll” button above the table to upload and process the CSV file.

The table of Students shows information for every Student in the class. You can click the checkbox “Hide dropped Students” to exclude Students you’ve dropped from the Vocareum class from this table. The totals are included in the table heading. Clicking “Section details” will reveal the number of enrolled Students for each section of the class.

For each Student, you can choose to either “Edit” their information, “Drop” them from the Vocareum class, or “Enroll” them back into it.

Clicking the “Edit” button shows the following menu, where you can edit any of the Student’s information, including their section membership.

Clicking the “Click here to load” yellow box will load a log of all emails ever sent by Vocareum to the Student. At the moment, these are just enrollment emails. By clicking the orange “Resend Email” button, you can re-send an enrollment email to the Student. This email contains the name of the course, a link to the login page, and a password that can be used to login to Vocareum directly instead of clicking links in Brightspace.

4.2.   Adding Graders

Teaching assistants are enrolled into your Vocareum class as “Graders”, which allows them to only view submissions to which they’ve been assigned to grade, the ability to override grades from the auto-grader for those submissions, and leave feedback for the Students.

At the moment, there is no Brightspace link to enroll Graders automatically, so you will have to use this page - unlike the Student enrollment manager.

Again, you can either enter the Grader information manually one at a time, or upload a CSV file. The only mandatory information is the Grader’s email address (“Email” column), but you can also include a “Name” column and a “Section” column. Note that unlike Students, Graders don’t have to belong to any section - in which case they default to having access to all sections.

If you are using the CSV upload feature, click the big blue “Add” button above the table in order to upload and process the CSV file.

You can click “Disable” to disable a Grader’s ability to access the course, which will take effect immediately. An “Enable” button will take its place and allow you to undo this change.

You can also change the Grader’s details, including their section assignment, using the “Edit” button.

This only allows you to add another section to the Grader’s load. In order to remove a section, click the blue “X” next to the section name in the Grader’s table entry.

You can also click the orange “Resend Email” button to send a welcome email in a similar fashion to Students as mentioned earlier.

4.3.   Adding Teachers

Likewise, you can add other Teachers to the course. They will have the same access to the entire course as you do. You cannot limit Teachers to specific sections.

As before, you can either enter the new Teacher information manually one at a time, or upload a CSV with emails (and optionally names). Click the big blue “Add” button above the table to upload and process the CSV file if you select one.

Clicking “Remove” on any Teacher will disable their access to the course - but only as a Teacher (they may still have access as a Grader or Student, and vice versa for dropping Students or disabling Graders). This will take effect immediately, but you can click the blue “Add” button in their table entry to reinstate their access at any time.

Editing a Teacher allows you to change their name or email address.

You can also click the orange “Resend Email” button to send a welcome email in a similar fashion to Students as mentioned earlier.

5.1.   Assignment settings

Across the top of the screen are three blue buttons.

Save

You must click “Save” after making any changes to the assignment.

Warning

You will lose any unsaved changes if you navigate away from the page without saving, which includes opening part settings!

Your changes will take effect immediately. If the assignment is currently published, you will be asked to confirm that you want to save your changes first.

Publish or unpublish

This changes the visibility of the assignment for Students only. Graders always have access to submissions to which they’ve been assigned for an assignment, regardless of the assignment’s visibility. Also note that this does not affect visibility of anything on Brightspace.

Your change will take effect immediately.

Configure workspace

This opens the assignment workbench which is where you can upload your solution program, define test cases, and change other auto-grader settings.

5.1.1.   Assignment information

The section below that lets you change the assignment name, its position on the assignment list, and add an optional description that is shown to Students before they enter their workarea.

5.1.2.   Assignment parts

The table below the assignment information settings lists all of the “parts” of an assignment. Each part can be submitted and graded independently from one another, and the overall grade for the assignment can either be the sum of the parts’ grades, or the maximum grade obtained on any of the parts.

The blue link for the part’s name takes you to the part settings for that part. The actions you can take on each part in this table include:

• The “info” icon (circled lowercase “i”) opens a pop-up that shows the Vocareum-internal identification numbers for the course, assignment, and part.

• The “chain link” icon opens a pop-up that shows the direct link for the assignment. You typically won’t need this since you will be linking the assignments through LTI on Brightspace.

• The “trash can” icon allows you to delete a part from the assignment. You will be asked to confirm this. This action will take effect immediately afterwards, even without clicking the “Save” button.

  Note

  There appears to be a bug in Vocareum where the names of deleted assignment parts cannot be used again as a name for a part in that particular assignment.

• The “up/down” icon lets you rearrange parts visually if there is more than one.

You can add a new part using the gray button with a plus sign (“+”) in it, if the assignment is unpublished. If it is currently published, you must first click the blue “Unpublish” button at the top of the screen before this “new part” button becomes visible.

5.1.3.   Basic assignment options

Publish grades

This makes the grades for the entire assignment visible to Students on Vocareum (it does not effect visibility of grades on Brightspace). You can also toggle grade visibility using the publish/unpublish grades option in the “controls” menu of the assignment dashboard.

Team project

Enables multiple Students to work on the assignment at the same time and receive the same grade for it.

For CS 15900…

All homework assignments in our class are individual, while all lab assignments are “team projects”.

Selecting this checkbox reveals a blue button that reads “Team assignment setup”, which opens the following pop-up.

This minimum and maximum are the limits for Student-created teams. Students cannot form a team on their own with less than the minimum amount of Students specified here (including themselves), or more than the maximum.

The Teacher has the ability to override these limits in several ways:

• Allowing particular Students to work alone.
• Add partners yourself, optionally in excess of the limit imposed here.
• Dissolve a team if nobody on the team has opened the workarea yet.
• Upload teams CSV before the assignment is published to force Students to be in particular teams.

Section based timelines

Set different deadlines for each section. You can manually set these using the interface, or upload a CSV file. Each of the deadlines is optional.

The format of the CSV file is the following. You can leave columns empty if they are not applicable.

#	Name	Description
01	Section name	Name of the section for which these deadlines should apply.
02	Submission deadline	Students must submit the assignment before this date and time in order to be considered an “on time” submission, subject to the grace period setting.
03	Early submission deadline	If Students submit before this date and time, they get some bonus points added to their final score.
04	Early submission points	The number of points to add if the submission is made before the early submission deadline.
05	Late submission	If enabled, Students can continue to submit the assignment after the submission deadline up until the late submission deadline, subject to the grace period setting. There will be a deduction from their final score according to the late penalty settings.
06	Access date	Restrict the ability to open the assignment until after this date and time passes.

These features have not yet been investigated.

The rest of the options should be disabled:

• Peer review
• Leaderboard
• Auto-submit: in particular, this could be problematic for CS 15900 as a program that fails to compile might be submitted and taken as the last submission which would jeopardize Student scores.

5.1.4.   Assignment LTI

This checkbox controls whether the currently selected assignment can be added as an external learning activity in Brightspace or not. When LTI is active, Students must access the assignment through Brightspace at least once - the direct link from the assignment parts table will not work, even if they have already been enrolled as Students in the class.

Once LTI is enabled, you need to go to Brightspace in order to link the assignment. On a content module, click “Existing activities”, then “Vocareum”.

You will be presented with a list of Vocareum assignments. Only assignments with LTI enabled will be show in this list. For each assignment, you can either choose to embed the Vocareum page inside of Brightspace (iframe), or open the page in a new tab.

Caution!

Even if an assignment has already been linked into Brightspace, it will still appear in this menu, allowing you to possibly link the same assignment multiple times. It is unknown as to what happens if this occurs.

Note

Linking a Vocareum assignment into Brightspace will not change the assignment’s visibility on Vocareum. You must publish the assignment in the assignment settings (click the blue “publish” button at the top of the page) in order for it to actually be accessible by Students.

Note

There appears to be a bug with either Vocareum or Brightspace at the moment where the “new tab” button does not work, instead behaving exactly the same as the iframe button which forces new assignments to always be embedded inside of a Brightspace page.

To work around this, first click the arrow next to the assignment’s name in the Brightspace content module, then select “edit properties in-place.”

Then, click the checkbox that says “open as external resource”. Your changes should be saved immediately.

5.1.6.   Advanced assignment options

There are some more settings hidden under the “advanced” headline. Click it to reveal them.

Passcode protect

Students can not view the assignment at all until they enter the password you specify here. Once Students have entered the password, however, they cannot be locked out again.

Caution!

In particular, this means it is useless to add a password after Students have already opened their workarea for the first time, as it will not apply to them.

Grading visibility

You can choose between “assigned submissions” (recommended) or “all submissions”, which shows a new tab in the Grader interface with every Student’s submission for a given assignment.

You can assign Graders in the assignment controls, or add a Grader to a specific Student in the Student summary, both located in the assignment dashboard.

Grading visibility: anonymous

Hide the Student’s names from the Graders. Instead, only their Vocareum-internal user identification number will be shown.

Warning

This has no effect on the contents of the submission, so if any of the files in the submission (whether created by the Student, the submission script, or the grading script) contains the Student’s name or email address, the Grader will be able to see that!

Copy startercode

All of the files in the resource/startercode/ folder will be copied into the Student’s workspace when it is first created.

Auto-grade on submit

Run the grading script immediately after each submission is made. Grades will still not be visible until you use the publish/unpublish grades control in assignment dashboard or toggle the “publish grades” option in the basic assignment options.

Grace period

If enabled, Students can still submit and be considered “on time” with respect to the submission deadline or late submission deadline, as long as the grace period has not yet elapsed. You can configure it to be between one minute and 60 days, and have it only apply to the submission deadline or both the submission deadline and the late submission deadline.

Aggregation rule

Determines how the assignment’s total score is calculated based on the scores from the assignment parts. You can either add them all up (“sum of all parts”), or select the best score on any of the parts (“maximum score”).

Some of these features have not yet been investigated.

As such, they should be left disabled or with their default values.

• IP address range
• Generate startercode
• No submission
• No workarea

5.2.   Part settings

Click on one of the part names in the assignment parts table. This opens the part settings page. Similarly to the assignment settings, there are three blue buttons across the top.

Save part

Save any changes made to this part’s settings.

Warning

You will lose any unsaved changes if you navigate away from the page without saving, which includes returning to the assignment settings!

Publish

Same as in the assignment settings - change the visibility of the assignment on Vocareum.

Configure workspace

Same as in the assignment settings - open the assignment workbench.

Above the “save part” button is a left arrow (<) button that will allow you to return to the assignment settings.

5.2.1.   Basic part options

Below the blue buttons are the most basic settings for a part, including the part’s name and type of Vocareum setup.

Part name

The name of this assignment part. When there is more than one part, these names become visible to Students.

Note

Part names should not contain spaces, as it seems there are bugs in Vocareum that will consider names containing spaces as equivalent. Each part must have a unique name.

Lab type

The type of Vocareum interface to use for this part. The options available are:

• Vocareum Standard
• Vocareum Basic
• Vocareum Elite
• Jupyter Notebook
• RStudio
• MySQL Workbench
• Thonny
• Desktop
• Container Lab
• VM Lab
• RStudio Elite
• Jupyter Elite
• Skulpt Basic

For CS 15900…

Always select “Vocareum Elite”.

Terminal

There are two options here: “Terminal v1” and “Terminal v2”. “Terminal v1” is the terminal used (regardless of the part settings) when inside the assignment workbench. It consists of dark text on a light gray background, with a dark blue bar across the top.

The “+” icon allows a new tab to be opened, with the currently selected tab denoted by a white dot (”•”), and other tabs denoted by gray dots. The terminal starts with only one tab by default.

Output from the submission script will appear in this terminal when a Student clicks the “Submit” button.

“Terminal v2” has white text on a black background, and no tab interface. Output from the submission script does not appear in this terminal.

5.2.2.   Lab resources

Note

The settings here depend on the choice of “lab type” in the previous section. We will only cover the settings applicable to the “Vocareum Elite” lab type.

This section configures the workarea in which Students will complete their assignments, as well as the machine on which submission and grading will occur.

Database

Whether the workarea will have access to a database server of some kind or not. For more information, consult the Vocareum documentation on databases.

Spark cluster

Give the workarea access to machines that have Apache Spark and Hadoop installed on them. For more information, consult the Vocareum documentation on clusters.

Interactive server type

The operating system to use for the interactive workareas; in particular, this is the operating system with which Students will be directly interacting.

Batch server type

The operating system to use for batch processes like submission and grading. This should match the option chosen for the interactive server type for consistency.

5.2.3.   Lab interface

Note

The settings here depend on the choice of “lab type” in the previous section. We will only cover the settings applicable to the “Vocareum Elite” lab type.

These settings control the Vocareum GUI for the Student workarea.

Panels: Source

A text editor panel similar to the one shown in the assignment workbench. You can control the settings of this text editor for the class as a whole in the other class settings.

Panels: Console

The interactive terminal panel. The terminal type is configured in the basic part options.

Panels: File browser

A file tree pane similar to the one shown in the assignment workbench, but with the resources/ folder hidden and a Submissions/ folder added that shows all of the Student’s previous submissions.

Panels: Nav bar

Show or hide the Vocareum main menu.

Controls: Reset

Include a button named “Reset” that will confirm if Students want to wipe their workarea clean and start again from scratch.

Information: Details button

This option does not appear to change anything for the “Vocareum Elite” lab type.

Information: README button

Include a button named “README” that will display the README file from $ASNLIB/public/docs/ folder in a pane. You can choose from .txt, .pdf, .html, and .md file formats, and even create READMEs in multiple languages by putting them in $ASNLIB/public/docs/lang/$language/. For more information, consult the Vocareum documentation on READMEs.

Information: Assignments

This option does not appear to change anything for the “Vocareum Elite” lab type.

5.2.4.   Part submission

Due date

Similar to the assignment’s submission deadline, but for this part only. See section based timelines.

Late submission

Allow submission after the due date for this part, subject to an optional penalty.

Early submission bonus

Allow submissions made before this day and time for this part to earn a fixed amount of bonus points on their final score.

Number of submissions

You can restrict the maximum amount of submissions a Student can make for this part to be anything between 1 and 600 submissions, or an unlimited amount.

For CS 15900…

Failed submissions (e.g. program doesn’t compile, missing email address, etc.) would count towards this limit.

Submission interval

The minimum amount of time that must elapse before a Student can submit their assignment again. Setting it to zero hours and zero minutes means there is no limit to how often the Student can submit.

Show solution

This feature has not yet been investigated.

You can choose to publish files from $ASNLIB/solution immediately, after a fixed number of submissions (1 to 30), after the submission deadline, or after a custom “solution publish date”.

What is not known is whether these files would be viewable in our configuration of the Student workarea.

For more information, consult Vocareum’s documentation on showing solutions.

5.2.5.   Part grading

Late penalty rule

You can choose a percentage of either the Student’s score or the maximum total score to deduct from their final score, or an exponential deduction.

Late penalty type

Either fixed (one deduction only), or a compounding deduction that occurs every day, hour, or minute after the submission deadline.

Grading rule

Either the latest submission from each Student, or the maximum score that each Student attains across any of their submissions.

Max. grading CPU time

The maximum amount of time (in seconds, up to a limit of 1800 seconds which corresponds to 30 minutes) to spend running the entire grading script for this part for each Student’s submission. The default of 300 seconds corresponds to 5 minutes.

Generate scripts

Have Vocareum generate a grading script based on the grading criteria you specify below.

Grading criteria

These are the rubric items that are used to break down the Student’s score. The score for each criterion can be freely manipulated by both the auto-grader and human Graders.

You must specify a name and a point value. The sum of the point values for all criteria yields the maximum possible score for the assignment part (shown here, that would be 10 points).

The “autograde” checkbox is only relevant if you have “generate scripts” (above) enabled, so leave it disabled.

The “exclude” checkbox tells Vocareum to exclude this criterion from calculating the overall score and maximum possible score for the part.

The red “X” icon lets you remove a criterion.

The double-headed arrow icon allows you to rearrange the order the criteria will appear in when viewed by Graders and Students.

5.3.   Assignment workbench

Clicking the “configure workspace” button at the top of either the assignment settings page or the part settings page will open the “assignment workbench”.

At the top left of the screen is the name of the assignment, which you can click to return to the assignment settings. If there are multiple parts, you will see the part name and select a different part on which to work.

To the right hand side of the screen are various controls that you can use.

Generate quiz

This feature has not yet been investigated.

Vocareum allows you to create multiple-choice or multiple-answer quizzes using a special Vocareum quiz Markdown flavor. You can read more about how to use this feature on Vocareum’s documentation on releasing quizzes.

View: Jupyter

This is only useful if the Vocareum lab type involves Jupyter. If you accidentally click this option, click “View: Standard” to return to the normal workbench view.

Student view

Open a demonstration Student workarea for the current assignment.

Update

When you make changes to most of the files using the workbench, those changes will not take effect immediately. Click this button in order to push your changes to all Students.

There will be an asterisk if Vocareum thinks you’ve made changes that need to be pushed, but note that changes made through the terminal are not usually detected.

Build image

Create a Docker container image based on the contents of the workbench. This is only useful if lab type involves containers.

Run scripts: Grading

Simulate a run of the grading script based on the contents of the work/ directory.

Run scripts: Submission

Likewise, but with the “submission” script.

The three buttons labeled “Files”, “Terminal”, and “Source” allow you to temporarily hide workbench panes that you are not using.

5.3.1.   Files pane

On the left side of the screen is a file tree browser that shows the contents of the workbench. This tree is similar to what Graders see, but you will be able to see more of the files, and edit any of them. The currently selected file has an orange marker next to it.

At the top are a few buttons to rename, delete, copy, or download the currently selected file. If a directory is currently selected, you can create a new file or directory, upload files into the directory, copy it, or download it.

The various directories are explained below.

Path	Description
resource/asnlib/	A library of files specific to this assignment or part. The $ASNLIB environment variable points to this path, which contains all of the auto-grader’s configuration files. Only Teachers and Graders can view these files, and only Teachers can edit them.
resource/asnlib/public/	Files specific to this assignment or part that are also viewable by Students.
resource/asnlib/publicdata/	Like the above public/ folder, but changes take effect immediately without needing to use the “Update” button.
resource/lib/	A library of files that is shared across the entire course. Changes made here will affect all assignments, though it is unknown if one must push “Update” on every assignment for it to take effect. The $LIB environment variable points to this path.
resource/lib/public/	Same as the asnlib/public/ folder, but shared across the entire course.
resource/lib/publicdata/	Same as the asnlib/publicdata/ folder, but shared across the entire course.
resource/scripts/	The location of the various scripts (build, run, submission, grading) for this specific assignment or part.
resource/startercode/	The contents of this directory are copied into each Student’s workarea when they first open the assignment if the copy startercode assignment setting is enabled.
work/	Workarea directory, for simulating script executions using the “run scripts” button (see above). This is also the directory that the terminal pane opens to by default, and the environment variable $HOME points here.

5.3.2.   Source pane

Selecting a file in the files pane will open its contents in a text editor within the source pane. There are rudimentary syntax highlighting, undo/redo, and search features.

If a change is made to a file using the source pane, a button labeled “save pending” will appear in the upper right hand corner of the pane. After a few moments of inactivity, the file will be automatically saved. However, changes will generally not take effect until you click the “Update” button to push the changes to all Students (unless you are editing in a publicdata/ directory).

5.4.   Configuring the auto-grader

The settings for CS 15900’s auto-grader are configured by editing files in the assignment workbench. A summary of the different configuration files is given below.

resource/asnlib/public/vocTerminalInit.sh

This script is sourced on terminal startup in Vocareum, and in various parts of the auto-grader. It should export the following environment variables:

$AG_FILES (required)

The space-delimited list of files that must be present at submission time. The base name of the first file in the list is used as the short assignment name and the primary file, unless $AG_NAME and $AG_PRIMARY_FILE are provided.

$AG_PRIMARY_FILE (optional)

This file is used implicitly with $stylecheck and the various *RunTest commands if no file is specified.

$AG_NAME (optional)

The short name of the assignment (e.g. lb01, hw01, ex01).

resource/asnlib/plan

This is the “assignment plan file”, which describes the auto-grader’s configuration for a given assignment. The only variable that must be specified is $chapter; the rest have default values. For more information, see the section on other auto-grader settings.

This file also contains the specification of test cases that will be run against the Student’s submission inside of a shell function called plan().

plan() {
     # At submission time, creates the "lb01.elf" executable file from
     # the Student's submitted "lb01.c" file and saves it, otherwise
     # "fails" the submission (aborts here, and the grading script will
     # assign a score of zero when it runs)
     submissionCompileCProgram lb01.c

     # Runs the test against lb01.elf with "1 2 3" as standard input.
     # The output from the Student's submission executable will be
     # compared with the output from the $ASNLIB/lb01.elf executable.
     #
     # This test will only occur during submission time, and has no
     # impact on the Student's grade.
     submissionRunTest lb01.elf <<<"1 2 3"

     # If no standard input is needed, use /dev/null (otherwise the
     # script will hang, waiting for input)
     submissionRunTest lb01.elf </dev/null

     # You can also add command-line arguments that will be passed to
     # both the Student submission and the solution executables, and mix
     # that with either of the forms above.
     #
     # For example, the following will test the program with "1 2 3" as
     # standard input and "4", "5", and "6" as command-line arguments.
     submissionRunTest lb01.elf 4 5 6 <<<"1 2 3"

     # Here's "4", "5", and "6" command-line arguments with no standard
     # input.
     submissionRunTest lb01.elf 4 5 6 </dev/null

     # Any of the above forms can be used with gradingRunTest (only run
     # at grading time, and NOT during submission time) or alwaysRunTest
     # (run during BOTH submission and grading). Compared to
     # submissionRunTest, these two functions WILL affect a Student's
     # grade.
     #
     # Thus, you can make "secret" test cases using gradingRunTest, and
     # public test cases that either affect the Student's score
     # (alwaysRunTest) or don't (submissionRunTest).
     alwaysRunTest lb01.elf <<<"0 0 0"
     alwaysRunTest lb01.elf 1 </dev/null
     gradingRunTest lb01.elf 1 10 100 <<<"5"

     ##########################
     # !!! IMPORTANT NOTE !!! #
     ##########################
     #
     # Do NOT pipe things into the test functions! This will spawn a
     # subshell, which means that the test case can no longer update
     # global state in the auto-grader. What this means practically is
     # that TEST CASES THAT ARE PART OF PIPES WILL NOT AFFECT A STUDENT'S
     # SCORE!!!
     #
     # For example, the following would execute a test case with "1 2 3"
     # as standard input, but since it is part of a shell pipeline, it
     # will NEVER change a Student's score, even though the test case
     # result will appear in the report files!
     #
     #echo "1 1 1" | gradingRunTest lb01.elf
}

$ gcc -o $ASNLIB/lb01.elf $ASNLIB/lb01.c
$ # Or, equivalently:
$ gcc $ASNLIB/lb01.c
$ mv a.out $ASNLIB/lb01.elf

7.1.   Create a new assignment

1. In the assignment editor, click “copy assignment”.

   

   You will be prompted to select either the current course (default) or a previous course. If you are creating a new lab, then copy an existing lab. If you are creating a new homework, then copy an existing homework.

   !DANGER!

   Make sure you select a SPECIFIC assignment to copy, as the default option is to copy ALL ASSIGNMENTS from the selected course!

   

2. Set the assignment title and an optional assignment group in the assignment information. Click the “info” icon to copy the assignmentid for use in the lab manager.

   

   Click the blue “Save” button above the assignment title.

3. Go to the lab manager and create a new assignment.

   

   Click the green “Save assignment” button when you are done.

4. Back in Vocareum, scroll to the “Options” section.

   For homework assignments, set all options OFF except for send webhook which should be ON.

   

   For lab assignments, set the following options: publish grades OFF, team project ON, section based timelines ON, send webhook ON.

   

5. For lab assignments, click the blue “team assignment setup” button. Verify that the options shown below are enabled, and click OK.

   

6. For lab assignments, click the blue “section timelines” button. Scroll to the very bottom of the pop-up window.

   

   In the lab manager, select export timelines to Vocareum. Select the appropriate assignment and click the green “download CSV” button. This will save a CSV file to your computer.

   Note

   If the assignment does not show up in the list, then you did not enable section-based timelines for the assignment in the lab manager. You can edit the assignment and check the box shown in step 3.

   Next, click the “browse” button back in Vocareum to upload the file you just downloaded from the lab manager. Click “apply”, then (again at the bottom) “save”. Then, click the gray “Close” button.

   Back in the assignment editor, click the blue “save” button once more above the assignment title.

   Note

   If clicking “save” in the section-based timelines window is taking a while, close it anyway, then re-open it. It is likely the settings were saved anyway.

7. Make sure the LTI option is enabled.

   

8. Scroll to the bottom and open the “Advanced” section. Most of these settings should already be set to the correct values, but you must change the “grace period” to 10 minutes.

   

9. Click the blue “Save” button once more above the assignment title.

10. For homework assignments, scroll back up and click part1.

    

    Scroll down to the “Submission” section, and select “Due Date”. You should set this to 23:00 ET for the Monday on which the homework is due.

    

    Click the blue “Save Part” button at the top.

11. Open the assignment workbench by clicking the blue “configure workspace” button in the top right corner.

12. In the file tree on the left hand side, navigate to the path resource/asnlib/public/vocTerminalInit.sh by clicking the plus icons and selecting the file. The currently selected file or directory has a light blue background.

    

    Note

    You can also access this file using $ASNLIB/public/vocTerminalInit.sh from the terminal.

13. Change the $AG_FILES variable to contain the submission filename in single quotes. For Lab #1, this would be lb01.c, and for Homework #1 this would be hw01.c. This is the exact filename that students must submit, and the name is also used in the terminal command prompt.

    

    The GUI file editor will save automatically after a time, or you can click “save pending” to save it immediately.

14. Navigate to the solution source file. You will need to rename this using the “Rename” button or the mv command to match the name you provided in the previous step.

    

15. In the GUI file editor, paste your solution source code and save it.

16. You now need to compile your solution into an ELF file (a.out). Run the following command from the terminal:

    $ gcc -o $ASNLIB/lb01.elf $ASNLIB/lb01.c
    

    This will use the same settings as the students use.

    The name should be identical to the solution file, but with the file extension changed from .c to .elf.

    You should delete any old ELF files.

17. Navigate to the resource/asnlib/public/docs directory. Click the “Upload” button to upload the assignment instructions (PDF of HTML).

    

    Delete any existing files in the directory and rename your new file either README.pdf or README.html.

18. Navigate to the resource/asnlib/plan file. This contains the instructions on how the assignment should be tested and graded.

    Set chapter based on the assignment requirements. The letters in the code stand for functions (F), selection (S), loops (L), arrays (A), pointers (P), files (t - lowercase only), and program arguments (m - lowercase only). Lowercase code letters are requirements, uppercase code letters are prohibitions. You may specify either a chapter based on the table below (R = required, Y = allowed, N = forbidden), or provide a code directly.

    Ch	Code	Functions	Selection	Loops	Arrays	Pointers	Files
    1	FSLAP	N	N	N	N	N	Y
    2	FSLAP	N	N	N	N	N	Y
    3	FSLAP	N	N	N	N	N	Y
    4	fSLAP	R	N	N	N	N	Y
    5	fLAP	R	Y	N	N	N	Y
    6	flAP	R	Y	R	N	N	Y
    7	fltAP	R	Y	R	N	N	R
    8	flaP	R	Y	R	R	N	Y
    9	flp	R	Y	R	Y	R	Y
    10	flap	R	Y	R	R	R	Y
    11	fla	R	Y	R	R	Y	Y

    Set factor equal to 1 for homework (assignment is worth 10 points), and 0.5 for labs (assignment is worth 5 points).

    Inside the plan() function, you indicate the source must be compiled (submissionCompileCProgram on a line by itself), and then the test cases follow.

    Use alwaysRunTest for test cases that should be shown during submission and included in the grade.

    Use submissionRunTest for test cases that should only be shown during submission and NOT graded.

    Use gradingRunTest for test cases that should only be graded and NOT shown when submitted.

    Each test case should be on a line by itself with the test type, followed by a space and <<-EOF. This indicates that the input for the test case follows, up until EOF is on a line by itself. Use literal tab characters for indentation.

    The first test case in the example below will have the inputs 10 and 15, will be shown during submission, and will also be included in the grade.

    The last test case will have the inputs 4 and 6, will NOT be shown during submission, and WILL be included in the grade.

    #!/bin/bash
    # vi: noet ts=4 sts=4 sw=4
    chapter=1
    factor=0.5
    
    plan() {
      submissionCompileCProgram
    
      alwaysRunTest <<-EOF
              10
              15
      EOF
    
      alwaysRunTest <<-EOF
              10.5
              15
      EOF
    
      alwaysRunTest <<-EOF
              10
              14.5
      EOF
    
      alwaysRunTest <<-EOF
              11.3
              2.7
      EOF
    
      gradingRunTest <<-EOF
              4
              6
      EOF
    }
    

    Save the plan file.

19. You can test that everything is working by copying the solution source file to the work directory and submitting. Do this either using the GUI, or the following terminal command (when the terminal starts, you are put in the work directory by default):

    $ cp $ASNLIB/lb01.c .
    

    Any files in the work directory are never seen by students.

20. Next, click “Run scripts”, then “Submission”.

    

    If all goes well, you should receive an email confirmation, and the submit.rst/submit.html files will appear in your work directory. Verify that the test cases are as you configured.

21. Click “Run scripts”, then “Grading” to test the grading script. No email confirmation will be sent, but the grade.rst/grade.html files should appear in the work directory. Verify that the test cases are as you configured.

22. Finally, click the blue “Update*” button. This will make all changes that you have made to the assignment on this screen available to the students. Until you click this button, your changes ARE saved, but they ARE NOT YET in effect for students.

7.2.   Release a solution to TAs

1. Go to the DEV course on Vocareum.

   

2. Go to Edit Assignments, and then click “copy assignment”.

   

   Select the assignment you just created in the student course, and click “copy”.

   !DANGER!

   Make sure you select a SPECIFIC assignment to copy, as the default option is to copy ALL ASSIGNMENTS from the selected course!

   

3. Change the assignment title to add “solution” to the end, e.g. “Lab #0 solution”. Move it to the appropriate group.

4. Turn all options OFF except send webhook.

   

5. Click the blue “Save” button at the top.

6. Above “Options”, click part1.

   

7. Under “lab interface”, enable “reset” and “nav bar”. The resulting set of options should look like the following. Click “save part” at the top.

   

8. Click “configure workspace” in the upper right hand corner.

9. You will need to copy the .c solution file from resource/asnlib to resource/startercode. You can do this using the file tree, or through the terminal. For example, for Lab #0:

   $ cp $ASNLIB/lb00.c ../resource/startercode
   

10. Click the blue “Update*” button.

11. Click the assignment title to return to the settings page.

12. Click the “publish” button.

7.3.   Release an assignment to Students

1. Click the assignment title to return from the workbench to the assignment settings.

   

2. Publish the assignment on Vocareum by clicking the blue “Publish” button.

   

3. In Brightspace, go to the content module where you want the assignment to appear. Select “Existing Activities”, then “Vocareum”.

   

4. Find the assignment in the list, then click either “iframe” or “New Tab” (the choice does not matter at this time).

   

5. Edit the properties of the new topic.

   

6. Make sure “Open as External Resource” is enabled. You might also enter a start date, change the title, and add a description.

   

7. Go to the “Grades” tab at the very top of Brightspace, then the “Manage Grades” tab, and find the corresponding column for the assignment you just added. It will likely be at the bottom of the list. Select “Edit”.

   

8. Add “Raw” to the end of the name, optionally add a short name, and change the category to “Raw Vocareum scores”. This will hide the column from students and exclude it from the final grade calculation. Click the blue “Save and close” button when you’re done.

   

7.4.   Process an assignment after the deadline has passed

1. Ensure that the grace period, if any, has elapsed in addition to the deadline. This is usually 10 minutes.

2. On Vocareum, go to “dashboard” and select the assignment.

   

3. In the assignment controls menu, select auto-grade. The grading script will be run over all submissions for the assignment.

   

4. As the auto-grader processes the submissions, it will output a username and a score for each one. For labs, this will be a generic team identifier rather than any particular student’s email.

   

   You should look for any entries that say [] instead of a score - these will need to be re-processed.

5. When you want to release the assignment to TAs for grading, go to the lab manager and select “export graders to Vocareum”.

   Select the assignment, then click “download CSV”.

   When you do this, you may be prompted to manually assign a few TAs if a student changed sections and the lab manager is unable to decide who to assign them to.

   Once the CSV file is downloaded, the lab manager immediately emails the TAs to let them know that the assignment will be available for them to grade soon.

6. Back in Vocareum, go to “Controls” and “Assign graders” in the same menu as you did “auto-grade” previously. Select “Use CSV file”, then click “Browse” to find the CSV file you just downloaded.

   

   Click “Assign”. Vocareum will list any submissions that did not receive a grader. This is normal if those submissions were unsuccessful, or (for labs) they are not yet ready to be graded.

7. For homeworks, you should publish the grades immediately. Go to “edit assignments”, change “Publish grades” to “On” in the options section, then click “Save”. This will allow students to see their grades in Vocareum, and emails will begin to deliver to students as soon as their TA has finished grading the assignment.

   

8. For labs, you should wait to perform the above grade publishing step until the last lab section has submitted the assignment. This is typically Friday afternoon.

7.5.   Sending final grades to Brightspace

1. You can monitor grading progress using the “view grading progress” option in the lab manager.

2. Once the TAs have all finalized their grades for a particular assignment, select “push grades through LTI” in the assignment exports to send the final grades to Brightspace.

   

3. After pushing the grades, go back to “edit assignments”. You will now disable LTI. This will NOT prevent students from opening the assignment, but rather will prevent further grade changes in Vocareum from taking effect. This is especially important for labs.

   

   Click “Save” after disabling LTI.

4. In Brightspace, go to the content module that the assignment resides in. Right click the assignment title, and open it in a new tab. You should see a screen like the following.

   

5. Click the “Assessment” area in the bottom right corner. Here you will select -- No grade --, then click “Save”. This further prevents additional changes to the grades from Vocareum.

   

6. For homeworks, you will go to the lab manager and select “view and edit assignments”. Here, you will check the “Exported” option at the bottom and click “Save assignment”. This doesn’t do much other than remove the homework from the “view grading progress” screen.

   

7. For labs, you will go to the lab manager and select “synchronize lab grades with attendance”. There are instructions there on how to export grades from Brightspace as a CSV. When exporting grades from Brightspace, you should have something like the following (also select the quiz column). Then click “export to CSV”.

   

   Next, you will upload the Brightspace CSV to the lab manager using the link above. The default settings should look like the following.

   

   Once you click “Upload”, you will be prompted to save an updated CSV from the lab manager. This new CSV file will contain only the necessary changes to individual lab, quiz, and attendance entries based on a student’s attendance status.

   In Brightspace, go to “Grades”, “Enter grades”, “Import”, and select the lab manager CSV file. Click “Continue” until the import is complete.

   You should also share the lab manager CSV file on Slack with the TAs and other instructors.

8. Next, go back to “Grades”, then “Manage Grades”. You will need to edit the column again.

   

9. Remove “Raw” from the name, then change the Category to “Lab tasks” or “Homework”. Click “Save and close”.

10. For labs, do the same for the attendance column: remove “Raw” from the name.

11. For labs, do the same for the quiz column: remove “Raw” from the name, and uncheck “exclude from final grade calculation”.

12. Click “Enter grades”. This will force Brightspace to update the point totals for all students. This may take a while if there are too many students.