CS 15900 & Vocareum: teaching assistant’s guide

Contents

1.   Lab records

Navigate to the “lab records” page of the lab manager. You will need to choose a lab section and an assignment.

1.1.   Vocareum status

Once you do so, the page will indicate how many records are available for the chosen lab section and assignment. Below this information, there is a table with a row corresponding to each student in your lab section. The table includes each student’s first and last name, their username, and the status of their Vocareum submission, if any.

ta_records.png

Here is the description of each student’s status, starting from the top:

  • student1 has dropped the class. In the unlikely event that you are assigned a submission from a dropped student, you should grade it as normal - it might be the case that the student will re-add the class.

  • student2 has not submitted anything for this assignment.

  • student3 has made a submission, but it is not assigned to you for review yet. This is probably because the assignment deadline has not yet elapsed.

  • student4’s submission has been assigned to another TA, even though they are in your lab section. This happens very rarely with team assignments when the student has switched sections at least once. You need not take any action, though you may hover over the word “assigned” to reveal the TA responsible for grading this submission.

  • student5’s submission has been assigned to you, and has not yet been reviewed. This is included in your “ungraded” count.

  • student6’s submission has been reviewed (graded) by you or another course staff member.

    An email notification to the corresponding student(s) is pending. Once grades for this assignment are published on Vocareum, the lab manager will begin sending these pending emails in batches every hour. You will be CC’d on each email that you initiated.

  • student7 has been notified by email that their work has been graded, reviewed by a course staff member, and is ready to be viewed by themself.

For student5, student6, and student7, the blue text in the “Vocareum status” column is a link to open the corresponding submission in Vocareum’s submission viewer.

1.2.   Section controls

Underneath the table are some controls and the status of your grading for this lab section and assignment.

ta_ungraded.png
  • The Refresh button will refresh the current view of this lab section and assignment with the latest data.

  • The Back button will take you back to where you might select a different lab section and/or assignment.

  • The Progress button will take you to the grade progress page: an overview of the assignments for which you still have ungraded submissions.

  • If you have submissions that remain to be graded, a yellow box will appear down here indicating as such.

  • Once all submissions for this section and assignment have been graded, the yellow box will be replaced with a green one:

    ta_section_done.png

1.3.   Attendance

For lab assignments that require attendance, there will be two extra columns between “Username” and “Vocareum status”: “Attendance” and “Override”.

ta_attendance.png

The “Attendance” column lists each student’s current attendance status.

  • The date and time indicate when the student opened the attendance page.
  • The red text beginning with ls- is the ITaP machine name from which they opened the attendance page.

Looking at each student in turn:

  • student0 submitted attendance, but was overridden by mcrees to be absent. Unless stated otherwise, students who are absent earn a score of zero on both the programming assignment and quiz, even if they either or both were submitted.
  • student1 has no record of having ever opened the attendance form for this assignment. This behaves in every way the same as absent.
  • student2 submitted attendance late. They will earn a score of zero on the programming assignment, but are still eligible for quiz credit.
  • student3’s attendance was not submitted. This behaves in every way the same as absent. They need to go back to the attendance form and click the “Submit” button, or you need to provide them with an override.
  • student4 submitted attendance on-time and will be eligible for full credit for both the programming assignment and the quiz.
  • Notice also that student3 and student4 are grouped together because they are on the same team and thus share a submission.

1.4.   Attendance overrides

The “Change…” drop-down box in the “Override” column allows you to modify a student’s attendance status, regardless of whatever it is currently or if an override already exists.

If you need to give one or more students an attendance override:

  • For each student, select the corrected status in the “Change…” drop-down box.

  • You may change the attendance for multiple students at a time.

  • If you do not need to change the attendance status for a particular student, leave it as “Change…”.

  • Click the green “Override” button at the bottom.

    ta_override.png
  • The lab manager will let you know if these overrides were successful or not.

  • The changed statuses will display in the new records table that is returned.

  • The table will also include your username and the time at which you gave the override.

The “Change…” drop-down box may be missing for two reasons:

  • The corresponding student has dropped the course. You should not need to change their attendance anyway.

  • The assignment’s attendance and grades have been exported to Brightspace, as indicated by a black “Exported” box at the top of the page.

    ta_exported.png

    If you still need to make revisions to a student’s grades or attendance at this point, you must contact the lecturer associated with your lab section.

Students are informed that they may receive at most one positive attendance override from you during the semester. You should not need to provide a second attendance override except in extraordinary circumstances, to resolve technical issues, or to withdraw credit if necessary.

Note

If you have sufficient evidence that indicates a particular student did not contribute to the lab programming assignment, you may give them a “Late” or “Absent” override to withdraw credit.

2.   Submission viewer

By clicking one of the links in the “Vocareum status” column of the lab records table, you can open the corresponding submission itself in Vocareum’s “submission viewer”.

There are many different parts of this interface which will be discussed in the subsequent sections. At the very top of the screen is the name of the assignment for which the submission was made.

ta_submission_viewer.png

2.1.   Submission pane

The pane across the top of the screen shows information about the submission, such as the student(s) name(s) and email address(es), the section of the student(s) and the last time the grade was modified (whether that was by the auto-grader, yourself, or another course staff member).

ta_submission_pane.png

The drop down box shows how many times a submission has been made for this assignment by this student or team, and at which time each submission was made. The latest submission should be selected by default.

Note

You may view earlier submissions, but keep in mind that the grade should always reflect the work of the latest submission only.

2.2.   Files pane

The “Files” pane on the left shows all of the files that are part of the student’s submission. This includes some files that are generated by the submission or grading scripts. The currently selected file, if any, has an orange icon next to it. The name of this file will also replace “Files” in the pane’s title. You can click the caret icon (^) to hide the Files pane.

ta_submission_files.png

The files generated by the submission and grading scripts include:

*.compile
Warnings and errors issued by the compiler. This information is also included in the submission and grading reports.
*.elf
Executable created by the compiler (instead of a.out).
submit.rst
Submission report in a plain-text format. This is essentially the same contents as the submission receipt email that gets sent to students.
grade.rst
Grade report in a plain-text format, as generated by the auto-grader. This is the file you should pay the most attention to, in addition to the student’s source file.

After grades have been published, you may see some additional files starting with .voc. These are essentially duplicates or alternative formats of the files listed above and can be safely ignored.

Note

While the submission may contain an a.out executable that the student created interactively, the file ending in .elf is the executable that was compiled and tested by the auto-grader.

2.3.   Source pane

The Source pane in the center shows the contents of the file you select from the Files pane. This includes some rudimentary syntax highlighting as well as line numbers. You cannot edit files using the source pane on this page.

ta_submission_source.png

The buttons along the top include:

README
Opens the assignment instructions as a floating pane.
Debug
Opens the submission debugger.
Download
Downloads the entire contents of the submission as a ZIP file.
Report
Opens the same HTML reports that the students can see, but might only work if grades have already been published. Instead, you can view the plain-text report files described earlier. Or, you can view the HTML reports inside the submission debugger.

The right arrow icon (>) can be used to collapse the grade pane.

If you double click on text inside of the source pane, you are presented with an option to create an in-line comment.

ta_submission_source_line_comment.png

Warning

Do not use this feature! students will not be able to see these comments due to the limitations of Vocareum in its current configuration. Hit “Cancel” to close the dialog. Only put your feedback in the grade pane (see below).

2.4.   Grade pane

Here you can adjust the grade for the submission and leave feedback. Adjusting the score for any of the rubric items (Format, Technique, Output) automatically adjusts the overall grade, which is shown in the progress bar below as “Score”. You can also click the blue “Full” link to give full points for the submission.

ta_submission_grade.png

You can click the speech bubbles next to each rubric item in order to leave rubric-specific comments. The speech bubble will change color from gray to black when you do this.

ta_submission_grade_rubric.png

When you are ready to save the scores and feedback, click the blue “Save” button.

If you want to reset your changes, refresh the page without hitting the “Save” button.

At the bottom of the pane it will say “Grades submitted by:”. After you click “Save” at least once and refresh the page, it will be replaced by your name and email address.

Note

You should always leave some sort of feedback, even if it’s simple as a “Great job!” for a perfect submission. This allows the lecturers to verify that you have reviewed all of the submissions assigned to you. Make sure you click the “Save” button!

You can click the right arrow icon (<) to make the grade pane slightly bigger.

3.   Submission debugger

By clicking the “Debug” button in the source pane, a new page will open from which you can start an interactive terminal similar to what the students use. Click the blue “Launch Terminal” button at the bottom of this new page to do so.

ta_submission_debugger.png

There is also a new button in the source pane: “Actions”. The following options are available:

Run Grading

Executes the auto-grader again.

Warning

This will immediately change the scores and save them.

Execute Submission Script

Re-runs the submission process.

!DANGER!

Using this feature could break the students’ submission, so you are advised to refrain from using it.

The “Edit Files” button allows you to make changes to the files in the submission using the GUI text editor. You may also edit files through the terminal below by clicking “Launch Terminal” and using e.g. vi.

4.   Workflow

  1. Go to the grading progress page in the lab manager. This will show you which assignments remain for you to grade. Your row (labeled with your name) will be highlighted in purple.

    ta_progress1.png

    If your name is “TA 2”, then you have 9 submissions total for Lab #1 and 24 submissions for Homework #1. You still have 9 and 23 left to grade.

  2. Open the lab records page. Select the lab section and assignment you need to work on. At the bottom, the number of ungraded submissions in this view is displayed.

    ta_ungraded.png
  3. Open each student or team’s latest submission by clicking the last column.

  4. View the grade.rst report file from the auto-grader. The test cases will be listed first, followed by a breakdown of the grade and any deductions that were made.

  5. Verify that these deductions are correct by reviewing the test cases and the student or team’s program source file. If a deduction made by the auto-grader is incorrect, give back the points by adjusting the scores accordingly and leave a remark in the “overall review comments” text box or rubric comment text boxes in the grade pane.

  6. Review the program source file for any additional deductions to be made. If there are any, include the deduction in the “overall review comments” text box or rubric comment text boxes, and adjust the scores as necessary.

  7. Make sure you leave some sort of feedback as well in the grade pane.

  8. Click the blue “Save” button.

  9. If you are done with this submission, click the green “share feedback” button when prompted. This will queue an email to the corresponding students. This email will include a link to the instructions on how to view their grades in Vocareum.

    If you want to save your work to come back later, click the red “save and close only” button. You will need to come back to this submission later to click the “share feedback” button. The “share feedback” button must be clicked at least once in order to finalize the grade for this submission.

    ta_share_feedback.png

    Each time you click the “share feedback” button, a notification email will be queued for the corresponding students.

  10. Return to the lab records page. Click the “Refresh” button to confirm that the submission you just graded has changed from “Assigned” to “Graded” or “Notified.

  11. Review the attendance of the corresponding students and submit overrides, if applicable.

  12. Click the link for the next submission in the table.

  13. Continue until you have reviewed all students or teams for this assignment and lab section. After a final refresh, the yellow box at the bottom should be replaced with a green one indicating that this assignment and section are done.

    ta_section_done.png
  14. Click the blue “Back” button to select the next lab section for the same assignment, if applicable.

  15. Return to the grading progress page in the lab manager and verify that you are done grading the current assignment.

    ta_progress2.png

You will be notified (via Slack) when grades have been pushed to Brightspace, after which you will need to tell your supervising lecturer if a grade needs to be changed.

You will be notified (via email and/or Slack) when you are behind on your grading.

5.   Additional deductions

The previous auto-grader on the guru server would leave additional deductions that needed to be reviewed manually by a TA under a line of dashes (---). These standardized additional deductions are now listed here. You should return to this section when grading every assignment to ensure that you make these deductions as appropriate.

Warning

These point values are based on a homework assignment (worth 10 points total). Divide by two for a lab assignment (worth 5 points total).

Category Points Description
Format -0.2 Program header block comment lacked an adequate documentation
Format -0.2 Function header block comment lacked an adequate documentation
Format -0.5 You must use meaningful variable names for all variables
Technique -0.5 Only major control processes belong in the main function
Technique -0.5 Each user-defined function must be devoted to only one specific task
Technique -0.5 Your algorithm does not meet the standards for efficiency
Technique -4.0 Not a reasonable attempt at the stated assignment (format deduction too)
Technique -4.0 Technique exceeded the current material covered (output deduction too)