During the first quarter of 2020, Trilogy had a corporate goal of cost cutting inside their boot camp courses. In order to make individual boot camp courses more profitable, they wanted to reduce the number of Teaching Assistants in each class. Initially there was approximately one TA per 10 students, but that was proving costly. One main job of TAs in Trilogy classrooms was to grade homework assignments turned in by students, but TAs were already under a heavy load to make it through all student assignments and get students feedback in a reasonable amount of time.
The problem was clear:
Through brainstorming and working with stakeholders, a solution came out: Create a centralized grading platform with a team of dedicated graders.
Users and audience
The users of BCS Grading would be central graders who were tasked with claiming homework assignments and then grading them, and central grading managers who were tasked with managing the central graders.
Roles and responsibilities
My role in this project was as the principal UX/UI designer
This project was triggered by a corporate push to reduce the costs associated with running a boot camp course. Looking at all costs related to courses, one suggestion was to reduce the number of TAs to a class.
But another problem remained that would only be exacerbated by the reduction of TAs: TAs graded homework assignments and many classes were already behind on homework grading.
Through brainstorming with stakeholders, the idea to centralize the homework grading process came about. A team of dedicated graders would work to grade assignments across three of Trilogy’s program offerings.
Trilogy already had a centralization effort that included services like centralized tutoring for students and would later include a Slack-based platform for answering students’ technical questions. The existing centralization team would manage the new central graders since the infrastructure was already in place.
Assumptions and constraints
A few assumptions were made during the initial design of the project: a custom platform could be built to power the grading operation side of centralized grading; the platforms where students turned in assignments could be integrated into to both get assignments into the new central grading platform and deliver feedback back to the student-facing platforms; central graders would be hired to make up the central grading team; and central grading managers would be hired to deliver communications to graders in addition to managing day-to-day administration.
Early in the process the technical feasibility of integrating into the Learning Management Systems (LMSes) was looked into. Trilogy ran online classes through an off-the-shelf LMS called Canvas, and in-person classes through an in-house LMS called Bootcamp Spot.
Because Bootcamp Spot was build in-house, it would be easy to modify. The design team was free to make changes to Bootcamp Spot when adding new features to central grading. However, Canvas was not as flexible. Canvas is a closed-source, hosted LMS and some features were not changeable without building LTI (Learning Tools Interoperability) add-ons that would function as applets inside the Canvas instance. This would become a problem as the assignment submission and grading tools built into Canvas were not a flexible as other features in the LMS.
One problem the team raised early on was about members of the instructional team having a say in the grades assigned by central graders. TAs and instructors had unique insight into students’ performance, so it was important for them to be both kept up to date with students’ homework grades, but also it was important for them to have a say if they felt a grade was unfair.
A TA approval step was added to the central grading process. After a grader graded an assignment, it would go to the in-class TA to approve. If approved, the grade would be visible to the student and if it was not approved, it would be flagged for review by the grader.
When a grader logged into the platform, they would be presented with a list of assignments that had been submitted by students. Graders could claim assignments which would then put the assignment in the grader’s personal queue and take them off of the queue for other graders. Graders had the ability to filter assignments on the queues to allow them to find assignments that would fit their specializations. Inactive filters would be shown as slightly transparent to indicate their status while active filters would be both opaque and also have a blue dot next to them. Hovering over the “?” icon in the filter sidebar would display a note explaining that particular filter.
When a central grader selected an assignment from either the main queue or their claimed queue, a page with assignment details would open.
Assignment details pages had a handful of features:
- Details on the assignment along with links to the assignments components
- Indications if an assignment was flagged for plagiarism
- Indications if an assignment had been in the queue for more than 24 hours
- Notes about if the assignment had been resubmitted by the student, and a toggle to request that the student resubmit
- Both original notes that went along with the initial assignment submission as well as a comment thread for communicating with the instructional team and student
- A comment thread that students could not see for communicating with other graders, central grader management, and TAs
- A place for graders to grade the assignment and mark the assignment for suspicions of plagiarism
Messaging features were added to allow communications in a few directions:
- Messaging from managers to graders
- Messaging from graders to managers
- Notifications for new comments left on graded assignments that were viewable to students
- Notifications for new comments left on graded assignments that were internal and not viewable to students
The central grading project is an ongoing project where new features are added in agile sprints. New features are handed over to developers in design specifications complete with:
- Detailed mockup notes and specifications
- Animations (if applicable)
- Interactive prototypes
- Task flows