Pull Request Experience
Evaluating GitHub's Pull Request UI to enhance usability, efficiency, and developer experience.
Evaluating GitHub's Pull Request UI to enhance usability, efficiency, and developer experience.
GitHub conducted a comprehensive usability study to evaluate the effectiveness and user-friendliness of its Pull Request Feature, focusing on recently introduced UI enhancements. The study aimed to assess users' ability to navigate the platform, complete tasks efficiently, and provide valuable feedback for improving the overall developer experience.
My role
Product Designer and Usability Tester
Duration
10 weeks (6th January - 10th March)
Tools Used
GitHub, Zoom, Google Suite, Figma
Some terms that might come in handy for this case study
What's GitHub ?
GitHub is a web-based platform that provides tools for version control, collaboration, and project management, enabling developers to host, review, and share code, track changes, and work together seamlessly on software development projects.
Learn more >
What's a Pull Request ?
A Pull Request (PR) on GitHub is like a design critique for code, where developers propose changes, get feedback, and refine collaboratively before merging them into the main project.
Learn more >
Usability testing in action
GitHub is a web-based platform that provides tools for version control, collaboration, and project management, enabling developers to host, review, and share code, track changes, and work together seamlessly on software development projects.
Learn more >
6 participants were recruited from existing GitHub users who regularly engage in pull request activities. Following were the recruitment criteria:
Pull Request Experience
Stakeholders identified the Pull Request feature as a critical functionality for developers collaborating on code. The project was centered on evaluating recent UI enhancements to determine their effectiveness in making the feature more user-friendly and intuitive for developers during their workflow.
Developer's Productivity
The stakeholders were concerned with improving productivity by enabling developers to navigate and complete tasks, such as commenting and reviewing, more efficiently. The study aimed to measure how well the updated interface supports these tasks and reduces friction in their daily use.
Qualitative Assessment
Stakeholders emphasized the importance of comparing the new UI with the old UI as they aimed to identify which new UI enhancements should be retained and which existing features from the old UI should remain unchanged to ensure continuity and meet user expectations effectively.
To gain a deeper understanding of the current GitHub Pull Request UI and identify pain points faced by developers, I conducted a contextual inquiry with a developer friend who regularly uses the platform. This session focused on observing their workflow, understanding their challenges, and gathering insights to inform the usability study.
Commenting workflow
The process of posting, editing, and replying to comments felt cumbersome, requiring extra clicks.
Space taken by a comment
While reviewing a pull request, a lot of space is taken by the comment boxes, creating a discontinuity.
Based on the insights gathered from the contextual inquiry and stakeholders' inputs, the next step was to refine our research questions to focus on the most critical usability challenges.
Can existing users complete tasks as successfully with the new UI as the existing UI?
Can users find and resolve comments as quickly with the new UI as the existing UI?
Will existing users be able to merge pull requests faster?
Will existing users understand mergeability at first glance?
How does the usability of the new UI compare to the existing UI?
What are users' attitudes towards the new UI?
To conduct this study, we utilized a remote A/B testing approach, supplemented by the System Usability Scale (SUS) questionnaire and qualitative assessment.
Remote A/B Testing
To evaluate the usability and effectiveness of both the existing and new UI versions of GitHub's Pull Request Feature, we utilized a within-subject design for remote A/B testing. This approach allowed each participant to interact with both UI versions during the same session, enabling a direct comparison of their performance and usability.
System Usability Scale (SUS)
After interacting with each UI version, participants completed the SUS questionnaire to assess their perceptions of usability. The SUS is a widely used and validated tool for measuring the usability of systems and interfaces. It consists of a 10-item Likert scale, with participants rating statements related to usability.
Qualitative Assessment
In addition to quantitative metrics, we incorporated qualitative questions to gather insights into participants' perceptions and experiences with both UI versions. Participants were asked open-ended questions to explore aspects such as the learning curve, overall satisfaction, and any challenges encountered during the interaction.
GitHub conducted a comprehensive usability study to evaluate the effectiveness and user-friendliness of its Pull Request Feature, focusing on recently introduced UI enhancements. The study aimed to assess users' ability to navigate the platform, complete tasks efficiently, and provide valuable feedback for improving the overall developer experience.
Task Completion
While all participants finished the tasks in both the old and new UI, a few attempts were made for tasks in the new UI.
System Usability Scale (SUS)
(SUS) scores showed clear differences between the existing and new UIs, with the existing UI scoring higher at 77 compared to the new UI's 67.
Overall Error Rate
(SUS) scores showed clear differences between the existing and new UIs, with the existing UI scoring higher at 77 compared to the new UI's 67.
GitHub conducted a comprehensive usability study to evaluate the effectiveness and user-friendliness of its Pull Request Feature, focusing on recently introduced UI enhancements. The study aimed to assess users' ability to navigate the platform, complete tasks efficiently, and provide valuable feedback for improving the overall developer experience.
To conduct this study, we utilized a remote A/B testing approach, supplemented by the System Usability Scale (SUS) questionnaire and qualitative assessment.
Remote A/B Testing
To evaluate the usability and effectiveness of both the existing and new UI versions of GitHub's Pull Request Feature, we utilized a within-subject design for remote A/B testing. This approach allowed each participant to interact with both UI versions during the same session, enabling a direct comparison of their performance and usability.
System Usability Scale (SUS)
After interacting with each UI version, participants completed the SUS questionnaire to assess their perceptions of usability. The SUS is a widely used and validated tool for measuring the usability of systems and interfaces. It consists of a 10-item Likert scale, with participants rating statements related to usability.
Qualitative Assessment
In addition to quantitative metrics, we incorporated qualitative questions to gather insights into participants' perceptions and experiences with both UI versions. Participants were asked open-ended questions to explore aspects such as the learning curve, overall satisfaction, and any challenges encountered during the interaction.
.png){kind=small}
