Application name and one sentence description
Clatter (Class + Chatter) is a web-based, classroom-focused chat application. Users and teachers can send messages, ask/answer questions, post relevant course material, and tag and search content. Chat rooms exist at unique, easily-bookmarked URL’s.
Goal of Test
Purpose: Comparison test + Field test
Objectives
What feature is missing to make this application “real good” at the core functionality it offers: chatting with students in the class?
How better to avoid clutter in the class chat window?
Try laptop and phone interface to compare the two, find how the layout and functionality is rendered at different screen sizes
Subject Profile
User1 was a 26 yr. old male graduate student of Linguistics. He is a student most of the time but occasionally lectures.
User2 was a 22 year old female, intermittent student in Biology and statistics. She works at a museum most of the time and does statistics courses to help her job.
Instruction Script
Create an account and Log In to Clatter
Write a message into Winter CS101 2012 class chat in response to a message already present.
Using Search, find the line(s) that mention the word “homework”.
Enter the Fall CS 101 chatroom. How many messages have been written?
Log out of the system.
Performance Criteria Results
User 1
| Create an account and log in in no more than 20 seconds | Completely functional. |
| Enter chat room and write a message within 5 -10 seconds | Pass |
| Search for a keyword and find a line that contains something about it within 5 seconds. | Pass |
| Exit current chat and enter the chat room for a different course within in 5 seconds | Pass |
| Log out of system in 5 seconds | Pass |
| Scale of 1 to 10, rate ease of finding your way around the application | 8 |
| Would you use voice-recognition technology instead of typing, not in lecture hall? | Probably not |
Open Ended Questions
If you could add just one feature, what would it be? Better Search, File uploads
What interface would you use this most on? Laptop or phone
What do you find most annoying about the application? Laggy
Would class size, subject, location (online vs. tradition) change your usage habits of the application? If so, how? Larger class sizes might create environment where chat would be more useful. More engagement. More input from other people. Hard to get excited about class and chat about it if you don’t enjoy the class material though. I’d probably be more likely to rely on the chat room if the class was remote.
History of conversations: how long to keep them? Forever?
User 2
| Create an account and log in in no more than 20 seconds | Pass |
| Enter chat room and write a message within 5 -10 seconds | Pass |
| Search for a keyword and find a line that contains something about it within 5 seconds. | Fail |
| Exit current chatroom and enter chat room for different course in 5 seconds | Fail |
| Log out of system in 5 seconds | Pass |
| Scale of 1 to 10, rate ease of finding your way around the application | 10 |
| Would you use voice-recognition technology instead of typing, not in lecture hall? | Maybe, in the car |
Open Ended Questions
If you could add just one feature, what would it be? Separated Threads (by topic)
What interface would you use this most on? Laptop (because better typing on laptops), or tablet
What do you find most annoying about the application? if you open a sidebar, and try to type something while it’s open, the sidebar pops back in.
Would class size, subject, location (online vs. tradition) change your usage habits of the application? If so, how?
Class size: if 5 person class, threads wouldn’t be needed, but with huge class, you would need threads.
Subject: Boring class, then user would use the app to ask questions about what they want to do this weekend, but if interesting class, then user would actually use it for class-related questions. i.e. social studies: user wouldn’t really ask questions related to the class. Doodle could be a nice add on, especially on tablets.
Location: for online, definitely use it more. For traditional classes, maybe, but would need to “catch on” - would ask for clarifications or little things, but she wouldn’t necessarily answer questions.
History of conversations: how long to keep them? Active until end of class, or when everyone
Learnings on Application
User 2 helped learning on application by thinking aloud. This is the first time she has played with it.
List of classes: not sure if it is a list that is default or those that user is enrolled, but would be obvious if she knew she was enrolled in. They are clearly classes.
When clicked left arrow, she thought it would take her to the next class, but got a navigation bar. Didn’t bother her, but she thought that because that’s all the information she saw before
Right arrow she thought “was also something hidden” - should show users when they’re signed in (correct assumption). She prefers that there be an option to keep sidebars open on right and left - useful on a laptop. With a cellphone, with side view, should be able to drag in sidebar with finger.
Same feedback as user from previous studies: too many students, too many messages. User 2 likes idea of a thread - start a thread -> enter topic -> if any of those topic match a previous thread, then you can enter one of those, or start a new one, choose privacy level (by user/ users/everybody). chose to join, leave (close), to join again, search for thread again. Having a “History” tab to show threads you were in. Could work as a search or asking a question, or posting something.
Learnings on Usability Test
Users may have a completely different idea that may require rewrite of the organization of chats (in our case). Based on the feedback, it sounds like most users are interested more in a forum than they are real-time chat. The idea of chat by topics was the most important thing to user 2, which is not supported, and may need complete rewriting of how the conversations are organized. Using the same group of users throughout (and as many users as possible) would have helped us correctly design the application from the start rather than getting new requirements in the last phase.