The Yahoo UX Research and Accessibility (UXRA) team is heading to the ACM SIGCHI Conference on Human Factors in Computing Systems (otherwise known as CHI 2017) in Denver!
This year we have a series of great talks from our researchers and interns about the research we have conducted across consumer and advertising products. If you’ll be at CHI, you can find the room numbers for talks listed below. We’re also hiring, so stop by our booth at the breaks. If you can’t make it to Denver, links to download our papers are available below.
CHI is an example of how research in industry builds off of the best academic research available and contributes back to the research community with our own findings and systems that we create. Each of our papers below cites dozens of studies from other corporate labs or universities, and we hope contributes to future scholarship in these venues. We are an applied research group, and the findings from this research have a direct impact on our products and research culture within the company, as well as helps to put our users first in creating new product experiences at Yahoo for our billion+ users.
This week, Tumblr released a cool new iOS app called Cabana. The idea behind it is simple but really powerful. When you find a funny or interesting video you can’t wait to share with your friends, fire up Cabana, tap their names and invite them to join you. You’ll all see each other, and the video, together live. Share a good cry, make jokes, shout at the video, or just make crazy faces. It’s up to you what you say or do. The only caveat is to be careful—it’s totally addicting!
If you or a friend have a disability, you might be wondering if Cabana is something you can do too. Of course! We want everyone to be part of this new experience and invite their friends so we’ve optimized Cabana in a bunch of ways to make it accessible:
Cabana incorporates live video conferencing so you can use sign language
The color scheme has been tweaked to make it higher contrast for improved readability
You can drive Cabana using a physical keyboard or alternate input device
The app has been optimized for VoiceOver users (iOS) so if you’re a screen reader user, you can invite your friends, answer an invitation, and choose and control the video. Even the welcome and setup screens, including the camera feature used to capture your avatar photo, have been optimized.
Image: The Cabana app running on an iPhone showing six friends watching a video. The caption next to the phone reads “Watch videos together.”
Image: The Cabana app running on an iPhone showing the Friends list. The heading on the Cabana app reads “Hang Out, watch stuff.”
Cabana is free, and available today on the App Store for your iPhone. Check it out and invite your friends.
By Andrew Schulte, Chief of Staff to the CEO, Yahoo
A few weeks ago, we received nearly two dozen handwritten letters from 7th grade students at Chaboya Middle School in San Jose, all addressed to Marissa Mayer. These incredibly thoughtful and inquisitive letters penned questions about leadership, STEM, being a female CEO, the future of tech and general advice on working for a company like Yahoo. These letters touched and inspired Marissa and the team here, so what better way to answer their questions than a field trip to Yahoo headquarters!
We kicked off the day with an assembly that included an engaging Q&A discussion with Marissa addressing all the students’ top questions. Afterwards, we took them on an all access behind-the-scenes tour of what it’s like to work in Silicon Valley, including what it takes to launch some of our latest mobile products.
The first stop on their tour was our User Experience (UX) Lab where students had a chance to learn first hand about the importance of user feedback in consumer product development. Students met our UX and product teams, and learned how real world testing is critical to innovating and improving on products.
Next up, was a tour of Yahoo Studios, where students got the inside scoop on production, and how we put together a lot of our original video content. A few students even showed great promise as future on-air talent!
Then it was off to the Accessibility Lab, where they learned about the importance of building products that serve all our users, regardless of their physical capabilities. The students had an opportunity for hands-on problem solving to help individuals with hearing or visual impairment, or limited mobility use of our products.
We finished off the tour with a stop by our Publisher Products team, which works on some of Yahoo’s most popular products, like Yahoo News, Yahoo Sports and Yahoo Finance. Here, they got an inside look at a day in the life of an engineer – from coding the personalized homepage stream, to the data and analytics team that helps us make data-driven decisions, to the release engineers that manage quality control for final publishing.
We had just as much fun as the students, and we’d like to send our deepest thanks to Chaboya Middle School, Mr. Joe Ennes, and the 7th grade class for bringing their best and brightest energy to Yahoo. We had an amazing time hosting you, and have tremendous optimism for the great things you’ll achieve in the future!
At Yahoo we move quickly. Our product teams are often shipping new features every week or two and designers are constantly exploring new solutions. While User Research methods have traditionally been fairly slow to execute, especially in the academic traditions of anthropology and human-computer interaction, we have a need to move more quickly and fit our work into the weekly sprints of our product teams.
For Yahoo Mail, we tried an experiment last year where we conducted weekly user research every Friday. This gave teams Monday through Thursday to work on their new designs and implementations and Friday for us to conduct research and for them to observe and plan for their next week from our research labs. Concepts could be tested in a design state iteratively from week to week, and then tested again in a working build in a subsequent week once it was implemented. This allows us to rapidly test new designs, iterate each Friday until we get something that both provides value and is usable by our participants, and test final implementations once they are ready. We augment this with larger scale surveys and field studies as necessary for bigger features and strategic directions.
Along with testing prototypes, these sessions also include in-depth interviews on particular topics of interest, including walkthroughs of current behavior and understanding of recent experiences. These questions help us to understand particular domains over several weeks and dozens of participants and help to set strategy when combined with other qualitative data from surveys and usage logs.
These weekly studies have transformed the design and product process to truly be user focused. We test a wide variety of concepts and implementations each week. Product Managers and Designers don’t have to worry about setting up a special study for their topic, getting users recruited, and waiting several weeks for studies to be executed. They can rely on having spots in the weekly study for last-minute additions of new prototypes or qualitatively testing topics that have appeared in logs or online user feedback.
All of this has helped bring the Yahoo Mobile Mail app to its highest ratings ever, hitting and then staying at 4.5 stars on both iOS and Android for most of the year, while shipping a wide variety of new features for our users.
As a UX researcher working on advertising products, a decent portion of my day is spent observing how advertisers use advertising solutions. During various studies I have conducted, it came to my attention that their frustration often rooted in time and efforts required to complete a task. Additionally, we heard search advertisers’ needs in achieving even better results with their search campaigns on Yahoo Gemini while spending less time and effort managing multiple platforms. We came to realize that “time is money” to our advertisers.
Informed by these insights, Yahoo Gemini recently introduced an account sync feature that addresses advertisers’ needs. Account sync is a powerful tool to help search advertisers manage large-scale search campaigns across multiple search engines. The ability to sync accounts can make it simple for advertisers to import their entire eligible portfolio of campaigns from other platforms into Yahoo Gemini, with just a few clicks. By seeking a deep understanding of users’ values and beliefs, UX Research helped to inform product opportunities and we hope to continue improve the Yahoo Gemini experience.
Usability testing is a method for evaluating the ease-of-use of a product or feature by testing with with people, or “users” of a system. The goal of usability testing is to observe behavior during task flows, observe whether and how users complete flows, and note any conceptual misunderstandings that occur. Participants’ reactions are important for other methods such as surveys and concept testing however in usability testing we care whether participants can get the task done. Usability testing provides direct feedback by observing how real people use a product. Usability testing tends to be structured as a one-on-one task-based interview where a facilitator prompts a participant to complete a series of tasks; for example, for an ecommerce website, we might test the usability of adding an item to the shopping cart and completing the checkout process.
One-on-one testing takes time and a facilitator. What if, to save time and money, we tried to have users report usability problems – would they be able to do it? Of course it depends! Participants who frequently participate in usability testing for services such as usertesting.com have a disproportinate amount of experience completing tasks and articulating any hiccups that they encounter along the way. They are more likely to be able to describe their mental model of what they expected and how what they encountered was different than what they expected. However, these participants have so much experience with different systems and design patterns that they may no longer be representative of the general population, but instead representative of the extremely tech savvy, or of amateur UX evaluators. Armed with this experience, these types of participants may encounter a different set of usability problems compared to the general population.
Those with less awareness of their mental model and less experience articulating it won’t be able to report usability problems. A case in point: in a recent quantitative diary study with a few hundred participants in a gen pop sample, a market researcher measured agreement with the statement “This app is easy to use.” Perhaps unsurprisingly, most participants reported the app was extremely easy to use! Behavioral data showed that participants were not able to complete basic tasks. Also, dozens spontaneously asked for instructions for the app. The behavioral data and requests for instructions directly contradict the self-report data. Why is this? Participants may’ve genuinely experienced the app as easy to use and didn’t realize that they weren’t able to complete basic tasks. Based on literature that says that Western people feel competent and don’t realize it when they aren’t [1], participants may feel confident in their abilities and thus don’t entertain the possibility that they weren’t able to figure out an interface. The bottom line is that self-reported usability data are easy and cheap to measure but most likely misleading. As we say in our group, bad data is worse than no data. Observing the behavior of representative participants is a gold standard in usability testing and should remain so.
[1] Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of personality and social psychology, 77(6), 1121.
By Frank Bentley, Senior Principal User Researcher
Often a design or product team will have a question in the course of their work where an immediate answer is needed in order to proceed. Perhaps they want to know if it’s worth it to support a feature given people’s current behaviors. Or perhaps they want to test some different icons for intelligibility. Ideally, they’d have answers in a few hours so that they could continue their work. Larger product teams might have multiple questions like this per week in the course of their work.
However traditional User Research activities take time. In-person studies have to be planned, participants need to be recruited and data needs to be analyzed. This can typically take a week or more. Large market research studies often take even longer and are very expensive. But product teams in agile environments need to work faster than these traditional methods can support.
Over the past two years, we’ve developed a process for quickly designing, running and analyzing mid-scale surveys to support design and product questions within hours or a day at most. Frequently we can have a question one afternoon, run the survey that evening and have data by the morning of what a representative sample of Americans do. We can run dozens of these surveys every month to understand which use cases to support, where current frustrations lie and the intelligibility of designs. The key is to have a few focused questions and to deploy the survey using a large online survey platform (e.g. Survey Monkey or Amazon’s MTurk) to a broad audience. These are not marketing-style attitudinal questions, but questions on actual behaviors that people engage in as a part of their everyday life, so that we can quickly quantify whether a use case is worth supporting. This method has helped us to learn that over 90% of Americans receive coupons and deals in their email and that 64% had used one in the past two weeks. We’ve also learned about the percentage of email users who turn off their notifications by platform and have tested a variety of icons for the Stationary feature of Yahoo Mail, finding one with 7x the recognition rate of the current design within hours of receiving the files from a designer.
Behavioral surveys like these help User Research to work at the speeds of agile teams and provide quick data about user behavior in the wild to designers and product managers to help set product strategy. And we’ve seen that the results are often within just a few percentage points of larger, much more expensive market research studies that can take months to fully execute. In a study using eight questions covering a range of user behaviors (everything from using coupons received in email to enabling mobile mail notifications to subscribing to particular video streaming sites), we observed an average error of only 3.6% with SurveyMonkey panels and 7.1% with MTurk panels compared to a gold-standard n=1000 market research panel. This is well within the margin of error needed for rapid feedback in the design process. If you have $100 (the approximate cost of a user study) and 2-3 hours, you can quickly get representative data from 150 participants and move design along at a rapid pace! Running multiple studies over time and triangulating findings can serve to further increase confidence in the results.
By Mike Shebanek, Senior Director of Accessibility
Starting today, you’ll notice some significant changes to our blog, both in design and in content, that better reflect who we are and what we do. I’d like to to tell you why we’re making these changes and what you can look forward to as you follow our blog.
One of the things that makes Yahoo’s approach to accessibility unique is that its Accessibility Team and its User Experience Research Team are integrated. That may sound like an obvious approach, but it hasn’t always been this way at Yahoo and we’ve come to find out it’s pretty unusual in the tech industry. In the all-too-rare organizations that actually have a dedicated Accessibility Team, they often report to the chief legal counsel, or are part of an administrative or compliance group. Other times they’re siloed in one division, which prevents them from being able to impact product development or customer support in another. Having a dedicated Accessibility Team and where that team is placed in an organization says a lot about a company’s values, and this is especially true of Yahoo.
In our case, we’re part of a team called User Research and Accessibility or UXRA for short. Our team supports the entire company – every division, every VP, every employee – and it is tasked with representing our users’ unique needs, requirements and points of view in every activity we undertake. We are not just “closely aligned” with our user research team – we are the same team! We share user study labs, training, techniques, findings and the same mission: to accurately represent all users. We work to ensure that users come first in whatever we do, and that this includes people with disabilities, of all ages, of all genders and of all abilities. So, whenever we perform a user study, take a survey, gather product feedback or identify requirements, people with disabilities are included along with everyone else. We can’t imagine any other way.
To better reflect this mission and to give you deeper insight into how we go about understanding and representing our users, you’ll be hearing more from our User Researchers in addition to our Accessibility Team members on this blog. We have a lot of exciting things to share, and we hope it is informative and inspiring to you and our industry peers as we all continue to reach for greater inclusion.
