Prior to 2020, our research team in Tokyo rarely had reason to conduct remote interviews.
The pandemic changed that.
Remote has since become the default, and it will continue to be the case for the foreseeable future.
The transition to remote studies was, for some projects, smooth: “only” requiring that we take interviews online over Zoom or the phone. Other studies were not so easily converted.
What about research that requires equipment or technology that participants don’t have on hand?
Over the past year, the AQ research team has gone through rounds of experimentation to establish operations for getting a lab environment to participants’ homes and offices as efficiently and reliably as possible.
Here’s what we’ve figured out so far.
A note on context: AQ runs B2B and B2C qualitative research for mostly digital solutions in the Japanese market, with a scaffolded recruiting process. That means access to reliable shipping options, very low no-show rates and moderate-to-high familiarity with technology. We recognize that it’s a particular environment for remote research. For example, if we ship an iPad to a participant, we can reasonably expect it to arrive within the reserved timeframe, and, that we’ll get it back. That’s a lot of potential points of failure!
That context may look different from yours, but many of the considerations needed to design for a respectful, thoughtful participant experience are transferrable, with variety being in the execution. These are the lessons we’ve learned from our efforts, and we look forward to learning from yours, too.
An example of a study that stumped us at first was one where we were working in the maritime navigational technology field. We had been part of this product team since late 2019, and in this project’s latest phase, had a prototype ready for onboard testing with ship captains and crew members. Not willing to give up on getting customer feedback at this critical moment in product development, we set out to find a way.
The original plan had been to install the prototype on the bridge so that we could carry out observations and interviews in a true-to-life environment.
In the end, we decided to hold off on the parts of the study that could only be run onboard or in person. We re-scoped the study to focus on the software itself, and collected impressions of key features and evaluations of the UI.
As remote research relies on video conferencing services like Zoom, ensuring sufficient Internet connectivity on the participants’ side is a key step in our preparation.
Depending on the target audience, it’s possible to prioritize recruiting for candidates with a stable home Internet connection and experience with online video conferencing software. When this is not an option, we need to assess the environment in which the participant will take the call, and provide additional equipment to maximize the chances of a good connection.
In the project mentioned above, we shipped a pocket-sized WiFi router and a tablet device to each participant. This allowed them to chat with us while moving freely around their space, whether that was a home, an office, or a tanker ship parked in Tokyo Bay.
In addition to conducting tech check calls prior to the interview itself, we’ve since added questions about Internet set-ups in the recruiting process.
Connection isn’t the only consideration. Sometimes studies require that participants test software on specific devices, in which case we’ll send tablets or smartphones. We’ve also shipped headsets, standalone microphones, cameras, tripods, extension cables, phone chargers and more!
In the lab, studies would only require one setup’s worth of devices and equipment. For a remote study, we typically need to ship the same setup to each participant. That requires more upfront planning for costs to be accounted for in project budgets, and shipping to be accounted for in project schedules. We have made bulk purchases for smaller items like cables or equipment that can be re-used across multiple projects. For equipment that’s project-specific, we rely on rental services.
For projects when we want participants to test a product on specific screen sizes or OS versions, the first step is to look for a rental company that has enough stock of particular models and make a reservation.
Getting a head start is crucial, as a first-time rental often requires submitting paperwork for an approval process to register our company. And availability written on a rental company website isn’t always up to date. We’ve had to scramble a few times to find other options, either a different company or a wider range of models.
Another thing to consider is what happens if we’re not able to return any of the devices, due to accidents or miscommunication with participants. We would then owe the rental company one (potentially expensive) tablet. It’s best to research insurance policies and potential expenses before such an incident occurs.
Audio technology has come a long way but it still feels like we encounter issues with howling sounds every other interview. That’s a slight exaggeration but to be certain we catch everything that participants says, we now consider earphones with a built-in microphones to be a requirement. Wired earphones are sufficient when the participant is seated. In studies where the participant is expected to walk around their environment, we provide Bluetooth earphones with resilient battery life.
Note that not all devices have a standard 3.5 mm headphone jack. One time we had to make a last-minute shipment of iPad lightning-to-headphone jack adapters to all participants.
Bluetooth headsets need to be paired to a device, which can be finicky and not something we want participants to be doing at the beginning of the interview. We either ask them to complete it ahead of time or walk them through it during the tech check call.
Rental devices come with all sorts of apps installed, OS versions not up to date, or arranged in a way that isn’t conducive to the study. To remove friction for participants, we run OS-level software updates, install relevant applications, and arrange the home screen so that participants can easily find the apps required for the interview. The Notes app is perfect for providing Zoom links and additional instructions.
There are times when a particular prototyping app doesn’t work on the day — it’s rare but it happens. It’s stressful to troubleshoot tech from afar while the clock is running, and may have an averse effect on the interview. To have an alternative solution in our pocket, we save PDFs or videos of the UI on the test devices before shipping it to participants. This kind of preparation goes a long way in providing peace of mind so that we can focus on the participant.
Participants aren’t necessarily familiar with the test devices and apps. While preparing the interview guide, we note down all of the UI label copy and steps required, so that our guidance is crystal clear.
For example, switching between Zoom and a screen-shared browser can be tricky. If a participant feels discomfort or doubt before we get to the app or prototype in question, it may affect how they respond. The key is to facilitate a sense of separation between the set-up process and the interview itself. We often set aside time to walk participants through a relaxed dry run of the tech.
While a participants’ main concern for in-person lab research is to get themselves there, there’s a lot more work required for remote interviews. Becoming a participant includes agreeing to do things like:
To avoid unpleasant surprises for either side, we set these expectations early in the recruiting process.
Once all of the equipment is ready, the next task is to package it up for shipping.
This piece of paper is placed on top, so it’s the first thing participants see when they open the box. It has a complete list of the contents of the box (down to the cables, attachments, and accessories) along with instructions regarding preparation and return shipping. The same content will be e-mailed to them, too.
The amount of effort and eye for detail required to run these kinds of operations tends to require more time and energy than we expect. Our recommendation is not to do it alone. Do pair up with a buddy and plan in advance with plenty of scenario mapping, so that when it’s time to moderate the interview, everything is in place.
Respectful preparation leads to smooth cooperation — maybe they’ll even enclose a friendly thank you note in the package returned to you!
🌻 We have a decade of experience with user research and design in Japan. Interested in understanding your current and future customers in Japan? See our research services and reach out to us for a chat.
This article was written by Eiko Nagase and translated into English by Johnny Linnert and Tomomi Sasaki.
