Update (12/12/17): Our work on Growing Up NYC won an award for “Best External Application” at “The Best of New York City 2017 Awards” at The NYC Technology Forum in Brooklyn, NY. Find out more here.
For digital leaders at many nonprofits, government institutions and brands, the idea of building a prototype and testing it with users can be both exciting and daunting.
On the one hand, getting feedback on an early prototype sounds like a great way to learn if your product ideas will actually deliver value to your audiences. On the other, this way of working can conflict with the operational culture of more traditional organizations, which tend to only reveal finished, polished products to their audiences. While the appetite for lean startup methodologies like prototype testing has obviously grown beyond startups working in Silicon Valley, selling them internally remains a challenge.
The sell can be more difficult still when the product needs to launch by a certain date. How do you simultaneously build a prototype and test it with users, which will likely uncover problems that need to be fixed, while making progress toward launch? Will prototyping and testing cause delays in shipping the product?
We recently faced these challenges on our work on Growing Up NYC, a platform that helps parents and families find programs, resources, and activities for kids. While the approach we developed was specific to this project, the following tips apply to instances where you’re looking to validate an MVP through prototype testing and still launch on time.
1) Share a lean mindset.
At the start of our work on Growing Up NYC, we knew our clients from the Mayor’s Office wanted to put prototypes in front of users, given the City’s requirement to involve New Yorkers in the development of new digital products and services. However, because iterating based on what you learn could in theory go on forever, it was key for us all to align on the notion of designing and building an MVP — a first product release that would deliver value to our NYC audiences, while not containing every single feature we dreamed up. We had a shared mindset that guided us through the project: We would test and iterate, but also ship by a firm date, prioritizing based on what we learned and not letting the perfect be the enemy of the good.
2) Plan and commit to a testing day.
After kickoff, we worked with our client to prioritize the features we believed were crucial for the first release of Growing Up NYC. We began designing the experience, working in two-week sprints. We decided to hold our testing days toward the end of sprints — right before design would hand off to development. Testing our prototype at that point gave the team enough time to update designs after tests, allowing us to incorporate the highest-priority findings into the current development sprint.
Sprint cadence with prototype testing toward the end of sprints
3) Align on what you want to learn.
As we approached each testing day, we created a script that we would use to walk test participants through the prototype. Crucially, the script writing process forced us to align on the assumptions we were most interested in testing. It’s natural to feel the need to prototype and test everything. But to make testing a repeatable process, it’s useful to give yourself and your team the permission to focus. Instead of boiling the ocean, be explicit about what you want to learn and prototype the parts of the experience that will get you those learnings.
Prototype of Growing Up NYC
4) Design with real content.
When prototyping, there’s always the question of fidelity — how refined and true to life do you make content, data, visual design, and interaction? For our prototype of Growing Up NYC, we went high-fidelity on content — from headlines and help text to the events and resources that would be part of the final build. While there’s a time and place for “lorem ipsum,” usability testing during an agile development process is not one of them, as copy is such a huge part of any digital experience. This approach also had the benefit of forcing the team to think much earlier in the process about the finer points of content, including the metadata for each module. These questions can often hold up launch if left to the end.
5) Choose the path of least resistance for user recruitment.
Our clients in the Mayor’s Office coordinated with various branches of the New York Public Library to enable us to set up a testing area and recruit patrons on the spot. This allowed us to test with five to eight users in each two-hour session — no scheduling or follow-up required. This was crucial, as recruitment and test scheduling have the potential to be a huge time suck. Indeed, the idea of making user tests a repeatable part of a sprint schedule is sometimes met with skepticism for this very reason. To overcome this roadblock, pursue the recruitment method that requires the least legwork to find your target audience.
6) Invite the whole team to participate.
Scrum rituals like standups and retrospectives promote visibility into what everyone is doing, awareness of challenges to be solved, and shared investment in the outcome of the project. Usability testing, when coupled with an open door policy allowing anyone on the team to participate, can offer the same. On Growing Up NYC, we had representatives from all disciplines — as well the client team — rolling up their sleeves to test together.
Members of the BSD and NYC Mayor’s Office teams conducting a user test at the Harlem Library
7) Report your findings in real time.
Following each round of testing, we didn’t pack up and go home for the day. Rather, we sat in a room with our clients from the Mayor’s Office and talked about what we observed. Each person proctoring and taking notes shared their top three takeaways from the tests. This conversation allowed us to quickly identify the most salient problems with our prototype. Plus, getting everyone in the room together allowed us to forgo a lengthy reporting process and get right back to work.
8) Prioritize, prioritize, prioritize.
Putting your prototype in front of users is bound to stir up ideas and solutions the team hadn’t considered before, but not every idea should be implemented now or ever. Using a simple framework, we made decisions together on what design solutions to pursue. Our primary criterion was whether each solution was “high impact,” i.e. whether it would fix the biggest problems observed during testing. Then, we determined which of those high-impact items were “low effort,” i.e. possible to accomplish in the current sprint or before launch. High-impact items that required a too-high effort were slated into later testing rounds or added to the backlog for a future release.
Prioritization framework
Takeaway: Testing doesn’t have to slow you down.
Testing a prototype while making progress toward launch can be tricky — and this is just one of the hurdles faced by digital leaders looking to innovate and bring lean methodologies to their organizations. But with the right ideas and partners, prototyping doesn’t have to slow you down. It’s possible to move quickly on development and validate your MVP at the same time. Doing so gives you and your organization the best shot of delivering a product that will resonate with your audiences.
Daniel Kalick is the Principal UX Designer in the NY office.