How Collected Over 120 User Tests While Still in Stealth

When CPO Jon Loyens and his co-founders Brett Hurt (CEO), Matt Laessig (COO), and Bryon Jacob (CTO) set out to build, even with nearly a dozen startups under their belts (most notably with Brett as the Founder of Bazaarvoice, where 3/4 of the team first assembled; and Matt, Bryon and Jon as execs at HomeAway), they knew their vision was a lofty one. They’d tasked themselves with building a social network that would make data easier to find, use, organize, and collaborate on.

And it wasn’t just a platform they had their sights on, but a community-driven movement to remove the friction data scientists, research institutions, governments and others felt when collaborating on data projects. From cancer research projects to market data collected by financial institutions, historically, anyone who touched data went through the same machinations when working on a project: first came the administrative time collecting and organizing data, followed by collaboration on that data with colleagues via email, Slack, IMs, and hallway conversations. And it’d start up all over again when the project passed on to the next person in line: a time suck well before any actual analysis could take place. is now live and open to users, though the team’s been working for 9 months to get to where they are today. We talked to Jon about how they got to this point, and how their use of a flexible, functional demo evolved their product.

Just Real Enough

In late 2015, Jon sat with his newborn twins tucked comfortably under his arms on his couch, Mac in lap, working feverishly on the finishing touches for’s first product demo. As Chief Product Officer, Jon and CTO Bryon Jacob, friends and colleagues for 16 years, were getting ready to roll out their demo to share on an investor roadshow Brett Hurt, CEO, had been orchestrating.

Jon’s varied career up to this point (he’d hopped between engineering to product and back again at both big and small companies), had tee’d him up to expertly stage product demos. He has a reverence for user-centered design, and aimed to combine design, metrics and product management in equal measure at

So while working on the first demo, he instinctively knew this was going to be something users would need to see, click, and play with to fully grasp the power. “I’ve always been a big fan of a clickable demo, of more usability testing, of user-centered design – anything to put the person that you’re showing the tech to in the frame of mind of the end user. They need to use it to see the potential.” While a picture is worth a thousand words, Jon muses, something you can touch and feel is worth a million.

It needs to be “real,” Jon says, but only just real enough to be convincing. “Building a great demo is like building a great Hollywood set – you build just enough to make the actors and the audience feel like they’re in that environment.”

When the Duct Tape Holds 

So Jon and Bryon began building with the goal of getting to a place where they could begin user testing. They focused on making the demo usable enough to show the magic of what they were creating: a place where data was easier to find, use, organize and collaborate on. They decided the key would be showing a file being uploaded and making the data instantly queryable and linkable. So with some “duct tape and bailing wire,” they built the demo with a live graph database running on the backend of local machines to show the user’s journey. Once the user saw the potential by clicking and interacting, the duct tape and bailing wire behind the curtain didn’t matter – the user was already sold.

Demo built, the team amassed 126 user feedback sessions while still in stealth mode, holding 2 interviews a day for the first quarter they were in business. To land interviews, they relied heavily on their networks across researchers, academia, and public and private institutions, along with the connections of some of their closest advisors and investors. And those early users were key: “We believe heavily in a constant cycle of feedback” Jon says.

Throughout this constant feedback loop, keeping the company’s vision in mind to avoid getting bogged down by the constant influx of recommendations was crucial. “Part of the initial art of building a demo is knowing what’s important and what can be done within a certain time frame” Jon says. With over 100 interviews, patterns emerge around a few key items that can be honed in on. To get specific, valuable, actionable feedback, Jon suggests designing tests and interviews to focus on the questions that need answered.

Maslow’s Hierarchy of Data Collaboration 

After spending hundreds of hours collecting their own data on users, what did Jon and his team learn? They uncovered their own “Maslow’s Hierarchy of Needs” for data collaboration: demo users, the team found, had a base need that had to be solved first – in this case that need was a social network of data professionals. “When it comes to collaborating on data projects,” Jon says, “people are still looking for food, shelter and water – they’re not looking for self actualization.”

With this in mind, they developed a go-to-market strategy targeted at satisfying the base level user need by building the network first.

Plus, Jon says, listening to what users say they want versus watching how they actually interact with a product can be at odds. “This is where the combination of metrics and analytics in concert with user-centered design starts to be really important,” Jon says. He’s guided his team to keep watch of specific features, how they’re used, and how they can be improved. In the end, the usage metrics and analytics don’t lie.

The team operates with complete transparency, where each member observes testing, accesses analytics, and watches how people use the platform. “Too many founders and product management teams get caught up in the Jobsian mentality where they think they’ve created a great vision and know exactly what the market needs,” Jon says. His team works counter to this thinking, and with the base level need now satisfied, they can focus on observing how the user interacts with the product to guide much of their continued development.

Feedback Strategy 2.0 

Post-launch, the team continues to collect feedback, though under a different strategy than with the demo. They still seek it out, but now after the launch, more often that feedback is coming directly to them. “We can gather incredible insight into how people are using the platform as more and more people join,” Jon says. The team now selects members in the community to do live platform walk-throughs, while also keeping a close watch on analytics of how people join, retention rates, and how they use the product. The users who are stumbling – those who are submitting support tickets, providing feedback, and contacting them on Twitter – now have the same high value as those first demo users who helped shape the initial product. 

A just-real-enough demo, hundreds of early testers, and an influx of analytics post-launch doesn’t mean, however, that’s platform is complete. “As a product and engineering leader,” Jon says, “one of the worst things you can do is to consider something done.”

Lindsay Knight

Posted On

August 31, 2016


, ,

Share This