How to Sell Digital Marketing to a Skeptic

 01

The critical question about the metaverse that no one is asking

We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 - 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!

As the metaverse becomes an inevitable part of our future, little is discussed about the ethical implications for marketers entering this new frontier. As we head into 2022, the conversation surrounding this new digital landscape will only accelerate and deepen, with its fair share of cheerleaders and skeptics. On the one hand, I read articles every day from fellow marketing executives salivating at the thought of advertising to people — mostly youth — in the metaverse. On the other, figures like Elon Musk, Charlamagne tha God and Scott Galloway are doubtful of or outright hostile to the hype behind things like Web3, DeFi, NFTs and, yes, the metaverse. 

No matter which side you fall on, I think we can all agree that a serious discussion needs to be had about marketing in the metaverse. What disturbs me is the lack of both prudent, incisive articles about where the humanist line is, and questioning how far we are willing to invade any kind of space. As a digital marketer with decades of experience running an award-winning agency, I’ve seen how marketing has been radically transformed by technology — for the good and the bad. 

Since 2001, I’ve built my business on lead intelligence. In that process, I’ve used technology that allows me to put tracking code on peoples’ devices to monitor their engagement levels and literally score them on certain behaviors. When they’re not behaving in a way that benefits my clients, it’s my job to use persuasive content to push them in a direction that would make them more likely buyers. 

So I understand the powerful sword I’ve been wielding to sway and shape minds. In the most ideal sense, I’ve done it with the aim of getting the right message to the right people at the right time to make educated buying decisions. But as we’ve seen, the same tools and tactics that can be used for good are the same tools and tactics that have persuaded people to act on and believe in things they previously wouldn’t have. If we’re to gleefully step into a new world like the metaverse and the power is in the wrong hands, an important question must be raised: is this right? Can we go too far? And can the same tools that persuade in a noble way also be used to manipulate both impressionable people and, more critically, the youth?

The answer, simply put, is yes. We are just now coming to grips with the implications of huge social media platforms to steer discourse and impact the next generation’s mental health. Repeated intrusions into user privacy, rampant misinformation, and pinpoint-accurate political advertising has left regulators wondering about the extent to which, like telecoms, social media may need oversight. For the rest of us the question is: should the likes of Mark Zuckerberg, Microsoft and other Silicon Valley executives really be our shepherds into this brave new world?

A perfect storm: automation, UBI, and AI

While we linger on that question, let’s take it one step further to see why the metaverse may become a part of daily life. Unless you’ve been living under a rock, you’ve seen how automation coupled with the Great Resignation and remote work has changed how businesses will use human capital in the future. According to Tristan Harris, director of the Netflix hit The Social Dilemma, it’s almost inevitable that 30% of Americans could be “radically unemployable” within the next decade.

You’re starting to see this everywhere you look. Kiosks, self-checkout, and self-ordering on iPads are taking over retail and restaurants. The advent of self-driving cars could leave commercial truckers and Uber drivers without work. AI and robotics have even been rumored to someday replace radiologists and other medical personnel.

Only the most specialized, creative, and innovative jobs will retain some level of human touch and those employees can pivot during this sea change. But a large chunk of blue collar and service-sector professions, which together make up nearly 85% of American jobs, will suffer. As the number of permanently unemployed Americans balloons, there will need to be an ever-widening social safety net, including the once-ridiculed possibility of universal basic income (which Andrew Yang popularized during his 2020 campaign) along with expanded welfare, Medicare and related programs.

So how does this perfect storm of automation, AI and UBI play into the idea of the metaverse? Let’s remember where the origin of the term metaverse was coined: a 1992 science fiction novel titled Snow Crash where people were provided a refuge from the dystopian reality that they were living in in the form of digital avatars they could use to explore the online world. 

Studies show Americans derive a lot of their meaning or purpose in life from work. The term “workism” is used to describe the phenomenon among Americans that work is not only a means to an economic end, but is a pillar of identity from which we derive much of our meaning. So how does a world where that’s increasingly absent look? And furthermore, where will we look for meaning? The metaverse could be that compelling place when it’s finally seamless: the moment where you can’t tell what’s real and what’s virtual. An alternate universe indistinguishable from your life.

Augmented reality is a prelude to this as I’ll discuss below, and stands in as a kind of boiling frog for our current situation where the failure to act against a problematic situation until it’s already too late is clear.

When all of this does unfold and tech giants make people painfully addicted to the metaverse, marketers will do what they do best: exploit our data, time, attention and privacy for profit. 

Just because we can, should we? Lessons from Jurassic Park

“You were so preoccupied with whether or not you could, you didn’t stop to think if you should.” –– Dr. Ian Malcom from Jurassic Park 

So here it is, the critical question about the metaverse that no one is asking: just because we can, should we? On a personal level, I wish that the metaverse wouldn’t exist; that the bad far outweighs the good in terms of mental health and the ever-widening divide in human contact. But I’m not naive enough to think that it’s anything but inevitable. Maybe not in its current form, but definitely in the future, the metaverse will be here to stay. What I do know is that whether this thing succeeds or fails will be because of people like me — marketers and advertisers yearning for exposure for our clients.

Let’s be clear: all social media platforms monetize their users by selling data to advertisers. If the metaverse was just a place where you could go ad-free, it wouldn’t be able to sustain itself — there’s no revenue. Therefore, people like me who are against it belong to a larger group of people who will keep it alive if everyday people make the choice to utilize it. Even if we market for noble purposes, it’ll just keep the metaverse alive long enough for the bad actors to come in and exploit the system.

Knowing my role in this, I don’t think that I’m in a position to demand that individuals not use the metaverse. What I am saying, though, is to act with caution. Know that you’re way more susceptible than you’ve ever been. And if you’re a marketer, know that the next shift of humanity could be radically changed by this platform if we don’t use it responsibly. Google’s motto was once Don’t be evil. With social media, we all gleefully dove in with no thought of consequences. This is our chance to not let history repeat itself.  

It’s already too late to reverse the damage on current social media platforms. Let’s not make the same mistake twice. The power of history is when people learn from their mistakes, especially as we simply cannot fathom the power of this yet. Combined with AI and machine learning, the metaverse can quickly spiral out of our control by adapting to our next move in a way that is so incredibly scary. The pause between our response and the act is the only thing currently separating us from going down the path of destruction as fast as we can. But the adjustment will eventually be seamless, without a moral code to check or monitor it and ask: is this the right thing?

The pace of adoption

My hope is that the hardware issue in VR is slowing the pace of this down just enough for us to get our act together. Mass adoption hasn’t occurred yet, as evidenced by the sales numbers of the Oculus VR headset in comparison to more traditional, handheld competitors like the Nintendo Switch.

We saw the early failure of 3D TV eyewear for home use and in theatres. It was too cumbersome and made our eyes hurt. Even VR in its current form is uncomfortable and can make some feel sick to their stomach. As long as the hardware is a roadblock toward true mass adoption, things like VR and 3D TV will be fun to dabble in as a novelty and little else.

But while everyone is waiting for a truly immersive and fully-sensory VR experience, I believe that the gateway drug for entering the metaverse will be augmented reality or AR. People will buy into this as they’ve done with the wildly-popular Pokémon Go or in recent museum exhibits, and AR will pave the way until VR is seamless. AR breaks the barrier to allow us to live in a hybrid world. Once adoption happens — and if the world around us isn’t as compelling anymore — it’ll expedite our willingness to go deeper and more virtual. AR feels safer currently because to live in a world that is 90% real and 10% fake is still doable. But as we warned in the boiling frog analogy above, that 10% will take up more and more real estate until the jump to VR or something like Elon Musk’s Neuralink will be the natural progression. At that time the hardware and experience will be so realistic, it’ll be indistinguishable from the real world.

The silver lining: The possibility of the metaverse as a force for good 

“Instead of collective power we ended up with mass exploitation” –– Krystal Ball

If this past year has proven anything, there’s a hunger for more decentralized, user-driven platforms. If the metaverse can stay decentralized, it could be a true force for good that flips the current social media paradigm of the person as a product for data collection and targeting, instead of putting data solely in the hands of the user to monetize as they wish. If we let big tech be the on-ramp, they’ll be the big winners at our expense.

My suggestion is that we take this as seriously as the Manhattan Project. If history is any indicator of how slowly government moves — remember how long it took to mandate seatbelts in cars? — we don’t have the time to spend years trying to discover viability, this will be upon us in 3-5 years. To quote one of the Winklevoss twins in the movie The Social Network about how quickly early Facebook took off: “If I was a drug dealer, I couldn’t give free drugs to 650 people in one day.” The next decade’s drug, if we choose to take it, is a serious problem for humanity and a serious problem for the ethics of marketing. 

The metaverse is here whether we like it or not, but we must use it responsibly. Congress is just now trying to figure out how to regulate social media; they are so woefully behind on this new landscape.

In the absence of regulation, we as marketers are going to have to stand in the gap and say: Knowing that the metaverse will probably be the wild west, it’s our choice whether we want this new frontier to just repeat history, or to become a brave new future. I hope we choose the latter.

Chris Carr is the President and Founder of Farotech

DataDecisionMakers

Welcome to the VentureBeat community!

DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.

If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.

You might even consider contributing an article of your own!

Read More From DataDecisionMakers

02

The Wolf of Crypto

To hear more audio stories from publications like The New York Times, download Audm for iPhone or Android.

MIAMI BEACH — Jordan Belfort was lounging by the pool on a sunny April morning, sipping Red Bull and sharing a cautionary tale. Not the usual one about his imprisonment on 10 counts of securities fraud and money laundering: This time, he’d been the victim. Last fall, he explained to a group of businessmen gathered at his palatial home, a hacker had stolen $300,000 of digital tokens from his cryptocurrency wallet.

He had gotten the bad news at dinner on a Friday, he said, while he was telling a venture-capitalist friend about the time he sank his yacht during a drug-fueled romp in the mid-90s. After breaking into Mr. Belfort’s account, the hacker transferred large quantities of Ohm, a popular cryptocurrency token, to a separate wallet — a publicly visible transaction Mr. Belfort could do nothing to reverse. “You can see where the money is,” he said. “It’s the most frustrating thing.”

Mr. Belfort, 59, is best known for “The Wolf of Wall Street,” a tell-all memoir about his debauched 1990s career in high finance, which the director Martin Scorsese adapted into a 2013 movie starring Leonardo DiCaprio as the hard-partying protagonist. These days, the real-life Mr. Belfort is a consultant and sales coach, charging tens of thousands of dollars for private sessions.

This month, at his house in Miami Beach, he hosted nine blockchain enthusiasts and entrepreneurs for a weekend-long crypto workshop — a chance to hang out with the Wolf and enjoy an “intimate financial experience” with his crypto-industry friends.

A long line of celebrities has tried to profit from the cryptocurrency boom, appearing in widely mocked crypto commercials or flogging nonfungible tokens, the unique digital collectibles known as NFTs. Mr. Belfort said he has refused to participate in the worst of the shilling. He has declined offers to launch a line of Wolf-themed NFTs, he said, even though “I could easily make $10 million.”

He is also a recent convert away from crypto skepticism. Not long ago, he shot a YouTube video about the dangers of Bitcoin, which he called “frickin’ insanity” and “mass delusion.” Over the years, he said, he gradually changed his mind, as he learned more about cryptocurrencies and prices skyrocketed.

Now, Mr. Belfort is an investor in a handful of start-ups, including a new NFT platform and an animal-themed crypto project that he said is “trying to take the dog-and-pet ecosystem and put it onto the blockchain.”

Whatever his crypto bona fides, Mr. Belfort is unquestionably qualified to discuss the subject of financial fraud, a major problem in the digital-asset industry. In the 1990s, the firm he founded, Stratton Oakmont, operated a sophisticated stock-manipulation scheme. At the height of their wealth, he and his business partners consumed enormous quantities of cocaine and quaaludes and regularly employed prostitutes. Mr. Belfort eventually served 22 months in prison.

Given that history, it can feel slightly surreal to hear an older, more grizzled Mr. Belfort proclaim that he is “massively looking forward to regulation” in the crypto industry. “I’m not interested in separating people from their money,” he said. “That’s the opposite of how I act right now.”

Still, the crypto workshop at his house was not free: Guests paid one Bitcoin for a seat, or the cash equivalent, which is roughly $40,000.

The workshop began at 9 a.m. on Saturday. The guests — chosen from a pool of more than 600 applicants — milled around Mr. Belfort’s backyard, eating made-to-order omelets and trading tips about Bitcoin mining and tokenomics. A crypto miner from Kazakhstan relaxed in the sun with an aspiring blockchain influencer who runs a roofing company in Idaho. A Florida businessman explained his plan to use NFTs in a start-up that he’s pitching as Tinder for music. Some of the guests said they paid for the workshop because they are die-hard fans of the Wolf; others simply wanted to network with fellow entrepreneurs.

By 9:15 a.m., the mimosas were flowing, but Mr. Belfort was nowhere to be seen. “The U.S. dollar is going to crap,” said the roofing executive, Doug Bartlett. A few minutes passed. Still no Wolf. “The Wolf is still sleeping?” one guest wondered aloud.

At last, Mr. Belfort emerged from the house, wearing faded jeans and dark sunglasses. Mr. Belfort has short dark hair; he’s more wrinkled than he was in the ’90s, but his face is still set in a perpetually boyish grin. He stopped on the staircase down from the porch to survey the scene: nine men dressed in various shades of business casual — polo shirts, flip-flops, untucked button-down shirts. “I guess we still need to work on feminine adoption of cryptocurrency,” he said. “We got to get some girls here next year.” He paused. “Women.”

Someone handed Mr. Belfort a can of Red Bull. (It was about 9:30 a.m.) “I’m gonna need the sugar,” he said. After a few minutes of chitchat, he ushered the group into the dining room, where each place at the table was set with a notebook and a copy of “Way of the Wolf,” a sales manual Mr. Belfort published in 2017.

Mr. Belfort has spent the past two decades trying to rebuild his reputation, but signs of the old Wolf were everywhere. Behind his spot at the head of the table, a fully stocked liquor shelf took up most of the wall. (He hasn’t gotten high in 25 years, he said, but he sometimes drinks.) Next to the shelf hung a poster designed to resemble an entry on the periodic table — Qu for quaalude — listing various “drug facts,” including “best sex ever.”

After a round of introductions, Mr. Belfort began a lecture on the minutiae of cryptocurrencies, from the differences between Bitcoin and Ethereum to the rise of decentralized autonomous organizations. He shared his wisdom on crypto-based “smart contract” systems (“some of them are really smart; some of them are stupid”) and recounted old stories about his collaboration with Leo and Marty.

“Leo had never done drugs,” he said. “I had to educate him on that.”

For a gathering of crypto evangelists, it was striking how much time everyone spent reliving their biggest losses. Nearly half the group said they’d been hacked. One guest said he’d lost money when the cryptocurrency exchange Mt. Gox collapsed in 2014. Two others said they’d burned large quantities of tokens in risky trades.

The energy in the room lifted with the arrival of Chase Hero, one of a series of guest speakers Mr. Belfort had recruited for the weekend. A crypto investor and gaming enthusiast, Mr. Hero declared that stablecoins — cryptocurrencies whose value is pegged to the U.S. dollar — are “the biggest innovation since sliced bread.”

“It seems vivacious and insane and almost borderline a Ponzi scheme,” Mr. Hero said of his favorite stablecoin project. “Which makes it the perfect asset for cryptocurrency because that’s what these kids love.”

One of Mr. Belfort’s guests, Svein-Erik Nilsen, a Norwegian entrepreneur, started describing his own business ambitions. Did Mr. Hero have any tips? The key to starting a new venture, he replied, is aggressive marketing to stand out from the crowd. “Imagine going to a Brazilian beach and trying to find one single hot chick. There’s eight million,” Mr. Hero said. “The idea is the same thing here. You have to do stupid, dumb marketing to get it out there.”

A few hours later, the group adjourned for dinner at Carbone, a high-end Italian restaurant in Miami Beach where Mr. Belfort eats as often as twice a week. As they dined on caviar and rigatoni, some of the guests shared stories of their own debauchery; Mr. Belfort, it turned out, was not the only wolf in the room. Two guests discussed the mechanics of pursuing younger women without risking entanglement in a “sugar baby” situation. Someone speculated about how an enterprising strip club owner might incorporate NFTs into the business.

Soon conversation turned to a club in Japan where women are said to cavort with octopuses. Mr. Belfort wanted to know more: Were the women in Japan beautiful? Later, he showed the group an iPhone video he took at an S-and-M-themed bar, where the waitresses flog the customers.

Artem Bespaloff, the chief executive of the crypto mining company Asic Jungle, leaned across the table to describe his personal conversion to the way of the Wolf. He was planning to go to medical school, he said, when he found a copy of “The Wolf of Wall Street” at the library.

“I said, ‘This is what I want to do,’” Mr. Bespaloff recalled. “I ended up stealing the book from the library.”

“So I was a good influence,” Mr. Belfort said with a laugh. Still, he said, he regrets his behavior in those days — it was wrong, and he could have gotten even richer if he hadn’t broken the law. “I missed the internet boom,” he said. “I would’ve made 100x more money.”

“Well,” Mr. Bespaloff replied, “you’re in crypto now.”

“You live and learn,” Mr. Belfort said.

Audio produced by Adrienne Hurst.

03

Act Like a Scientist

Idea in Brief The Problem

Many leaders overrely on their gut instinct or personal experiences when making decisions—despite decades of admonitions about the dangers of doing so.

The Root Cause

Executives often think that what worked for them in the past—the successes that earned them their leadership roles—will work in the future. And their subordinates often reinforce those feelings.

The Solution

Senior managers should take a scientific approach to making decisions. They should challenge assumptions and investigate anomalies by articulating testable hypotheses and conducting rigorous experiments that generate conclusive evidence.

Every day, managers make decisions about products, customers, resource allocation, employee pay, and more, basing them on assumptions that have never been critically examined, much less challenged. “I’ve always been successful doing it this way and never thought about doing it another way” is what we often hear when managers are asked why they didn’t question practices that turned out to be faulty. But when skeptics show that ideas underlying practices are wrong, confounding, or even costly, leaders grasp the importance of systematically testing assumptions.

Consider what happened at the hotel and casino company Harrah’s Entertainment in the early 2000s, when one of us, Gary, who was then its chief operating officer, worked with his analytics team to reevaluate the company’s approach to marketing incentives. The leaders of Harrah’s had subscribed to the industry’s conventional wisdom that financial incentives such as reduced room rates, food credits, and vouchers for retail stores heavily influenced customers’ decisions to visit Las Vegas and that offering more of them increased the likelihood that people would book rooms there. Gary and the team set out to improve the efficiency of marketing spending by rigorously testing initiatives individually. (Trained as an economist, Gary understood the importance of assessing the incremental contributions of each element of the marketing program—instead of measuring the collective impact of the entire program, which was the industry practice.) They ran hundreds of tests to see which incentives induced people to stay at the company’s hotels and to what degree. The results revealed that some, such as retail-store discounts, didn’t affect hotel bookings and could be eliminated. Moreover, if the money spent on them was reallocated to effective incentives, such as deeper room discounts, Harrah’s could boost both responsiveness and profits.

By 2005 the company was using experiments to improve many other strategic and operating decisions. For instance, its executives had assumed that because people liked transparency and fairness, they preferred an orderly physical queue at Caesars Palace’s all-you-can-eat Bacchanal Buffet to a virtual queue—a digital notification system that allowed customers to leave the vicinity while holding on to a place in line. But a test revealed that if the restaurant sent customers a text 10 minutes before their turn to be seated, they used the time to buy drinks or gamble, generating revenue that far exceeded the revenue lost from people who didn’t want to wait. Over time similar experiences led Harrah’s to develop a culture of curiosity, where poking holes in the industry’s conventional wisdom became not only acceptable but celebrated.

If challenging assumptions is so valuable, why don’t managers make it a standard operating procedure? After decades of studying and practicing innovation and decision-making, we’ve concluded that the fundamental reason is that most business leaders don’t think or act like scientists. This is a huge lost opportunity. Research by one of us, Stefan, has found that rigorous experiments can help managers discover whether a new product, service, or business program will succeed. (See “The Discipline of Business Experimentation,” HBR, December 2014.) And in his roles as a chief operating officer, CEO, and president of large entertainment and health care businesses, Gary has seen that investments in data analytics lead to better decisions. But many managers are still reluctant to fund experiments, and despite decades of admonitions about the dangers of gut instinct, continue to overrely on intuition and personal experience in decision-making—even when the evidence contradicts them.

Acting like a scientist is difficult for leaders because it can challenge their legitimacy. Undoubtedly, that’s because someone’s position in the corporate hierarchy is often assumed to be the result of experience and a track record of successful moves and ideas. Senior executives live in a feedback loop of positive reinforcement that makes them unlikely to question the foundations of their decisions. The scientific method, in contrast, requires intellectual humility in the face of difficult problems and relies on an objective, evidence-based process, rather than predominantly personal insight, to frame and address decisions.

When we think scientifically, we recognize that human beings make cognitive and judgmental errors and can drift into a complacency built on flawed assumptions. When we act scientifically, we relentlessly probe our assumptions and change them if evidence shows that they’re wrong. Taking a scientific approach to decisions is critical for today’s organizations, particularly in light of the enormous upheavals the Covid-19 pandemic has wrought.

In this article we’ll discuss five elements of the scientific method that we find to be particularly useful in management practice.

[  1  ]

Be a Knowledgeable Skeptic

When business leaders adopt this mindset, their biases and errors won’t get in the way of finding the truth. They will employ reason, demand evidence, and be open to new ideas. In scientific practice this means seeking independent confirmation of facts, placing more value on expertise than on authority, and examining competing hypotheses. Above all, skeptics question assumptions. They ask, “Why do we believe this?” or “What is the evidence that this is true?” History is full of examples where such skepticism helped overturn commonly held ideas and led to important scientific advances.

When managers are knowledgeable skeptics, it can transform how a company operates. Consider Sony. When Kazuo Hirai was put in charge of its consumer electronics businesses, in 2011, the company was struggling. Its once-successful TV business had experienced increasingly deeper financial losses for years. That’s because Hirai’s predecessors had a core assumption: To restore profitability, the business needed to increase the number of TVs sold in order to cover Sony’s high cost of doing business. Hirai (who would become Sony’s CEO in 2012) was skeptical and commissioned an analysis. It revealed that the business would need to sell 40 million TV sets a year to be viable. But in 2010 the company had sold only 15 million. More problematic, to achieve volume targets, previous leaders had repeatedly instituted price discounts, which triggered a further cycle of losses.

Felix Schöppner explores themes of experimentation, perception, physics, and astronomy by artfully arranging everyday objects in his photo studio. 

Hirai ordered Sony’s sales organization to sell fewer TVs and raise prices. The company reduced the number of LCD TVs it sold in developed countries by 40% or so and cut the number of its U.S. models nearly in half. At the same time, it restructured to lower fixed costs, asked engineering to improve picture quality to justify higher prices, and launched a retail model that differentiated its products: a store-within-a-store at Best Buy. In 2015, Sony’s TV business reported the first operating profit in 11 years. The skeptic’s intervention had worked.

[  2  ]

Investigate Anomalies

In science the study of anomalies has been instrumental in identifying questionable assumptions. Anomalies are things that are unexpected, don’t look right, or seem strange, and they’re noticeable because they don’t cohere, or fit, with sought-after outcomes. Managers should watch for and explore them because they can lead to new business insights. (See “The Power of Anomaly,” HBR, July–August 2021.)

A famous anomaly, for example, led the scientist Louis Pasteur to make a major discovery while studying the causes of chicken cholera. In 1879, when he returned from a summer vacation, he realized that his cultures of chicken cholera had lost their virulence. He also noticed that when his assistant injected the spoiled cultures in hens, they developed only mild symptoms and fully recovered. When the same birds were injected with fresh, virulent bacteria, they remained healthy. His discovery—that weakened or dead microorganisms that produce mild disease can prevent that same disease in its lethal form—led to one of the biggest breakthroughs in fighting infectious diseases: live attenuated vaccines.

Business leaders who look for and act on anomalies can likewise unearth insights that lead to significant opportunities, as Gary discovered in 1999, after he became COO of Harrah’s. One night in the elevator of the company’s hotel in Las Vegas, he overheard one customer telling some other customers, “I can’t win in Vegas. The slot machines are much tighter here than in Atlantic City”—meaning they had lower average payouts. The other customers agreed.

The conversation surprised Gary. First, he knew that slot machines in Las Vegas had more-generous average payouts. (Machines in Las Vegas paid back 94.5% of customers’ money, on average, while those in Atlantic City paid 93%.) Second, the long-held industry assumption was that tighter slot machines drove customers to casinos with more-generous payouts. What if most customers were like those in the elevator and couldn’t tell the difference? Could an entire industry have gotten this wrong? He asked his analytics team to investigate.

The team found that the industry misunderstood how individual customers experienced playing. Customers would never encounter average payouts during a typical visit or even multiple visits; they would have to play the machines 80,000 times to do so. Consequently, they couldn’t possibly detect the difference in average payouts between Vegas and Atlantic City. The elevator conversation ultimately led to a revolution in the casino business. Companies started to hire data scientists to use analytics and experimentation to determine the optimal payouts and locations of slot machines. Over time average payouts have fallen as casinos have become more confident in their ability to lower them without discouraging customers from playing.

History is full of examples where skepticism helped overturn commonly held ideas and led to important scientific advances.

Anomalies can also reveal significant problems that are about to hit an organization. One person who ardently believes this is Jørgen Vig Knudstorp, the executive chairman and former CEO of the Lego Group. He told Stefan that even when the percentage of customers complaining about a product is extremely small, a company should “really listen and listen very actively.” He learned that when the company shipped 15,000 units of a particular Lego set without a critical component but heard from fewer than 5% of the customers who had bought them. “This illustrates an important lesson,” he said. “When you hear a complaint from somebody, I think it’s healthy to assume there are a lot more people who are unhappy.”

[  3  ]

Articulate Testable Hypotheses

To be effectively challenged, assumptions must be framed as hypotheses that can be quantifiably confirmed or disproved. “When you can measure what you are speaking about and express it in numbers, you know something about it,” said Lord Kelvin, a leading figure in 19th-century science and engineering. “But when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.” An experiment that produces evidence contradicting a hypothesis allows us to recognize errors in our thinking and judgment, modify the hypothesis, and then retest it. This iterative process of testing and refining ultimately leads to stronger hypotheses.

A Strong Hypothesis Versus a Weak One

Every effective experiment begins with the articulation of a good testable hypothesis.

Strong Weak

Source

Strong

Qualitative research, customer insights, problems, observations, data mining, competitors (example: “We observed fewer customers during the first store hour”)

Weak

Guesses not rooted in observation or fact (example: “We think that wealthier buyers will like our products”)

Design

Strong

Identifies possible causes and effects (example: “Opening our stores one hour later has no impact on daily sales revenue”)

Weak

Does not identify possible causes and effects (example: “We can extend our brand upmarket”)

Measurement

Strong

Quantifiable metrics that establish whether the hypothesis should be accepted or rejected (example: time and revenue)

Weak

Vague qualitative outcomes driven by several variables that are hard to isolate and measure (example: brand value)

Verification

Strong

The experiment and its results can be replicated by others

Weak

The experiment and its results are difficult to replicate

Relevance to a meaningful business outcome

Strong

Will have a clear impact (example: “Opening an hour later will reduce store operating expenses”)

Weak

Won’t necessarily have a significant impact, or the link between the metric and business impact is fuzzy (example: “It’s unclear how extending the brand affects profitability”)

Here’s an example from science: For centuries the assumption was that the universe comprised matter called ether, which light traveled through. The ether hypothesis arose because scientists believed that light waves required a medium to propagate in empty space. In 1887 the physicists Albert Michelson and Edward Morley set out to prove this thesis was right. They conducted an experiment that measured the speed of light in perpendicular directions. Any difference in speed would be evidence of either’s existence. But no such difference was found, undercutting the hypothesis and accelerating the search for a new scientific theory of space and time: special relativity. The experiment opened the door to another way of thinking about how the universe worked.

Businesses can apply a similar approach. At Bank of America it was used by a team tasked with improving customers’ experiences in branch offices. One problem the team sought to address was the irritation customers felt when waiting for service. An internal study involving about 1,000 customers (whose findings were confirmed by two focus groups and an analysis by Gallup) revealed that after a person stands in line for about three minutes, a wide gap opens between actual and perceived wait times. A two-minute wait, for example, usually feels like a two-minute wait, but a five-minute wait may feel like a 10-minute one. Aware of studies suggesting that when you distract a person from a boring chore, time seems to pass much faster, the team articulated a straightforward hypothesis: Putting television monitors above the row of bank tellers will reduce perceived wait times. To test it, the team set up an experiment: It installed televisions tuned to CNN above the tellers in one Atlanta branch and compared the perceptions of waiting customers there with those of customers in a comparable branch without monitors. After allowing a week for the novelty of the TVs to wear off, the team measured customers’ estimates of wait times for two weeks. In the branch with the TVs, the overestimation dropped from 32% prior to the test to 15%; at the control branch it increased from 15% to 26%.

In business, ideas for hypotheses can come from multiple sources. A good starting point is customer insights derived from qualitative research (focus groups, usability labs, and the like) or analytics (data collected from calls to customer support, for example). As we have seen, hypotheses can also be inspired by anomalies, which can be found in everything from overheard conversations to successful practices that deviate from the norm at other companies.

[  4  ]

Produce Hard Evidence

Explaining the key to science in a lecture at Cornell University in 1964, the theoretical physicist Richard Feynman declared: “It doesn’t make any difference how beautiful your guess is. It doesn’t make any difference how smart you are, who made the guess, or what his name is. If it disagrees with experiment, it’s wrong. That’s all there is to it.” Senior business leaders should take that advice to heart. An endeavor’s underlying assumptions shouldn’t be based solely on the feelings, experiences, guesses, or status of those holding them. They should also stem from conclusive evidence. If such proof doesn’t already exist, disciplined experiments can provide it. This tenet should be a pillar of a company’s culture. (See “Building a Culture of Experimentation,” HBR, March–April 2020.)

Business settings offer many opportunities to conduct such experiments. Let’s look at another effort that was led by Gary. In late 2009 many Las Vegas hotels and some hospitality companies elsewhere began to impose resort fees, which were single, all-inclusive charges that replaced à la carte charges for Wi-Fi, bottled water in rooms, access to the fitness center, and so on. When customers sought to book a hotel room, they would first be presented with the nightly rates. But once they moved to reserve it, they would see a resort fee added to the total, along with taxes.

Felix Schöppner

At that point Gary had been CEO of the combined Harrah’s and Caesars Entertainment for four years. He and his senior operating team assumed that prospective guests would view the resort fee as a price increase. He worried that it would reduce demand for rooms—especially from price-sensitive customers—and cause the occupancy rates to fall. (In Las Vegas high occupancy is especially critical. Guests who stay at hotels with casinos often spend more on gambling, food and beverages, entertainment, and other resort amenities than they spend on their rooms.) There was anecdotal support for their assumption: Southwest Airlines, for instance, was attracting customers by not charging for checked bags while competitors did. Gary and his team therefore decided not to follow the pack with resort fees. In 2010 the company ran ads and promotions highlighting the fact that its hotels were a “resort fee free zone.”

When the first data on the occupancy rates of the company and its competitors arrived, however, there was no evidence that the decision to forgo fees was working. After about three months, Gary asked his senior operating team to test the initial assumption with an experiment. The company began by imposing a resort fee only on the guests who were expected to react with the least hostility: convention and meeting attendees and customers who weren’t in the upper tiers of a reward program. After three months of testing, it was clear that customers weren’t sensitive enough to resort fees to move their business to other hotels (most of which already charged them). The company continued its experiments by applying fees to its hotels beyond Vegas. Finally, enough hard evidence accumulated to convince Gary and his team that customers were less sensitive to resort fees than they were to room rates.

[  5  ]

Probe Cause and Effect

Relying on assumptions about cause and effect is dangerous for managers. We humans often see connections between unrelated actions and outcomes—confusing correlation with causation—and respond to irrelevant “noise” factors when making decisions. (See “Noise: How to Overcome the High, Hidden Cost of Inconsistent Decision Making,” HBR, October 2016.) We also tend to happily accept “good” evidence that confirms our causal assumptions but challenge and investigate “bad” evidence that goes against them.

Scientists probe causality in different ways. In conventional experiments they change one or more variables (the presumed cause) and observe changes in the outcome (the effect) while holding all other variables constant. When they can’t keep all other variables constant, they rely on randomization, which prevents systemic bias, introduced consciously or unconsciously, from affecting the experiment. Randomization evenly spreads any remaining potential causes of the outcome between test and control groups.

The company’s transformation required overhauling its business systems and constantly asking questions such as “Is this really true?” and “Do we really believe in that?”

In natural experiments the variables are outside the control of the investigator, but they can still reveal insights about causality. (Last year the researchers Joshua Angrist and Guido Imbens won a Nobel Prize for showing how. To examine whether unearned income changed people’s incentives to work, for instance, Imbens and his collaborators looked at data on lottery winners in Massachusetts. Because prizes in the state are paid out over many years, they are very similar to guaranteed basic income. By studying people who had won the lottery and comparing them with people who hadn’t, Imbens could infer the causal effect of guaranteed basic income.)

When conventional experiments aren’t feasible—say, because the interplay between the variables can’t be observed—simulations often are useful. Finding evidence for “A causes B” gives scientists confidence that what they’ve observed isn’t just a correlation. But a stronger test of causality is the use of counterfactuals, such as “Would B have occurred if not for A?” For business leaders, that means not just looking for evidence that a 10%-off coupon increased sales but also exploring whether the increase would have occurred even if the company hadn’t offered the discount. Asking what-if questions and thinking about counterfactuals is a powerful way to examine scenarios under different assumptions and arrive at insights about cause and effect.

Leadership

Must-reads from our most recent articles on leadership and managing people, delivered once a month.

Leaders should use this approach to test assumptions about the fundamental factors that drive their companies’ success. Knudstorp did just that after he became CEO of Lego, in 2004. When he took the helm, the company was on the ropes, suffering from depressed sales and stagnant growth. Over the next decade he transformed it into a leader in the toy industry. Getting there required overhauling its business systems and constantly asking questions such as “Is this really true?” and “Do we really believe in that?” One of the things the management team reexamined was the company’s decision to outsource its operations to Flextronics. The assumption had been that the move would streamline Lego’s supply chain, reducing costs, but it turned out that it actually led to longer lead times, higher purchasing expenses, and shorter lifetimes for injection molds. Lego’s leadership recognized that bringing manufacturing back in-house would make the company more competitive. For example, by investing in cutting-edge injection-molding technology, Lego was able to provide users a better building experience that competitors couldn’t match. (The connecting forces of bricks had to be strong enough to hold them together but not so strong that they couldn’t be pulled apart by a small child. In addition, the new bricks had to be compatible with those manufactured decades ago. Only very tight molding tolerances could achieve that.)

The probing process also involved listening to the products’ community of fans, which led to the insight that Lego’s building instructions were more important than the company had realized, because they allowed ordinary users to create extraordinary constructions. In response Lego expanded the resources devoted to the creation of instructions, whose quality and style improved. Today many are digital and 3D.

. . .

The global pandemic has introduced us to a world full of peril and much greater uncertainty. Assumptions about how we work and live have been turned on their heads. Supply chains no longer seem to function, and answers to the most pressing business problems appear elusive. What happens to organizational cultures, for example, when people no longer work in offices? Can a manufacturer run a factory with no people? Can we bring down skyrocketing insurance costs by motivating employees to take action to improve their health? But a time of great uncertainty is also an opportunity to rethink what business leaders have assumed to be true. It would be a mistake to rely only on experience, intuition, and judgment to guide us through this tumultuous era.

Men and women who have practiced the scientific method have given us amazing medical remedies; a vastly safer and more plentiful food supply; new kinds of energy, transportation, and communication; and so much more. It’s a highly effective way to help businesses increase the likelihood of success, reduce errors in judgment, and find sources of innovation and growth. It should play a central role in their decision-making processes.

A version of this article appeared in the May–June 2022 issue of Harvard Business Review.