"Indispensable book for any serious experimentation practitioner. This book is highly practical and goes in-depth like I've never seen before. It's so useful it's feels like you get a superpower. From statistical nuances to evaluating outcomes to measuring long term impact, this book has got you covered. Must-read." -- Peep Laja, Founder and Principal of CXL, and top conversion rate optimization expert
"Online experimentation was critical to changing the culture at Microsoft. When Satya talks about 'Growth Mindset,'
experimentation is the best way to try new ideas and learn from them. Learning to quickly iterate controlled experiments drove Bing to profitability, and rapidly spread across Microsoft through Office, Windows, and Azure." -- Eric Boyd, Corporate VP, AI Platform, Microsoft
"As an entrepreneur, scientist, and executive I’ve learned (the hard way) that an ounce of data is worth a pound of my intuition. But how to get good data? This book compiles decades of experience at Amazon, Google, LinkedIn, and Microsoft into an accessible, well-organized guide. It is the bible of online experiments." -- Oren Etzioni, CEO of Allen Institute of AI and Professor of Computer Science at University of Washington
"Internet companies have taken experimentation to an unprecedented scale, pace, and sophistication. These authors have played key roles in these developments and readers are fortunate to be able to learn from their combined experiences." -- Dean Eckles, Associate Professor at Massachusetts Institute of Technology and former scientist at Facebook
"A wonderfully rich resource for a critical but under-appreciated area. There are many real case studies in every chapter showing the inner workings and learnings of successful businesses. The focus on developing and optimizing an 'Overall Evaluation Criterion' (OEC) is a particularly important lesson." -- Jeremy Howard, founder of fast.ai, faculty at Singularity University, and former president and chief scientist of Kaggle
"There are many guides to A/B Testing, but few with the pedigree of Trustworthy Online Controlled Experiments. I've been following Ronny Kohavi for eighteen years and find his advice to be steeped in practice, honed by experience, and tempered by doing laboratory work in real world environments. When you add Diane Tang, Ya Xu to the mix, the breadth of comprehension is unparalleled. I challenge you to compare this tome to any other - in a controlled manner, of course." -- Jim Sterne, Founder of Marketing Analytics Summit and Director Emeritus of the Digital Analytics Association
"An extremely useful how-to book for running online experiments that combines analytical sophistication, clear exposition and the hard-won lessons of practical experience." -- Jim Manzi, Founder of Foundry.ai, Prior founder, CEO and Chairman of Applied Predictive Technologies (acquired by Mastercard), and author of Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society
"Experimental design advances each time it is applied to a new domain: agriculture, chemistry, medicine and now online electronic commerce. This book by three top experts is rich in practical advice and examples covering both how and why to experiment online and not get fooled. Experiments can be expensive; not knowing what works can cost even more." -- Art Owen, professor of Statistics at Stanford University, where among other things, he teaches experimental design.
"This is a must read book for business executives and operating managers. Just as operations, finance, accounting and strategy form the basic building blocks for business, today in the age of artificial intelligence, understanding and executing online controlled experiments will be a required knowledge set. Kohavi, Tang and Xu have laid out the essentials of this new and important knowledge domain that is practically accessible." -- Karim R. Lakhani, Professor and Director of Laboratory for Innovation Science at Harvard, Board Member, Mozilla Corporation
"Serious 'data-driven’ organizations understand that analytics aren't enough; they must commit to experiment. Remarkably accessible and accessibly remarkable, this book is a manual and manifesto for high-impact experimental design. I found its pragmatism inspirational. Most importantly, it clarifies how culture rivals technical competence as a critical success factor." -- Michael Schrage, research fellow at MIT’s Initiative on the Digital Economy and author of 'The Innovator’s Hypothesis: How Cheap Experiments Are Worth More than Good Ideas'
"This important book on experimentation distills the wisdom of three distinguished leaders from some of the world's biggest technology companies. If you are a software engineer, data scientist, or product manager trying to implement a data-driven culture within your organization, this is an excellent and practical book for you." -- Daniel Tunkelang, former Director of Data Science and Engineering at LinkedIn, Tech Lead at Google, and Chief Scientist at Endeca.
"With every industry becoming digitized and data-driven, conducting and benefiting from controlled online experiments becomes a required skill. Kohavi, Tang and Yu provide a complete and well-researched guide that will become necessary reading for data practitioners and executives alike." -- Evangelos Simoudis, Co-founder and Managing Director Synapse Partners; author of The Big Data Opportunity in Our Driverless Future
"Ronny, Ya and Diane offer over 10 years of hard-fought lessons in experimentation, from beginner rules of thumb to advanced topics, in the most strategic book for the discipline yet." -— Colin McFarland, Director Experimentation Platform at Netflix
"The practical guide to A/B testing distills the experiences from three of the top minds in experimentation practice into easy and digestible chunks of valuable and practical concepts. Each chapter walks you through some of the most important considerations when running experiments--from choosing the right metric to the benefits of institutional memory. If you are looking for an experimentation coach that balances science and practicality, then this book is for you." -- Dylan Lewis, Experimentation Leader, Intuit
"The only thing worse than no experiment is a misleading one, because it gives you false confidence! This book details the technical aspects of testing based on insights from some of the world’s largest testing programs. If you're involved in online experimentation in any capacity, read it now to avoid mistakes and gain confidence in your results." -- Chris Goward, Author of You Should Test That!, Founder and CEO of Widerfunnel
"This is a phenomenal book. The authors draw on a wealth of experience and have produced a readable reference that is somehow both comprehensive and detailed at the same time. Highly recommended reading for anyone who wants to run serious digital experiments." -- Pete Koomen, Co-founder, Optimizely
"Ronny, Ya, and Diane are pioneers of online experimentation. The platforms they’ve built and the experiments they’ve enabled have transformed some of the largest internet brands. Their research and talks have inspired teams across the industry to adopt experimentation. This book is the authoritative yet practical text that the industry has been waiting for." -- Adil Aijaz, Co-founder and CEO, Split Software