How to make an AI talent factory

Twenty years ago the product recommendations team at Amazon.com could hire as many data scientists as they wanted and hired Ph.D.’s in computer science, mathematics and economics.  Ten years later a San Francisco-based startup that also did product recommendations raised $100 million in venture capital, served 600 million shoppers per month, but had difficulty hiring top data science talent since the best candidates were all going to Google and Facebook.  Hiring data science talent is hard.  Some estimate that there are three job openings for every candidate and, even if you can hire some, they stay for less than three years.  What’s a manager with a long AI project backlog to do?  Options include:

• Call Palantir and others.  They do great work but are expensive, often leave behind brittle apps, and generate little organizational knowledge.
• Hire in a regional AI consulting firm.  This can work well for a few projects but doesn’t scale to big backlogs or enterprise-wide, data-driven transformations, and again you risk never gaining AI organizational knowledge.
• Hire the best data scientists you can and hope for the best.  A large Canadian retailer recently tried this, competed directly with Google and Shopify for talent, and found that what they could get at scale was low quality.  They lost two years on their AI roadmap before the approach failed.
• To paraphrase a former U.S. Secretary of Defense, go to market with the talent you have, not the talent you might want or wish to have at a later time.  Upskill everyone.  Recognize, promote and reward rising stars as they appear.

Successful companies have long taken this last approach with another rare and valuable skill—leadership—so much so that whole industries turn to them to hire the talent that they develop.  These so called academy companies include PepsiCo and Procter & Gamble in consumer packaged goods, Goldman Sachs and JPMorgan Chase in financial services, General Electric in manufacturing, McKinsey and Bain in business management, and more recently, Amazon and Netflix in Internet companies.  How can you become an “academy” company for AI?

Key elements of an AI talent factory

Three key elements to a sustainable AI talent factory are:

• An AI technology platform that all data analysts can use, that boosts productivity, and that is satisfying to use
• Self-service upskilling for all skill levels
• A platform that allows you to spot and nurture your rising stars

We’ve seen it implemented successfully in many industries in three layers:

Model Factory:  Implement integrated tools that cover the whole AI product lifecycle—data integration, exploration, visualization, cleaning, enrichment, cataloging, drift monitoring and governance; and model development, experimentation, transparency, fairness, operations, A/B testing, drift monitoring and governance—so that you don’t have to throw projects over the wall from one disparate team to another.  Great data scientists can cover the entire lifecycle with minimal tools, such as Python notebooks, but there aren’t enough great data scientists to go around.  A talent factory should enable all analysts, not just the elite.  Dataiku has helped companies in many industries develop AI talent factories and we’ve found that only about 5% of the analysts who benefit from a model factory are data scientists.  The breakdown is generally something like:

Enabling everyone requires highly usable tools that are:

1. Easy for a wide range of analysts to learn 
2. Big productivity gains compared to previous tools
3. Easy to remember for analysts that don’t do AI full-time 
4. Prevent serious errors by novice and part-time users
5. Satisfying to use

The right modeling factory technology often drives 300% productivity gains.  It’s like moving from the C programming language to Python and open source libraries.  You can quadruple your throughput without adding headcount.

Adoption Program:  Tools don’t generate value until someone uses them so fund a program to drive adoption.  Recent data from McKinsey indicates that you need 7% to 25% of your data analysts involved to generate ROI.  Most businesses fall short of that by focusing on the elites.  Too often analysts try interesting new tech for a while but quickly revert back to the status quo, SAS or Python notebooks, generating negative ROI for an AI initiative.  We’ve found that key elements of an effective AI adoption program include solution development workshops, AI maturity assessments, hub-and-spoke organizational design, business value assessments, evangelism, onboarding, training, office hours, coaching, hackathons, badges, laptop stickers and a peer-to-peer learning community.  All for every skill level.   

Upskilling Program:  Adoption is the first step but the big ROI comes when workers not only upgrade their technology but also upgrade their skills.  Business analysts learn data engineer skills.  Data engineers acquire data science skills.  Data scientists start using video, images, audio, natural language and multi-layer neural networks (a.k.a. deep learning), and so on.  For example, a bank that Dataiku works with went from 20% of their advanced analytics projects using machine learning to 70% using machine learning, in just one year.  Managers can facilitate that type of rapid growth by providing self-service, multi-level certification, internal conferences, coaching and more badges and laptop stickers.  Self-service upskilling is critical because we just don’t know which nooks of an organization talent will come from and because “each of us is a disturbingly unreliable rater of other people’s performance.”

Of course, you can’t just give every analyst a Spark or Kubernetes cluster and deep learning software and hope for the best.  The aforementioned Canadian retailer did that and a single analyst spent $1 million on cloud computing in a month.  An AI talent factory needs guard rails, ideally via a single pane of glass to manage data access, compute costs, projects, AI model releases, teams and analysts.  With that visibility you can then identify, nurture and reward your rising analytics stars.  

Allocating resources for your AI talent factory

AI is still nascent and misunderstood by many businesses. Thus many companies consider it an achievement when they stand up purpose-built data lakes and a modeling factory.  In truth, they are only half way to significant ROI and a long way from a cultural transformation tipping point.  Companies should spend half of their analytics budget on adoption and upskilling in order to develop a sustainable AI talent factory.

Summary

Hiring enough top-tier AI talent isn’t feasible for most companies.  However, you don’t have to lower your expectations or delay the execution of your AI strategy.  Five or ten years ago, organizations didn’t know much about AI skills so they had to hire them and then teach business and domain knowledge.  The balance between AI and domain knowledge has shifted.  AI best practices are now well understood. Today the fastest path to big AI ROI is to develop the business and domain talent that you already have by building a sustainable, scalable AI talent factory.  We’ve seen it done successfully in many industries.  Get a usable tech platform, drastically boost productivity, and then spend big on adoption and upskilling across all skill levels.  You can do it!  You can become an AI academy company.