The evolution of AI is fundamentally reshaping the role of product managers (PMs). In a recent panel discussion, experts from leading tech companies shared their insights on navigating this rapidly changing landscape.
In this post, we'll dive into six core competencies of the modern AI PM, and how our panel feels about each one. You can also view the recording below.
The Six Core Competencies of an AI Product Manager
Our panel first outlined their agreed-upon core competencies of an AI PM:
Strategy and Vision
Risk and Responsible AI
Data Depth
Model Development and Lifecycle
Evaluation
Getting to Launch
**Related Reading: **The Six Core AI PM Competencies
Effectively, there are many things that won’t stray far away from the fundamentals of product management. But, AI changes the speed, risk, and post-launch iteration for this new age of product management.
Here are the key takeaways that will help you stay ahead in the AI-driven future.
Your Strategy and Vision Should Focus on Improving the Lives of Users
The first core competency that our panel discussed was *strategy and vision. *While this doesn't differ dramatically from a traditional PM competency, AI changes the ways you should look at it.
AI Will Enable Personalized and Dynamic Interfaces
AI’s potential to revolutionize user interfaces is immense. Imagine no longer needing a one-size-fits-all approach. Instead, interfaces could dynamically adapt to each user's specific needs, both expressed and observed. This personalized interaction could eliminate the traditional learning curve, making software more intuitive and efficient.
Dylan Sellberg from HubSpot highlights this shift: "You can imagine completely dynamic interfaces or no interface at all that is entirely personalized to an end user's needs."
The future envisions a seamless integration where users interact with software that understands their workflow and preferences, reducing the need for extensive onboarding and training.
Customer-Centric Design is Critical
AI PMs must focus on delivering 10x value, ensuring that AI products meet real customer needs rather than merely showcasing tech capabilities. AI's promise lies not just in enhancing user experiences but also in streamlining processes and uncovering hidden insights.
Automate the Mundane, Focus on the Creative (Make People’s Lives Better) AI excels at automating tedious tasks, freeing up users to focus on more creative and strategic activities. For instance, HubSpot uses AI to handle repetitive tasks like optimizing SEO or localizing content, allowing users to dedicate their time to crafting engaging content and strategies.
Dylan also notes: "The real home run is actually in the surround sound work that people don't like doing... that is what we call toil work that I think we can really automate."
By taking over these monotonous tasks, AI not only boosts productivity but also enhances job satisfaction by allowing individuals to engage in more meaningful work.
These evolutions in user experience and interface hold some of the most powerful potential for a PM to capitalize on in their strategy and vision.
Prioritize Responsible AI
The second competency our panel discussed was the world of risk and responsibility in AI, arguably one of the biggest new dimensions with this technology.
Take a Stance on User Control and Transparency
Ensuring users have control over AI features is crucial. At Zoom, AI features are launched with default settings turned off, and users are informed when AI is active. This approach not only builds trust but also empowers users to decide when and how AI is utilized.
Ailian Gan from Zoom explains: "We want to create this effect where the user is always in control of how AI is interacting with their content."
This user-centric approach ensures that AI serves as an empowering tool rather than a disruptive force.
Ethical and Transparent Use of Data
Zoom also emphasizes that customer data is never used to train models without explicit consent. This policy, now adopted by many leading companies, underscores the importance of ethical data use and privacy in AI development.
Ailian further states: "We wanted to say, like, your customer data will never be used to train models, period. Full stop."
This clear stance on data privacy fosters trust and sets a standard for the industry. AI PMs must navigate complex privacy and compliance landscapes, particularly when dealing with sensitive industries like healthcare and finance.
Have Guardrails for Generated Content
Responsible AI involves anticipating and mitigating potential misuse of features. For example, Zoom’s forthcoming AI virtual backgrounds must be carefully managed to avoid generating offensive content. This involves modifying user prompts and implementing filters to ensure safe and appropriate outputs.
**Related Reading: **Prompt Engineering Template
By considering potential risks upfront, companies can prevent misuse and ensure their AI applications are both useful and safe.
Embrace Data Depth
The third competency our panel discussed was attention to the data sources your models are trained on, data security, and more.
Detailed Data Analysis - What Are Your Models Trained On?
AI relies heavily on data, making in-depth analysis essential. PMs must delve into data to identify outliers and understand the broader implications. This thorough approach ensures that AI systems are robust and reliable, even in edge cases.
Dylan reflects on this necessity: "Understanding being able to dig in deep and understand like that level of data and the knock-on impacts is important to be able to actually like understand when things are going wrong."
A deep dive into data allows PMs to preemptively address issues and refine their AI models continuously.
Streamline Model Development
The fourth competency our panel discussed was the question of whether to leverage foundation models or to develop proprietary ones. An AI PM needs to understand the tradeoffs.
Utilize Foundation Models to Save Time
Starting with foundation models can expedite development. These models provide a robust starting point, allowing teams to build and validate AI features quickly. Over time, these can be refined and tailored to meet specific customer needs more precisely.
Monitor and Improve Continuously - Iterate from the Foundation Investing in continuous monitoring and improvement of AI models is critical. This involves not only tracking performance but also identifying gaps and opportunities for refinement, ensuring the AI remains effective and relevant.
Dylan emphasizes: "I think almost nobody is investing heavily enough in AI performance right now. That is the kind of next trajectory and curve we see for really good products."
Continuous improvement is key to maintaining the relevance and efficacy of AI solutions.
Implement Rigorous Evaluation
The fifth competency our panel discussed was evaluation - the ability to evaluate metrics related to AI system performance and outputs.
Human-in-the-Loop Evaluations
Despite advances in automated evaluation, human judgment remains invaluable. LinkedIn’s experience with job matching highlighted the importance of involving real users in the evaluation process. This hands-on approach helps fine-tune AI models to better meet user needs and expectations.
Kumarash from LinkedIn shares: "Evaluation is the product in some ways. It defines the voice and tone and the direction that you're taking the product."
Involving humans in the evaluation process ensures that AI outputs align closely with user expectations and real-world scenarios.
Focus on Post-Launch Success
The last competency our panel discussed was how fluid the "launch" of an AI product is. The launch is really just the beginning.
Iterative Beta Testing - Launch Scrappy, Then Improve
A phased approach to launching AI features, including extensive beta testing, is essential. This allows teams to gather feedback, stress-test the features, and refine them based on real-world usage, ensuring a smooth and successful rollout.
**Related Reading: **Bridging the gap from Gen AI POC to production
Ailian discusses Zoom’s approach: "We do a period of internal beta testing where 6,000-7,000 employees are invited to test and beat it up as much as possible and give us feedback."
This iterative process helps identify and address potential issues early, leading to more robust and reliable AI features.
Comprehensive Sales Enablement - Help Your GTM Teams
Effective go-to-market strategies are vital. This includes equipping sales and customer success teams with thorough training and resources, such as detailed FAQs and security white papers, to address customer concerns and drive adoption.
Looking Ahead: The Future of AI Product Management
Specialization and Depth
The AI PM role will likely see increased specialization. PMs will focus on specific areas such as model evaluation, orchestration, and platform development. This depth of expertise will be crucial as AI continues to integrate into various facets of product management.
Kumarash predicts: "There’s going to be a lot of specialization within the AI PM role... identifying something that you do today will help you then be able to prepare yourself to be the specialized version of an AI PM."
Demonstrated Experience and Learning
Dylan emphasized the importance of building and demonstrating practical AI solutions. Real-world projects, even small-scale ones, can significantly bolster a resume. AI PMs must show a track record of learning, experimentation, and problem-solving.
AI as a Productivity Tool
AI will become an indispensable tool for PMs, assisting with drafting documents, creating prototypes, and predicting project timelines. This will enable PMs to focus on higher-level strategic decisions and creative problem-solving.
Ailian envisions: "AI will help you draft first draft everything... it changes how you function as an AI PM if your skill is about how to harness these tools in different ways."
Curating and Judging AI Outputs
The core skills of curation, taste, and judgment will remain critical. As AI takes over more administrative tasks, PMs will need to leverage their human intuition and expertise to guide AI outputs effectively.
The future of product management in the age of AI is bright and full of opportunities. By embracing these changes and continuously evolving, PMs can lead their teams to success and innovation.
Related Resources
Artifacts: Free, real work example from leading experts
PRD Template for AI-driven Features
AI/ML roadmap at Neurons Lab
Product spec for AI-powered malware scan MVP at BitNinja
Using Reforge’s AI to brainstorm around a new use case
GenAI Product Strategy and Roadmap at Spiffy.ai
Guides: Members-only, step by step instruction
Evaluate the value of Gen AI for your product
Evaluate techniques for incorporating LLMs into your product
Define Successful Conversational AI Products
Understand conversational AI technology
Design a Conversational UX Experience
Courses: In-depth upskilling with Reforge experts
Generative AI Products: How to Get from Idea to MVP - Polly Allen and Rupa Chaturvedi
GenAI Product Strategy - Aniket Deosthali
Blog Post + Demo Video - How we built the Reforge Extension!
Podcast: Reforge’s Unsolicited Feedback is available on the platform of your choice
Hear Box CTO, Ben Kus, evaluate AI models and discuss his approach to building enterprise AI tools.
Listen to Sachin Rekhi react to the shrinking S-Curve’s impact on Product and Marketing Strategy and learn how to quickly find product-market fit in an AI world.
Discover how Claire Vo built Chat PRD.
More from Polly (in addition to her Reforge Course and Artifacts listed above):
The Complete AI Product Leader Blueprint (8 weeks) - join the waitlist to be the first notified and get early access discounts for the next cohort in September
The AI Career Boost mailing list: We'll be diving in weekly all summer to each of the 6 core competencies with industry examples, resource guides and more.
The evolution of AI is fundamentally reshaping the role of product managers (PMs). In a recent panel discussion, experts from leading tech companies shared their insights on navigating this rapidly changing landscape.
In this post, we'll dive into six core competencies of the modern AI PM, and how our panel feels about each one. You can also view the recording below.
The Six Core Competencies of an AI Product Manager
Our panel first outlined their agreed-upon core competencies of an AI PM:
Strategy and Vision
Risk and Responsible AI
Data Depth
Model Development and Lifecycle
Evaluation
Getting to Launch
**Related Reading: **The Six Core AI PM Competencies
Effectively, there are many things that won’t stray far away from the fundamentals of product management. But, AI changes the speed, risk, and post-launch iteration for this new age of product management.
Here are the key takeaways that will help you stay ahead in the AI-driven future.
Your Strategy and Vision Should Focus on Improving the Lives of Users
The first core competency that our panel discussed was *strategy and vision. *While this doesn't differ dramatically from a traditional PM competency, AI changes the ways you should look at it.
AI Will Enable Personalized and Dynamic Interfaces
AI’s potential to revolutionize user interfaces is immense. Imagine no longer needing a one-size-fits-all approach. Instead, interfaces could dynamically adapt to each user's specific needs, both expressed and observed. This personalized interaction could eliminate the traditional learning curve, making software more intuitive and efficient.
Dylan Sellberg from HubSpot highlights this shift: "You can imagine completely dynamic interfaces or no interface at all that is entirely personalized to an end user's needs."
The future envisions a seamless integration where users interact with software that understands their workflow and preferences, reducing the need for extensive onboarding and training.
Customer-Centric Design is Critical
AI PMs must focus on delivering 10x value, ensuring that AI products meet real customer needs rather than merely showcasing tech capabilities. AI's promise lies not just in enhancing user experiences but also in streamlining processes and uncovering hidden insights.
Automate the Mundane, Focus on the Creative (Make People’s Lives Better) AI excels at automating tedious tasks, freeing up users to focus on more creative and strategic activities. For instance, HubSpot uses AI to handle repetitive tasks like optimizing SEO or localizing content, allowing users to dedicate their time to crafting engaging content and strategies.
Dylan also notes: "The real home run is actually in the surround sound work that people don't like doing... that is what we call toil work that I think we can really automate."
By taking over these monotonous tasks, AI not only boosts productivity but also enhances job satisfaction by allowing individuals to engage in more meaningful work.
These evolutions in user experience and interface hold some of the most powerful potential for a PM to capitalize on in their strategy and vision.
Prioritize Responsible AI
The second competency our panel discussed was the world of risk and responsibility in AI, arguably one of the biggest new dimensions with this technology.
Take a Stance on User Control and Transparency
Ensuring users have control over AI features is crucial. At Zoom, AI features are launched with default settings turned off, and users are informed when AI is active. This approach not only builds trust but also empowers users to decide when and how AI is utilized.
Ailian Gan from Zoom explains: "We want to create this effect where the user is always in control of how AI is interacting with their content."
This user-centric approach ensures that AI serves as an empowering tool rather than a disruptive force.
Ethical and Transparent Use of Data
Zoom also emphasizes that customer data is never used to train models without explicit consent. This policy, now adopted by many leading companies, underscores the importance of ethical data use and privacy in AI development.
Ailian further states: "We wanted to say, like, your customer data will never be used to train models, period. Full stop."
This clear stance on data privacy fosters trust and sets a standard for the industry. AI PMs must navigate complex privacy and compliance landscapes, particularly when dealing with sensitive industries like healthcare and finance.
Have Guardrails for Generated Content
Responsible AI involves anticipating and mitigating potential misuse of features. For example, Zoom’s forthcoming AI virtual backgrounds must be carefully managed to avoid generating offensive content. This involves modifying user prompts and implementing filters to ensure safe and appropriate outputs.
**Related Reading: **Prompt Engineering Template
By considering potential risks upfront, companies can prevent misuse and ensure their AI applications are both useful and safe.
Embrace Data Depth
The third competency our panel discussed was attention to the data sources your models are trained on, data security, and more.
Detailed Data Analysis - What Are Your Models Trained On?
AI relies heavily on data, making in-depth analysis essential. PMs must delve into data to identify outliers and understand the broader implications. This thorough approach ensures that AI systems are robust and reliable, even in edge cases.
Dylan reflects on this necessity: "Understanding being able to dig in deep and understand like that level of data and the knock-on impacts is important to be able to actually like understand when things are going wrong."
A deep dive into data allows PMs to preemptively address issues and refine their AI models continuously.
Streamline Model Development
The fourth competency our panel discussed was the question of whether to leverage foundation models or to develop proprietary ones. An AI PM needs to understand the tradeoffs.
Utilize Foundation Models to Save Time
Starting with foundation models can expedite development. These models provide a robust starting point, allowing teams to build and validate AI features quickly. Over time, these can be refined and tailored to meet specific customer needs more precisely.
Monitor and Improve Continuously - Iterate from the Foundation Investing in continuous monitoring and improvement of AI models is critical. This involves not only tracking performance but also identifying gaps and opportunities for refinement, ensuring the AI remains effective and relevant.
Dylan emphasizes: "I think almost nobody is investing heavily enough in AI performance right now. That is the kind of next trajectory and curve we see for really good products."
Continuous improvement is key to maintaining the relevance and efficacy of AI solutions.
Implement Rigorous Evaluation
The fifth competency our panel discussed was evaluation - the ability to evaluate metrics related to AI system performance and outputs.
Human-in-the-Loop Evaluations
Despite advances in automated evaluation, human judgment remains invaluable. LinkedIn’s experience with job matching highlighted the importance of involving real users in the evaluation process. This hands-on approach helps fine-tune AI models to better meet user needs and expectations.
Kumarash from LinkedIn shares: "Evaluation is the product in some ways. It defines the voice and tone and the direction that you're taking the product."
Involving humans in the evaluation process ensures that AI outputs align closely with user expectations and real-world scenarios.
Focus on Post-Launch Success
The last competency our panel discussed was how fluid the "launch" of an AI product is. The launch is really just the beginning.
Iterative Beta Testing - Launch Scrappy, Then Improve
A phased approach to launching AI features, including extensive beta testing, is essential. This allows teams to gather feedback, stress-test the features, and refine them based on real-world usage, ensuring a smooth and successful rollout.
**Related Reading: **Bridging the gap from Gen AI POC to production
Ailian discusses Zoom’s approach: "We do a period of internal beta testing where 6,000-7,000 employees are invited to test and beat it up as much as possible and give us feedback."
This iterative process helps identify and address potential issues early, leading to more robust and reliable AI features.
Comprehensive Sales Enablement - Help Your GTM Teams
Effective go-to-market strategies are vital. This includes equipping sales and customer success teams with thorough training and resources, such as detailed FAQs and security white papers, to address customer concerns and drive adoption.
Looking Ahead: The Future of AI Product Management
Specialization and Depth
The AI PM role will likely see increased specialization. PMs will focus on specific areas such as model evaluation, orchestration, and platform development. This depth of expertise will be crucial as AI continues to integrate into various facets of product management.
Kumarash predicts: "There’s going to be a lot of specialization within the AI PM role... identifying something that you do today will help you then be able to prepare yourself to be the specialized version of an AI PM."
Demonstrated Experience and Learning
Dylan emphasized the importance of building and demonstrating practical AI solutions. Real-world projects, even small-scale ones, can significantly bolster a resume. AI PMs must show a track record of learning, experimentation, and problem-solving.
AI as a Productivity Tool
AI will become an indispensable tool for PMs, assisting with drafting documents, creating prototypes, and predicting project timelines. This will enable PMs to focus on higher-level strategic decisions and creative problem-solving.
Ailian envisions: "AI will help you draft first draft everything... it changes how you function as an AI PM if your skill is about how to harness these tools in different ways."
Curating and Judging AI Outputs
The core skills of curation, taste, and judgment will remain critical. As AI takes over more administrative tasks, PMs will need to leverage their human intuition and expertise to guide AI outputs effectively.
The future of product management in the age of AI is bright and full of opportunities. By embracing these changes and continuously evolving, PMs can lead their teams to success and innovation.
Related Resources
Artifacts: Free, real work example from leading experts
PRD Template for AI-driven Features
AI/ML roadmap at Neurons Lab
Product spec for AI-powered malware scan MVP at BitNinja
Using Reforge’s AI to brainstorm around a new use case
GenAI Product Strategy and Roadmap at Spiffy.ai
Guides: Members-only, step by step instruction
Evaluate the value of Gen AI for your product
Evaluate techniques for incorporating LLMs into your product
Define Successful Conversational AI Products
Understand conversational AI technology
Design a Conversational UX Experience
Courses: In-depth upskilling with Reforge experts
Generative AI Products: How to Get from Idea to MVP - Polly Allen and Rupa Chaturvedi
GenAI Product Strategy - Aniket Deosthali
Blog Post + Demo Video - How we built the Reforge Extension!
Podcast: Reforge’s Unsolicited Feedback is available on the platform of your choice
Hear Box CTO, Ben Kus, evaluate AI models and discuss his approach to building enterprise AI tools.
Listen to Sachin Rekhi react to the shrinking S-Curve’s impact on Product and Marketing Strategy and learn how to quickly find product-market fit in an AI world.
Discover how Claire Vo built Chat PRD.
More from Polly (in addition to her Reforge Course and Artifacts listed above):
The Complete AI Product Leader Blueprint (8 weeks) - join the waitlist to be the first notified and get early access discounts for the next cohort in September
The AI Career Boost mailing list: We'll be diving in weekly all summer to each of the 6 core competencies with industry examples, resource guides and more.