This week on the Data Futurology podcast, we have an in-depth conversation with Conor O’Neill, the Head of Data Science at Compare The Market exploring his career journey and current role leveraging data and innovating with machine learning.
When O’Neill landed at Compare The Market, he quickly found himself in a senior data role within an organisation that needed to both transform and mature its approach to data. On the podcast, O’Neill walks through the various stages of transformation, and getting the rest of the organisation aligned with that vision.
He also shares some use cases that Compare The Market is effectively leveraging data for, as well as how they have been building ML products. He explains how he involves data scientists in this process and offers advice on building ML as a product when it comes to planning, delivery and infrastructure.
Finally, O’Neill shares some thoughts on the difference between a data scientist’s role and that of a senior manager, and how this shifts the perspective and how a data professional will look at projects. He then rounds out the conversation with some thoughts about where data science is heading as a profession.
For anyone interested in data science, O’Neill’s unconventional journey into and through the profession is both interesting and inspiring. Enjoy the show!
Connect with Conor: https://www.linkedin.com/in/conoroneill1/
Thank you to our sponsor, Talent Insights Group!
Join us for our next events Advancing AI and Data Engineering Sydney (5-7 September):
https://www.datafuturology.com/events
Join our Slack Community: https://join.slack.com/t/datafuturologycircle/shared_invite/zt-z19cq4eq-ET6O49o2uySgvQWjM6a5ng
What we discussed
2:26: Felipe introduces Conor O’Neill.
3:23: O’Neill shares his journey from astrophysics to data science.
6:49: In astrophysics, the data sets that scientists work on are massive. O’Neill shares some insights about how he managed data in that role.
8:40: O’Neill shares his journey at Compare The Market so far.
12:04: O’Neill shares some information about a current data project that he and his team are working on.
18:08: Compare The Market had to do significant foundational work in transformation. O’Neill shares insights into that process.
21:18: O’Neill shares his experience in getting the Compare The Market organisation aligned behind their data vision.
25:12: O’Neill explains the value of having data scientists involved at the earliest stages of transformation design.
28:44: O’Neill describes his experience in moving from a data scientist role to heading a team, and the differences between these roles.
32:56: O’Neill explains some of the thinking that goes into reusing data projects, as well as how they decide the projects to not follow through.
34:04: Getting a model in front of the end users and driving adoption is a critical step – O’Neill explains how he has approached it for Compare The Market.
37:54: O’Neill overviews the various consumers of the work done by the data team, and how the data team needs to think about each of them.
40:51: Tips and guidance for creating ML as a product to be consumed internally
45:48: O’Neill shares some thoughts on how the data science industry is evolving.
Key Quotes
“We’ve been on a transformational journey now for a little over a year, and that’s been really good. We’ve been migrating off our legacy on-prem stack to Databricks. We’ve also been focused on getting the right people, and then also establishing a process, because if you just change the tool, you haven't fixed the issues, typically.”
“You don't want your control group to be too large and you then miss opportunities. But you also don't want it to be so small that you don't get sufficient data. That's where the algorithm behind our recommendation system controls that, to optimise according to our confidence that we are or are not exceeding the required threshold, and adjust the weighting of the control group accordingly.”
#249 - Generative AI in Recruitment: Bridging the Gap Between Automation and Authenticity
#248: Navigating the Frontier of Generative AI in Business
#247: Navigating the Ethical Waters of Data and AI - Insights from NAB's Head of Privacy and Data Ethics
#246: Unlocking Value with Generative AI
#245 - Becoming a Successful Data Analytics and AI Leader
#244: Navigating Data Quality: Insights from the Chief Operator of Data Quality Camp
#243 Mastering DataOps and MLOps: Building a Strong Foundation for Success and Future Growth
#242: Tell me about the future of AI… Here Be Dragons?
#241 - Building AI systems with quality, holistic data
#240: Overcoming the challenges facing modern data engineering teams
#239: Building better business culture around AI
#238: Transforming Education with AI Advancements with Alex Jenkins
#237: Evolving data culture to deliver sustainable business impact, with Niall Keating, General Manager for Technology Data Platforms, Sportsbet
#235: Maximising the productivity of the data-led enterprise with UNSW, EG Australia and Compare the Market
#234: Innovating with Data in Healthcare: Part Two
#233: Innovating with Data: Part One, with the Head of Data Science at Maurice Blackburn Lawyers, Jonas Christensen
#232: Getting buy in and investment from senior execs for your data & AI projects, with Brian Ferris, Chief Data, Analytics and Technology Officer at Loyalty New Zealand
#231: Revolutionising Property Technology with Modular Analytics, with General Manager, Innovation & Advanced Analytics of Investa Property Group
#230: From Walmart to ASB Bank: Achieving some of the largest data transformations in the world, with Bora Arslan
Create your
podcast in
minutes
It is Free
Insight Story: Tech Trends Unpacked
Zero-Shot
Fast Forward by Tomorrow Unlocked: Tech past, tech future
Black Wolf Feed (Chapo Premium Feed Bootleg)
Bannon`s War Room