Exploring common challenges
ML models that go into production need to handle large volumes of data, often in real-time. Unlike traditional technologies, AI and ML deal with probabilistic outcomes – in other words, what is the most likely or unlikely result.
Therefore, the moving parts of the ML model need close monitoring and swift action when deployed to ensure accuracy, performance, and user satisfaction.
Data scientists must grapple with three key influencing factors to ensure the proper development of ML models:
- Data quality: Data can come from various sources in different volumes and formats. Since ML models are built on data, the quality, completeness, and semantics of data is critical in production environments.
- Model decay: In ML models, data patterns change as the business environment evolves. This evolution leads to a lower prediction accuracy of models trained and validated on outdated data. Such degradation of predictive model performance is known as concept drift, which makes creating, testing, and deploying ML models challenging.
- Data locality: Data locality and access patterns are used to improve the performance of a given algorithm. However, such ML models might not work correctly in production due to the difference in the quality metrics.
These factors push ML practitioners to adopt a 'change anything, change everything' approach – but this often leads to more problems.
Data-science teams waste time and effort navigating technology and infrastructure complexities. Costs increase due to communication and collaboration issues between engineering and data-science teams, and because of the trade-off between achieving business goals and providing stable and resilient infrastructure platforms, projects slow. On average, it could take between three months and a year to deploy ML models due to changing business objectives that require changes to the entire ML pipeline.