InfoQ Homepage AI, ML & Data Engineering Content on InfoQ
-
Easy Implementation of GDPR with Aspect Oriented Programming
GDPR compliance should be a default feature in every application that handles PII (Personally Identifiable Information). Most organizations have an impression that GDPR is a luxury feature that needs special tools to implement. But, we can see that the frameworks and design patterns we already use in our everyday development can very well be used to implement the GDPR rules.
-
AIOps: Site Reliability Engineering at Scale
AIOps can simplify and streamline processes which can reduce the mental burden on employees while improving communication and collaboration between departments.
-
The Wonders of Postgres Logical Decoding Messages
In this article, author Gunnar Morling discusses Postgres database's logical decoding function to retrieve the messages from write-ahead log, process them, and relay them to external consumers, with help of use cases like outbox, audit logs and replication slots.
-
Moving towards a Future of Testing in the Metaverse
In this article, Tariq King describes the metaverse concept, discusses its key engineering challenges and quality concerns, and then walks through recent technological advances in AI and software testing that are helping to mitigate these challenges. To wrap up, he shares some of his thoughts on the role of software testers as we move towards a future of testing in the metaverse.
-
Understanding and Applying Correspondence Analysis
Customer segments, personality profiles, social classes, and age generations are examples of effective references to larger groups of people sharing similar characteristics. Correspondence analysis (CA) is a multivariate analysis technique that projects categorical data into a numeric feature space which captures most of the variability in the data by fewer dimensions.
-
How I Contributed as a Tester to a Machine Learning System: Opportunities, Challenges and Learnings
Have you ever wondered about systems based on machine learning? In those cases, testing takes a backseat. And even if testing is done, it’s done mostly by developers themselves. A tester’s role is not clearly portrayed. Testers usually struggle to understand ML-based systems and explore what contributions they can make. This is a journey of assuring quality of ML-based systems as a tester.
-
Understanding and Debugging Deep Learning Models: Exploring AI Interpretability Methods
ML interpretability refers to a user's ability to explain decisions made by an ML system. Interpretability increases confidence in the model, reduces bias, and ensures that model is compliant and ethical. In this article, author Andrew Hoblitzell discusses several methods of ML interpretability and dives deep into Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Values.
-
Design Pattern Proposal for Autoscaling Stateful Systems
In this article, Rogerio Robetti discusses the challenges in auto-scaling stateful storage systems and proposes an opinionated design solution to automatically scale up (vertical) and scale out (horizontal) from a single node up to several nodes in a cluster with minimum configuration and interference of the operator.
-
InfoQ Software Trends Report: Major Trends in 2022 and What to Watch for in 2023
2022 was another year of significant technological innovations and trends in the software industry and communities. The InfoQ podcast co-hosts met last month to discuss the major trends from 2022, and what to watch for in 2023. This article is a summary of the 2022 software trends podcast.
-
DynamoDB Data Transformation Safety: from Manual Toil to Automated and Open Source
Data transformation remains a continuous challenge in engineering and built upon manual toil. The open source utility Dynamo Data Transform was built to simplify and build safety and guardrails into data transformation for DynamoDB based systems––built upon a robust manual framework that was then automated and open sourced. This article discusses the challenges with Data Transformation.
-
Create Your Distributed Database on Kubernetes with Existing Monolithic Databases
The next challenge for databases is to run them on Kubernetes to become cloud neutral. However, they are more difficult to manage than the application layer, since Kubernetes is designed for stateless applications. Apache ShardingSphere is the ecosystem to transform any database into a distributed database system and enhance it with sharding, elastic scaling, encryption features, and more.
-
Apache DolphinScheduler in MLOps: Create Machine Learning Workflows Quickly
In this article, author discusses data pipeline and workflow scheduler Apache DolphinScheduler and how ML tasks are performed by Apache DolphinScheduler using Jupyter and MLflow components.
QCon New York: Level-up on emerging software trends.
Don’t miss your opportunity to learn about key emerging software trends from senior software practitioners. Discover case studies, insights, real-world best practices and solutions in software development & tech leadership.