IBM Databand: Self-learning for anomaly detection

Almost a year ago, IBM faced a data validation issue in one of its time-sensitive merger and acquisition data streams. We faced several challenges while working to resolve the issue, including troubleshooting, identifying issues, modifying data flows, changing downstream data pipelines, and performing ad-hoc execution of automated workflows.

Improved data resolution and monitoring efficiency through Databand

After the immediate problem was resolved, retrospective analysis revealed that proper data validation and intelligent monitoring could have alleviated the problem and shortened resolution time. Rather than developing a custom solution just for the immediate problem, IBM sought a broadly applicable data validation solution that could handle this scenario as well as potential issues that might otherwise be overlooked.

That’s when I discovered one of our recent acquisitions: IBM® Databand® for data observability. Unlike traditional monitoring tools that use rule-based monitoring or hundreds of custom-developed monitoring scripts, Databand provides self-learning monitoring. Observe historical data behavior and identify deviations that exceed certain thresholds. This machine learning capability allows users to monitor data with minimal rule configuration and anomaly detection, even when there is limited knowledge about the data or its behavioral patterns.

Optimize data flow observability with Databand’s self-learning monitoring

Databand considers the past behavior of data flows and flags suspicious activity while alerting users. IBM has integrated Databand into a data flow consisting of more than 100 pipelines. It provided easily observable status updates for every run and pipeline, and more importantly, highlighted errors. This allowed us to focus and accelerate the resolution of data flow incidents.

Databand for data observability uses self-learning to monitor:

• Change schema: When a schema change is detected, Databand displays it in the dashboard and sends an alert. Anyone who works with data has probably encountered scenarios where schema changes are made to a data source, such as adding or removing columns. These changes impact workflow, which in turn affects downstream data pipeline processing, creating a ripple effect. Databand can analyze your schema history and immediately alert you to any anomalies, preventing potential outages.
• Service Level Agreement (SLA) Impact: Databand displays data lineage and identifies downstream data pipelines affected by data pipeline errors. If you have a defined SLA for data delivery, alerts help you recognize and maintain SLA compliance.
• Performance and runtime ideals: Databand monitors the duration of your data pipeline execution and learns to detect anomalies and flag them when necessary. Users do not need to know the duration of the pipeline. Databand learns from historical data.
• situation: Databand monitors execution status, including whether it failed, was canceled, or succeeded.
• Data validation: Databand observes ranges of data values ​​over time and sends alerts when anomalies are detected. It includes common statistics such as mean, standard deviation, minimum, maximum, and quartiles.

Innovative Databand notifications for improved data pipelines

Users can set up alerts using the Databand user interface, which features an intuitive dashboard to monitor and support workflow without complexity. Directed acyclic graphs provide deep visibility, which is useful when processing large data pipelines. This all-in-one system allows your support team to focus on areas that need attention and accelerate deliverables.

Through the acquisition and merger of IBM Enterprise Data, we were able to strengthen our data pipeline with Databand and we haven’t looked back. We’re excited to offer this innovative software to help you identify data incidents earlier, resolve them faster, and provide your business with more reliable data.

Deliver trusted data through continuous data observability Read Gartner report