Data Normalization Best Practices
Data normalization in BigPanda is the process of standardizing and structuring event data from various monitoring and observability tools into a consistent format.
This process involves:
Standardization: Identify a standard set of tags to map tags and data fields from different sources, ensuring uniformity across all integrations. Consider downstream impacts as you identify the tags.
Integration: Integrate your monitoring tools with these standardized tags in mind. BigPanda has 50+ standard integrations , or you can integrate using the Open Integration Manager (OIM) , Alerts API, Email Parser, or BigPanda Daemon (SNMP).
Cleansing: Ensure data within tags from your monitoring tools are cleansed where possible, such as removing domain names from the host tag names to align with how data is stored in enrichment sources.
Consistency: Adjust Primary and Secondary properties to drive event marshaling, deduplication, and compression
Enrichment: Enhance alert payloads by adding contextual information and consolidating similar tags into a single, consistent tag to ensure uniformity across all alerts.
Integrating observability tools from different vendors can be challenging due to differences in how they tag and report data. Without standardization, the downstream components of the BigPanda pipeline can be negatively impacted. When tags aren’t consistent or normalized, the entire system can be less effective and accurate, leading to slower response times and missed critical alerts.
Normalizing tags prepares you to introduce additional monitoring sources, allowing you to scale and evolve with your organization’s needs. Normalized tags must define and meet minimal criteria for downstream actions, such as ticketing or automation, ensuring that all necessary information is available for these processes to function smoothly.