AI-Based Anomaly Detection Frameworks in Distributed Enterprise Data Systems

Abstract

Enterprise information systems typically employ a decentralized architecture based on data stored across numerous heterogeneous computing environments. These environments are also employed by other organizations that provide services to a significant user base. With such user bases come volume and frequency of operations that are unprecedented. Furthermore, systems that allow the execution of user-controlled queries can have unpredictable types and patterns of operations. Consequently, unified enterprise data systems may lag behind such developments. Artificial intelligence can counteract human operators’ limitations on detecting atypical situations in these decentralized systems. However, current applications often result from piecemeal isolated initiatives by data scientists from diverse parts of the organization, which quickly become technical debts. Four decisive aspects of fully supporting the atypical event detection process with artificial intelligence and its implications on enterprise information systems have emerged from a synthesis of the academic literature over the last several decades.First, a comprehensive taxonomy of current solutions is essential to manage the large number of proposals, as the expressed needs of enterprises imply the possibility of artificial intelligence detecting atypical events in any part of the data systems. Second, defining general characteristics of the enterprise data systems’ architecture is key because many of the proposed solutions are strongly dependent on these characteristics, especially aspects related to the sources of data and their pipelines. Third, the data requirements and operational principles of the adapted technical solutions must be covered. Fourth, underlying requirements for data governance, privacy, and security must be considered, together with the implications of regulatory pressures for the application and development of artificial intelligence.