Hello, this is Andrew, the leader of the Core/Platform team at Checker. In this episode, we will examine the changes in search data systems and enterprise data system architecture, and explore how search technology is being utilized in the field of security based on these changes.

In particular, I have also included my thoughts on the future of corporate security and the development direction of the QueryPie product, based on my experience in the search domain for over 20 years.

‍

‍Changes in Search Data System Architecture‍

Search technology is a system that pre-indexes data in order to quickly find answers to user queries. To do this, data is collected, classified, and processed from various sources to create an index, and related systems are composed of distributed systems for large-scale data processing. Referring to the stream architecture concept featured in the white paper published by AWS can be helpful for understanding.

The architecture of search data systems has gone through many stages of evolution, and currently, various attempts are being made to incorporate deep learning into search. Fast application in similar sentence search, related image search, video recommendation, etc. are representative examples.

In the past, the process of collecting data in a dedicated storage in a simple ETL format and periodically indexing it was common. However, with technological advancements, the introduction of Hadoop systems for big data analysis, and the activation of various clustering and analysis tasks, the accuracy of search has significantly improved.

Subsequently, with the application of deep learning to search, inference systems were built, and deep learning-based results were used in big data clustering and analysis, which became an opportunity to go beyond natural language-based search technology and be utilized in various fields. The search data system pipeline has evolved into a structure that is suitable for continuously responding to various needs.

‍

‍

Changes in corporate data system architecture‍

In line with the digital transformation era, the data system architecture of companies has also undergone innovative changes. This includes the shift to cloud-based services, the adoption of data-centric decision-making, the increasing need for real-time data processing and analysis, the integration of artificial intelligence and machine learning, operation focusing on flexibility and scalability, the establishment of multi and hybrid cloud strategies, and the promotion of a transition to API and micro service-based architectures.

Understanding these changes in corporate data system architecture is crucial in the field of security. In particular, there is a need to deeply consider access control methods for the flow of sensitive data and the various systems used in this process. For example, in cases where sensitive data is stored in SaaS systems and NoSQL, excluding sensitive information from deep learning training data, or building a Private Cloud using K8S systems, the development of access control systems is essential.

‍

‍The Future of Security Technology: Proactive Recommendation Methods‍

Security technology also has a very similar approach to search technology. Security technology is a kind of recommendation system that collects, classifies, processes, and tags data generated from various sources or systems, and informs security personnel of the current situation and enables them to take appropriate measures if necessary. Currently, if known threats were controlled based on rules, in the future, it is expected to evolve into a way of proactively identifying potential threat elements and preventing and preparing for them in advance.

In addition, embedding techniques can be applied in access control systems to transform user behavior into a high-dimensional vector space. This is similar to embedding user search, purchase, and viewing behaviors in online shopping platforms or streaming services to provide personalized recommendation services.

In this process, normal user behavior is clustered in a vector space that shares similar characteristics, while abnormal behavior is embedded in a different vector space. At this time, it can help security personnel identify abnormal vectors quickly and recommend appropriate response methods to them, enabling them to take appropriate measures in advance.

The control method for sensitive data is also similar. Instead of security personnel periodically searching for and configuring data according to changes in the system, it will develop in a direction where data is classified and tagged using automated systems and recommend appropriate response methods to security personnel.

Security technology is changing in a way that can sensitively respond to changes in corporate systems, and CHEQUER is also developing QueryPie products that align with this direction. Building on my experience of operating various systems in the search domain, I will continue to provide the best security products that the current era demands. I ask for your continued interest and encouragement in CHEQUER's future endeavors.

I will come back with more interesting topics. Thank you.