Partial Indexing Applied to Search and Join Problems
Abstract
Due to the increasing volume of data and the diversity of processing requirements, there is now a growing trend to move away from on-the-fly data processing and instead focus on executing queries or their main aspects using pre-stored and prepared results. In many cases, DBMSs try to solve performance issues by increasing memory usage; however, it is essential to consider memory conservation while retaining the results of methods based on approaches such as indexing, hashing, and neural algorithms. This article discusses a method for improving the efficiency of search tasks in large tables. The proposed method is based on partial indexing of elements near convergence centers and the introduction of metadata concepts for these centers. Such clustering, with stored metadata for the centers around which the next intermediate nodes are arranged, can reduce memory usage for indexing. Firstly, this approach eliminates the need for nested indexing, which can lead to significant spatial costs. Secondly, this method can provide the ability to use one indexing for different combinations of column presence in search patterns without losing much search efficiency during indexing. Correct application of this approach can effectively process tables with different search needs for various column groups, for which storing indexing for each major query type or group of queries can lead to significant memory costs and performance loss when working with large memory blocks, the growth of which is not linear.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication policy of the journal is based on traditional ethical principles of the Russian scientific periodicals and is built in terms of ethical norms of editors and publishers work stated in Code of Conduct and Best Practice Guidelines for Journal Editors and Code of Conduct for Journal Publishers, developed by the Committee on Publication Ethics (COPE). In the course of publishing editorial board of the journal is led by international rules for copyright protection, statutory regulations of the Russian Federation as well as international standards of publishing.
Authors publishing articles in this journal agree to the following: They retain copyright and grant the journal right of first publication of the work, which is automatically licensed under the Creative Commons Attribution License (CC BY license). Users can use, reuse and build upon the material published in this journal provided that such uses are fully attributed.
