Data Processing Engine
The Cloud-Edge Data Processing Engine (DPE) enables ETL tasks to flow unhindered and in a natural and iterative fashion with analytic and relational data functions. This is achieved using the raw data processing horsepower of the engine’s heuristic algorithms and common data storage. Typical data processing solutions consist of a four-stage flow; an ETL system, a RDBMS, an analytics tool and a delivery system. This piecemeal architecture results in significant performance bottlenecks, which are eliminated as a result of the integrated environment of the Cloud-Edge Software Solution. Heuristic algorithms control the compression, sorting, scanning and indexing of data functionalities. This all as part of the ETL process without the need for further programming requirements.
CompressionThe Cloud-Edge Sort compression algorithm, using proprietary techniques compresses data with ratios between 4:1 and 20:1. This enables improved overall machine workload output and provides massive storage savings. Another significant feature is the ability for the data to be analyzed at a detailed level in its compressed state, without losing performance.
ScanningThe DPE uses RAID technology to efficiently move and filter data in parallel, to and from disk I/O subsystems. The scanning algorithms work closely with other low-level algorithms to keep pipelines and caches loaded efficiently.
SortingThe unique Sort heuristic algorithm is a cornerstone of the Cloud-Edge Sort system achieving substantial performance gains. It has benchmarked favorably against both stand-alone Sort Software Solutions and exotic sort algorithms from academia.
IndexingThe dynamic Cloud-Edge Sort indexing algorithm utilizes the parallel processing of scanning and sorting to create indices quickly. This allows for high performance in database queries. Multiple indices are created simultaneously within a single data scan, allowing for rapid data update processes. The index structure is dynamic in that the raw data used by the indices are heuristically examined to create and maintain optimal structure. This index structure is modified automatically to achieve and maintain optimal memory usage and one-disk-access performance. All this data is accessed in the compressed state and hundreds of indices can be kept open simultaneously.