Graphext allows you to work seamlessly with very large datasets (many rows and columns) with minimal performance reduction, thanks to the use of advanced technology. Among many other technologies, Graphext utilizes WebAssembly, enabling real-time frontend updates. So far, we have successfully worked with projects exceeding 30 million rows, and we have yet to encounter a limit.
We encourage you to help us find it 😉
Memory optimizations: All datasets are now compressed in memory during runtime. We’ve developed a custom compression algorithm based on the bitpacking technique that allows for random access with single-value granularity, without the need to uncompress the entire dataset or even blocks of it, while keeping overhead to a minimum.
We’re now using more memory-efficient helper data structures for things like caching filters.
Performance optimizations:We now take advantage of the multiple cores available in modern CPUs to speed up processing and let the browser main thread do its job, updating the UI without computation blocking it.
