Write Throughput
#1vortex2.70x2070 MB/s
#2parquet1.00x1098 MB/s
#3lance0.74x683 MB/s
Scan Throughput
#1vortex6.78x11.33 GB/s
#2lance1.38x3.14 GB/s
#3parquet1.00x2.98 GB/s
Geomean throughput ratio to Parquet | Aggregate throughput (higher is better)
TotalMBvs Arrow
#1parquetTotal: 1,546.6 MB5.92x
#2vortexTotal: 1,736.7 MB5.31x
#3lanceTotal: 3,324.9 MB2.93x
#4arrow-ipcTotal: 9,561.6 MB1.09x
vs ParquetMinGeoMeanMax
parquet: minimum 1x, geometric mean 1x, maximum 1x
vortex: minimum 0.677x, geometric mean 1.11x, maximum 1.48x
lance: minimum 1.55x, geometric mean 2.02x, maximum 2.74x
arrow-ipc: minimum 3.24x, geometric mean 5.42x, maximum 9.1x
Total compressed size, and geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)
TotalMBvs Arrow
#1vortex-compactTotal: 22,557.7 MB—
#2vortexTotal: 31,478.9 MB—
#3parquetTotal: 30,957.5 MB—
#4duckdbTotal: 60,947.5 MB—
#5lanceTotal: 67,354.0 MB—
vs vortexMinGeoMeanMax
vortex-compact: minimum 0.537x, geometric mean 0.724x, maximum 0.859x
vortex: minimum 1x, geometric mean 1x, maximum 1x
parquet: minimum 0.771x, geometric mean 1.02x, maximum 2.42x
duckdb: minimum 1.04x, geometric mean 1.36x, maximum 2.14x
lance: minimum 1.85x, geometric mean 1.97x, maximum 2.28x
Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)
Hot
#1vortex1.00x855.38 us
#2litdata2.24x1.74 ms
#3lance2.87x3.10 ms
#4arrow-ipc5.39x15.21 ms
#5parquet92.41x77.44 ms
Cold
#1vortex1.27x3.15 ms
#2lance1.79x4.55 ms
#3arrow-ipc2.24x16.82 ms
#4litdata19.64x46.55 ms
#5parquet35.32x78.51 ms
Geomean of take time ratio to fastest across every dataset (lower is better)
Hot
#1vortex1.00x19.95 ms
#2lance27.12x167.68 ms
#3parquet81.79x468.58 ms
Cold
#1vortex1.00x247.63 ms
#2lance2.28x566.55 ms
#3parquet2.57x629.10 ms
Geomean of take time ratio to fastest across every dataset (lower is better)
Performance Summary
#1duckdb:vortex1.11x3.57 s
#2datafusion:vortex1.24x3.80 s
#3duckdb:vortex-compact1.31x4.09 s
#4datafusion:vortex-compact1.51x4.41 s
#5datafusion:parquet2.01x5.92 s
#6duckdb:duckdb2.03x5.34 s
#7duckdb:parquet2.07x5.61 s
#8datafusion:lance17.74x49.11 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.12x335.69 ms
#2duckdb:vortex1.30x380.48 ms
#3duckdb:vortex-compact1.44x416.55 ms
#4datafusion:vortex1.70x524.94 ms
#5datafusion:vortex-compact1.81x547.27 ms
#6datafusion:arrow3.31x1.08 s
#7duckdb:parquet4.71x1.39 s
#8datafusion:lance6.43x2.03 s
#9datafusion:parquet6.59x1.87 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:parquet1.12x5.24 s
#2datafusion:vortex1.24x5.77 s
#3datafusion:vortex-compact1.31x6.12 s
#4duckdb:vortex-compact1.48x6.79 s
#5duckdb:vortex1.56x7.24 s
#6datafusion:lance1.58x7.69 s
#7duckdb:parquet1.65x7.56 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.15x1.01 s
#2duckdb:vortex1.29x1.15 s
#3duckdb:vortex-compact1.47x1.28 s
#4datafusion:vortex1.76x1.78 s
#5datafusion:vortex-compact1.95x1.89 s
#6duckdb:parquet3.24x2.89 s
#7datafusion:parquet4.15x3.60 s
#8datafusion:arrow14.60x16.39 s
#9datafusion:lance24.49x23.52 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex-compact1.11x10.67 s
#2datafusion:parquet1.17x11.75 s
#3duckdb:parquet1.30x12.36 s
#4datafusion:vortex1.31x12.57 s
#5duckdb:vortex-compact2.04x19.92 s
#6duckdb:vortex2.10x20.72 s
#7datafusion:lance3.77x43.02 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.14x7.11 s
#2datafusion:vortex1.56x12.52 s
#3duckdb:parquet1.93x11.32 s
#4datafusion:parquet3.15x19.90 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:parquet1.05x35.58 s
#2datafusion:vortex1.31x45.85 s
#3datafusion:parquet1.36x47.98 s
#4duckdb:vortex4.21x161.27 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.02x1.68 s
#2duckdb:vortex1.38x2.12 s
#3duckdb:vortex-compact1.56x2.38 s
#4duckdb:parquet1.80x2.81 s
#5datafusion:vortex2.21x4.18 s
#6datafusion:vortex-compact2.45x4.73 s
#7datafusion:parquet3.27x6.01 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.06x5.05 s
#2datafusion:vortex1.39x9.60 s
#3duckdb:parquet1.53x7.01 s
#4datafusion:parquet2.27x13.93 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.17x22.91 s
#2duckdb:parquet1.42x27.07 s
#3datafusion:vortex1.45x121.06 s
#4datafusion:parquet2.80x147.95 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:parquet1.29x56.85 s
#2datafusion:vortex1.31x55.40 s
#3duckdb:parquet1.46x54.81 s
#4duckdb:vortex1.55x62.40 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.22x225.41 s
#2duckdb:parquet1.34x146.70 s
#3datafusion:parquet2.12x338.68 s
#4duckdb:vortex2.24x297.02 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.25x3.22 s
#2datafusion:vortex-compact1.45x3.76 s
#3datafusion:parquet2.59x4.08 s
#4duckdb:vortex3.09x8.05 s
#5duckdb:vortex-compact3.46x8.56 s
#6duckdb:parquet6.54x7.19 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.00x517.51 ms
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.04x220.54 ms
#2datafusion:vortex1.26x216.74 ms
#3duckdb:vortex-compact4.00x952.23 ms
#4datafusion:parquet4.39x627.09 ms
#5duckdb:parquet5.25x791.69 ms
#6datafusion:vortex-compact5.52x1.04 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.02x2.91 s
#2datafusion:vortex-compact1.16x3.43 s
#3duckdb:vortex1.64x5.30 s
#4duckdb:vortex-compact1.67x5.23 s
#5duckdb:parquet2.10x5.26 s
#6datafusion:parquet2.49x5.99 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.13x1.62 s
#2duckdb:parquet1.23x1.67 s
#3duckdb:vortex1.31x1.82 s
#4duckdb:vortex-compact1.32x1.86 s
#5datafusion:vortex1.88x11.00 s
#6datafusion:vortex-compact1.88x11.05 s
#7datafusion:parquet2.36x11.43 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.06x208.10 ms
#2datafusion:vortex1.15x259.93 ms
#3duckdb:vortex-compact1.40x265.69 ms
#4datafusion:vortex-compact1.66x462.44 ms
#5duckdb:duckdb1.97x334.67 ms
#6duckdb:parquet2.21x451.47 ms
#7datafusion:parquet3.48x1.26 s
#8datafusion:lance77.91x20.17 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.08x4.81 s
#2duckdb:vortex1.28x5.50 s
#3duckdb:parquet4.87x20.11 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.13x743.21 ms
#2duckdb:duckdb1.24x791.90 ms
#3duckdb:vortex-compact1.46x945.60 ms
#4datafusion:vortex1.48x1.01 s
#5datafusion:vortex-compact1.73x1.15 s
#6datafusion:lance5.58x5.04 s
#7duckdb:parquet8.66x5.58 s
#8datafusion:parquet11.56x7.60 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.06x26.15 s
#2datafusion:vortex1.42x42.02 s
#3duckdb:parquet1.65x37.55 s
#4datafusion:parquet2.73x67.34 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:parquet1.15x495.15 ms
#2duckdb:vortex1.23x477.33 ms
#3datafusion:vortex1.61x638.93 ms
#4duckdb:parquet2.06x790.38 ms
Geomean of query time ratio to fastest (lower is better)
Write Throughput
#1fsst1.00x224 MB/s
#2onpair-120.65x155 MB/s
#3onpair-160.62x149 MB/s
Scan Throughput
#1onpair-161.24x4.54 GB/s
#2onpair-121.24x4.59 GB/s
#3fsst1.00x3.69 GB/s
Geomean throughput ratio to FSST | Aggregate throughput (higher is better)
TotalMBvs Arrow
#1onpair-16Total: 245.4 MB2.82x
#2onpair-12Total: 243.8 MB2.82x
#3fsstTotal: 322.3 MB2.13x
vs FSSTMinGeoMeanMax
onpair-16: minimum 0.607x, geometric mean 0.755x, maximum 0.9x
onpair-12: minimum 0.6x, geometric mean 0.756x, maximum 0.877x
fsst: minimum 1x, geometric mean 1x, maximum 1x
Total file size, and geometric means of file sizes versus Arrow (higher is better) and versus FSST (lower is better)