Write Throughput
#1vortex2.70x2076 MB/s
#2parquet1.00x1095 MB/s
#3lance0.74x678 MB/s
Scan Throughput
#1vortex6.54x13.39 GB/s
#2lance1.43x3.22 GB/s
#3parquet1.00x3.00 GB/s
Geomean throughput ratio to Parquet | Aggregate throughput (higher is better)
TotalMBvs Arrow
#1parquetTotal: 1,546.6 MB5.92x
#2vortexTotal: 1,649.8 MB5.73x
#3lanceTotal: 3,327.6 MB2.93x
#4arrow-ipcTotal: 9,561.6 MB1.09x
vs ParquetMinGeoMeanMax
parquet: minimum 1x, geometric mean 1x, maximum 1x
vortex: minimum 0.673x, geometric mean 1.03x, maximum 1.37x
lance: minimum 1.56x, geometric mean 2.02x, maximum 2.74x
arrow-ipc: minimum 3.24x, geometric mean 5.42x, maximum 9.1x
Total compressed size, and geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)
TotalMBvs Arrow
#1vortex-compactTotal: 22,554.8 MB—
#2vortexTotal: 30,121.5 MB—
#3parquetTotal: 30,957.3 MB—
#4duckdbTotal: 60,942.2 MB—
#5lanceTotal: 67,213.4 MB—
vs vortexMinGeoMeanMax
vortex-compact: minimum 0.607x, geometric mean 0.796x, maximum 0.884x
vortex: minimum 1x, geometric mean 1x, maximum 1x
parquet: minimum 0.847x, geometric mean 1.13x, maximum 2.42x
duckdb: minimum 1.19x, geometric mean 1.52x, maximum 2.25x
lance: minimum 2.1x, geometric mean 2.16x, maximum 2.34x
Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)
Hot
#1vortex1.00x836.94 us
#2litdata2.30x1.73 ms
#3lance2.97x3.13 ms
#4arrow-ipc5.19x16.07 ms
#5parquet95.82x78.54 ms
Cold
#1vortex1.32x3.16 ms
#2lance1.88x4.59 ms
#3arrow-ipc2.16x16.17 ms
#4litdata20.64x46.94 ms
#5parquet36.45x78.20 ms
Geomean of take time ratio to fastest across every dataset (lower is better)
Hot
#1vortex1.00x20.89 ms
#2lance30.83x190.13 ms
#3parquet82.77x479.15 ms
Cold
#1vortex1.04x403.13 ms
#2lance1.65x636.81 ms
#3parquet1.73x647.21 ms
Geomean of take time ratio to fastest across every dataset (lower is better)
Performance Summary
#1duckdb:vortex1.09x3.53 s
#2datafusion:vortex1.26x3.82 s
#3duckdb:vortex-compact1.30x4.11 s
#4datafusion:vortex-compact1.49x4.44 s
#5datafusion:parquet1.99x5.91 s
#6duckdb:duckdb2.00x5.26 s
#7duckdb:parquet2.05x5.63 s
#8datafusion:lance17.56x48.71 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.11x335.82 ms
#2duckdb:vortex1.31x384.94 ms
#3duckdb:vortex-compact1.44x418.19 ms
#4datafusion:vortex1.70x523.37 ms
#5datafusion:vortex-compact1.80x550.13 ms
#6datafusion:arrow3.29x1.08 s
#7duckdb:parquet4.69x1.39 s
#8datafusion:lance6.40x2.03 s
#9datafusion:parquet6.54x1.87 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:parquet1.13x5.74 s
#2datafusion:vortex-compact1.26x6.43 s
#3datafusion:vortex1.32x6.80 s
#4datafusion:lance1.45x7.69 s
#5duckdb:parquet1.60x8.09 s
#6duckdb:vortex-compact1.67x9.22 s
#7duckdb:vortex1.72x9.30 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.15x1.02 s
#2duckdb:vortex1.32x1.17 s
#3duckdb:vortex-compact1.49x1.30 s
#4datafusion:vortex1.77x1.79 s
#5datafusion:vortex-compact1.97x1.91 s
#6duckdb:parquet3.24x2.89 s
#7datafusion:parquet4.15x3.61 s
#8datafusion:arrow14.59x16.39 s
#9datafusion:lance24.53x23.58 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.27x13.00 s
#2duckdb:parquet1.33x13.42 s
#3datafusion:parquet1.36x15.89 s
#4datafusion:vortex-compact1.40x15.74 s
#5duckdb:vortex-compact2.12x22.18 s
#6duckdb:vortex2.17x22.67 s
#7datafusion:lance3.53x43.02 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.13x7.34 s
#2datafusion:vortex1.52x12.50 s
#3duckdb:parquet1.85x11.24 s
#4datafusion:parquet3.03x19.83 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:parquet1.05x36.69 s
#2datafusion:vortex1.31x47.30 s
#3datafusion:parquet1.40x51.40 s
#4duckdb:vortex4.04x162.52 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.02x1.70 s
#2duckdb:vortex1.38x2.14 s
#3duckdb:vortex-compact1.56x2.41 s
#4duckdb:parquet1.81x2.85 s
#5datafusion:vortex2.20x4.18 s
#6datafusion:vortex-compact2.41x4.72 s
#7datafusion:parquet3.24x6.02 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.05x5.12 s
#2datafusion:vortex1.41x9.72 s
#3duckdb:parquet1.48x6.96 s
#4datafusion:parquet2.21x13.93 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.15x23.89 s
#2duckdb:parquet1.31x27.02 s
#3datafusion:vortex1.47x123.17 s
#4datafusion:parquet2.59x147.68 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.26x58.72 s
#2datafusion:parquet1.34x65.36 s
#3duckdb:parquet1.41x59.04 s
#4duckdb:vortex1.57x68.83 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.22x233.94 s
#2duckdb:parquet1.29x150.54 s
#3datafusion:parquet2.21x360.67 s
#4duckdb:vortex2.27x306.98 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.24x3.33 s
#2datafusion:vortex-compact1.41x3.84 s
#3datafusion:parquet2.48x4.10 s
#4duckdb:vortex3.06x8.15 s
#5duckdb:vortex-compact3.42x8.79 s
#6duckdb:parquet6.50x7.46 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.00x527.92 ms
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.05x162.01 ms
#2datafusion:vortex1.24x162.19 ms
#3duckdb:vortex-compact4.57x952.47 ms
#4datafusion:parquet4.98x621.61 ms
#5datafusion:vortex-compact5.98x1.08 s
#6duckdb:parquet6.00x793.38 ms
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:vortex1.02x3.07 s
#2datafusion:vortex-compact1.18x3.71 s
#3duckdb:vortex-compact1.77x5.71 s
#4duckdb:vortex1.83x5.84 s
#5duckdb:parquet2.18x5.79 s
#6datafusion:parquet2.73x7.02 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.14x1.63 s
#2duckdb:parquet1.24x1.67 s
#3duckdb:vortex1.31x1.83 s
#4duckdb:vortex-compact1.32x1.86 s
#5datafusion:vortex1.87x11.01 s
#6datafusion:vortex-compact1.90x11.04 s
#7datafusion:parquet2.37x11.42 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.08x208.18 ms
#2datafusion:vortex1.14x252.29 ms
#3duckdb:vortex-compact1.40x268.63 ms
#4datafusion:vortex-compact1.67x504.59 ms
#5duckdb:duckdb1.96x336.23 ms
#6duckdb:parquet2.16x440.02 ms
#7datafusion:parquet3.44x1.26 s
#8datafusion:lance77.31x20.06 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:duckdb1.07x4.82 s
#2duckdb:vortex1.29x5.57 s
#3duckdb:parquet4.86x20.21 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.10x715.14 ms
#2duckdb:duckdb1.22x778.16 ms
#3duckdb:vortex-compact1.41x917.57 ms
#4datafusion:vortex1.51x1.03 s
#5datafusion:vortex-compact1.75x1.16 s
#6datafusion:lance5.57x5.04 s
#7duckdb:parquet8.64x5.58 s
#8datafusion:parquet11.55x7.59 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1duckdb:vortex1.07x25.96 s
#2datafusion:vortex1.43x41.75 s
#3duckdb:parquet1.66x37.52 s
#4datafusion:parquet2.74x67.27 s
Geomean of query time ratio to fastest (lower is better)
Performance Summary
#1datafusion:parquet1.15x496.76 ms
#2duckdb:vortex1.23x476.95 ms
#3datafusion:vortex1.76x715.54 ms
#4duckdb:parquet2.02x775.68 ms
Geomean of query time ratio to fastest (lower is better)
Write Throughput
#1fsst1.00x224 MB/s
#2onpair-120.65x156 MB/s
#3onpair-160.64x154 MB/s
Scan Throughput
#1onpair-161.24x4.57 GB/s
#2onpair-121.24x4.61 GB/s
#3fsst1.00x3.69 GB/s
Geomean throughput ratio to FSST | Aggregate throughput (higher is better)
TotalMBvs Arrow
#1onpair-16Total: 245.4 MB2.82x
#2onpair-12Total: 243.8 MB2.82x
#3fsstTotal: 322.3 MB2.13x
vs FSSTMinGeoMeanMax
onpair-16: minimum 0.607x, geometric mean 0.755x, maximum 0.9x
onpair-12: minimum 0.6x, geometric mean 0.756x, maximum 0.877x
fsst: minimum 1x, geometric mean 1x, maximum 1x
Total file size, and geometric means of file sizes versus Arrow (higher is better) and versus FSST (lower is better)