LakeSail @LakeSail

Spark rewritten in Rust. A new standard for unified compute, reimagined for modern data and AI infrastructure. lakesail.com San Francisco Joined September 2024

Tweets

112
Followers

2K
Following

5
Likes

100

LakeSail @LakeSail

3 weeks ago

AWS Lab's benchmark: awslabs.github.io/data-on-eks/do… LakeSail's benchmark: lakesail.com/blog/tpcds-ben…

0 1 2 115 1

View Details

LakeSail @LakeSail

3 weeks ago

We just published a derived TPC-DS for Sail at 3 TB and 1 TB. Here is a comparison of our benchmark to another independently published benchmark on Spark and Spark + Gluten/Velox by AWS.

1 2 6 556 3

View Details

LakeSail @LakeSail

3 weeks ago

If you want help with migrating off Spark, we provide support that guarantees your workloads will be cheaper, faster, and correct… or you pay nothing. lakesail.com/spark-migratio…

0 0 1 51 0

View Details

A data platform that's simple, transparently priced, and ridiculously fast. The LakeSail Platform runs your Spark SQL and Python jobs on a Rust-native runtime. It's faster than Spark, drastically cuts hardware costs, and scales from a single node to a multi-node cluster. It reads and writes your Delta Lake and Iceberg tables where they already live, so your lakehouse stays open and your data stays in place. For those who want to run Sail in production most efficiently, with all the operations handled for you, this platform was built for you.

1 2 6 291 1

View Details

LakeSail @LakeSail

3 weeks ago

Sail ran derived TPC-DS at 3TB and 1TB across a distributed cluster, and the results are the latest evidence for our vision in rethinking distributed compute. Hadoop pioneered open-source distributed processing, built around disk. Spark moved intermediate data into memory, though its shuffle stayed disk-first. When we started Sail, we envisioned another shift in distributed compute, toward doing even more of the work in memory, with an engine built entirely in Rust. We built the first version of Sail's distributed architecture in November 2024, with Arrow Flight carrying shuffle data between workers, and redesigned the cluster control plane earlier this year. Arrow Flight shuffle can look surprising at first. A TPC-DS query with dozens of stages and a few hundred partitions can have more than a million logical shuffle streams active in the cluster. Arrow Flight is built on gRPC and HTTP/2, so one TCP connection per pair of workers carries them all. The same approach has since come up in the DataFusion Comet and Riffle communities as well, and we are glad to have started proving this direction two years ago. The full benchmark ran in 1,289 seconds at 3TB and 444 seconds at 1TB, with no shuffle data written to disk. On similar hardware and settings, Sail outperforms the published TPC-DS results for Spark and Spark accelerators at these scales. The post covers the results, the design, and the full setup: lakesail.com/blog/tpcds-ben…

1 2 4 125 1

View Details

Alexy 🤍💙🤍 @ChiefScientist

3 weeks ago

Sail runs Doom with a single query! I ported SQLDoom to @LakeSail 's Sail, the AI-native expansion of @ApacheSpark whole ecosystem to Rust. rust.ai/sail-runs-doom/

0 2 2 191 0

View Details

LakeSail @LakeSail

3 weeks ago

No JVM. 10x faster. 98% cost reduction.

0 24 259 7.4M 57

View Details

LakeSail @LakeSail

3 weeks ago

Sail is the first open-source engine to ship with built-in Jev support! Check out the docs to get started: docs.lakesail.com/sail/latest/gu…

0 3 8 11K 1

View Details

LakeSail @LakeSail

3 weeks ago

Learn more and watch the full presentation here: lakesail.com/blog/adyen-spa…

0 0 2 70 0

View Details

LakeSail @LakeSail

3 weeks ago

Two results their Principal Engineer, Santosh, shared: • A CI test suite went from 1,335 seconds on Spark to 78.44 seconds on Sail, roughly 17x faster. • An Airflow job's median runtime dropped from about 57 minutes to 8.5 minutes, using fewer cores and less memory.

1 0 3 39 0

View Details

LakeSail @LakeSail

3 weeks ago

Adyen’s data team explored DuckDB and Polars but landed on Sail as a new engine. They explain their reasoning in a recent presentation they gave at PyData Amsterdam.

1 1 6 236 2

View Details

Shehab Amin @shehab_amins

4 weeks ago

Sail is the first open-source engine to ship with built-in Jev support lakesail.com/blog/jev-built…

1 2 7 163 1

View Details

LakeSail @LakeSail

4 weeks ago

Blog + examples: lakesail.com/blog/jev-built…

0 0 3 60 1

View Details

LakeSail @LakeSail

4 weeks ago

Sail now includes Jev built natively into the engine, the first of its kind. Any Spark SQL or DataFrame query can ask Jev about every row of a table, with nothing extra to install. The answers come back as typed columns you can filter, join, and aggregate like any other data. The same query can run on your laptop or across a cluster. Check out the full release and some examples of it in action. Link in thread.

1 3 7 255 2

View Details

LakeSail @LakeSail

a month ago

More engineering details: lakesail.com/blog/sail-cele…

0 0 2 107 1

View Details

LakeSail @LakeSail

a month ago

A shuffle in Sail, before and after Apache Celeborn.

1 0 3 212 2

View Details

LakeSail @LakeSail

a month ago

Sail 0.7.1 adds the first ever Rust implementation of Apache Celeborn as a shuffle backend. If you already run Celeborn for your Spark workloads, Sail uses that cluster as it stands. The same masters, same workers, nothing is changed on the Celeborn side. Read more on the engineering in our blog: lakesail.com/blog/sail-cele…