Polars 2.0 (pola.rs)

Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.

In that regard, I’m still waiting for a credible jq replacement…

from https://github.com/pola-rs/geopolars/tree/main

Comparison with GeoPandas

Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.

GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.

Polars is great, but I'm just too used to the Pandas API to use it as a replacement for the cases where DuckDB is overkill.

https://docs.pola.rs/api/python/stable/reference/expressions...

DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.

It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.

    >>> df = pl.DataFrame({"x": ["a", "b", "c"], "y": ["d", "e", "f"]})
    >>> df.with_columns(new=pl.col.x + " text " + pl.col.y)
    shape: (3, 3)
    ┌─────┬─────┬──────────┐
    │ x   ┆ y   ┆ new      │
    │ --- ┆ --- ┆ ---      │
    │ str ┆ str ┆ str      │
    ╞═════╪═════╪══════════╡
    │ a   ┆ d   ┆ a text d │
    │ b   ┆ e   ┆ b text e │
    │ c   ┆ f   ┆ c text f │
    └─────┴─────┴──────────┘

I tried my best to be as transparent and fair as possible, running everyone with out-of-the-box settings on a third party's queries (DuckDB), a third party's data generator (tpcgen-rs, from the DataFusion guys), on a stock setup available to everyone (AWS machines).

The one exception is that we also ran Polars locked to 32 threads on the large machine (in addition to the out-of-the-box setup), which was to highlight we can do a lot better on small data. We still suffer from a relatively high constant overhead on very high CPU count machines if the data isn't large enough, but I'm working on fixing that. It's possible that DuckDB / DataFusion have similar scaling issues with high thread counts and would do better with 32 threads as well, I didn't test that.

I've spent years of my life running TPC benchmarks. They're useful guidelines, but they're simply not diverse enough to show the whole picture unless your picture is very simple. Certainly no single TPC benchmark in isolation. Maybe you can get a better idea by running all of them.

The largest gaps are around non-relational style data, JSON querying and the like. Nearly every organisation has something in that space now, and TPC-H/TPC-DS don't touch it at all.

For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.

I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.

Apparently about something ))

I mean, it's very clear that the projects have a lot in common, even if building one on top of another directly is not a good idea.

more predictable

when I started, I tried mimicking a human trader. if you want to expand into full quant, be aware of overengineering (past mistake of mine).

you can copy or mimic institutional desk strategies. most of the concepts are fine, but the devil is in the details.

sometimes you open too early or too late. finding that sweet spot, where you don’t want a lot of drawdown, requires a lot of fine-tuning.