TornadoVM 7 Enables GPU-Native Workloads from Java Applications (infoq.com)

90 pointsby pjmlp3 days ago19 comments

mikepapadim 18 hours ago

https://github.com/beehive-lab/TornadoVM this is main repo.

You can use it with any JDK21+ and Nvidia or Apple hardware and their Nvidia libs or MLX libs.

Akseleu_omirbay 16 hours ago

And what about TPU support? Is it compatible with Google TPUs or strictly limited to gpu backends like CUDA and Metal?

mikepapadim 16 hours ago

no TPU support yet, just NVIDIA, Apple and any other HW thats supports OpenCL

zmmmmm 11 hours ago

Seems like a nice win even if it just does the example they have:

    void matmul(Matrix2DFloat A, Matrix2DFloat B, Matrix2DFloat C, int size) {
      for (@Parallel int i=0; i<size; i++) {
        for (@Parallel int j=0; j<size; j++) {
          float sum = 0f;
          for (int k=0; k<size; k++)
            sum += A.get(i,k)*B.get(k,j);
          C.set(i, j, sum);
        }
      }
    }

But the elephant in the room is the Vector API. Is that going to end up like logging - a standardised API nobody uses because it came 10 years late to the party and isn't as good as what was already established?

And why confuse everybody by calling it a VM when it ships as a JAR file that runs in an existing VM?

russli1993 9 hours ago

what about tiling? seems better to call into existing kernels, fused megakernels than rewrite kernel in java.

zmmmmm 7 hours ago

i might be overly optimistic, but I'm assuming it'll do the tiling by itself with the above. As long as there's not an algorithmic reason to do it a specific way, in theory you could just optimise it for the target hardware.

pjmlp 6 hours ago

I actually use the logging API, it is good enough, and maybe I am old enough not to like to cowboy install dependencies if I can avoid it.

the_arun 18 hours ago

Does this mean we can use Java to build AI infrastructure?

vlowrian 18 hours ago

Or an abstract AI infrastructure factory?

mikepapadim 17 hours ago

https://github.com/beehive-lab/jitllm there is already an vllm-like inference engine that uses tornadovm as the GPU backend all written in java

kitd 16 hours ago

I'll be disappointed if this isn't named AbstractAiInfrastructureFactory

pjmlp 16 hours ago

These kind of complaints get tiresome, here for some Objective-C fun,

https://github.com/Quotation/LongestCocoa

keeda 16 hours ago

That’s probably the next thing after Software Factories.

mike_hearn 15 hours ago

Yes (or any JVM lang). There is also this:

https://qxotic.ai/

... which is a single JAR that does SIMD accelerated inferencing of local models, dependency free. Everything is written in Java from the tokenizer up. It can also use Metal on macOS. Small models can be surprisingly powerful these days and all you need is a fast CPU. It works not only for LLMs but also image models, video models and speech.

Remarkably, it is significantly faster than llama.cpp on some tasks like prefill (decode is memory bandwidth constrained so language hardly makes a difference). At least as long as you run on GraalVM (JVM or native image, both work). So you can actually beat C++ performance for AI in Java.

It's impressive stuff.

vips7L 7 hours ago

Why GraalVM specifically? Is it something special with the Graal JIT?

mike_hearn 2 hours ago

It optimizes more aggressively, yes.

amir734jj 18 hours ago

Basically JAX for Java instead of Python

mikepapadim 17 hours ago

Many components for a jax for java are already tested in https://github.com/beehive-lab/jitllm

mathisfun123 14 hours ago

"your scientists were so preoccupied with whether they could they didn't stop to think if they should"