Between a line of array code and a GPU doing it there may be many programs, each turning one language into a lower one, and most of them are invisible. This post makes them visible for one small program. It follows a threshold and two ReLUs, written in X_eTaL, down to the instructions a GPU runs, by Mojo onto NVIDIA, Apple and AMD and onto the CPU, and down two other roads, CUDA C++ and OpenCL. The listings themselves are long, so they live in the repositories; this post is the guided tour, with a link at every stop.

Resource Link
The whole walk Top down: one X_eTaL program to five instruction sets, every level and listing
The Mojo road modular-fork: rust-demo
The OpenCL road X_eTaL-gpu
Earlier AL #7: X_eTaL on GPUs · AL #10: through Mojo
Comments Discord

The program

The Accel library’s threshold demo: an array, the array where it is above 0.5 and zero elsewhere, two ReLUs, and a select by an integer mask.

"ac:" u̲se< "Accel"
x ← ((r̲ange 8) ÷ 4) − 1.0
x
0.5 ᵃᶜt̲hreshold x
ᵃᶜr̲elu x
n ← (r̲ange 8) − 4
ᵃᶜr̲elu n
(n m̲od 2) ᵃᶜk̲eep n

X_eTaL’s own evaluator prints this, and every road below that runs has to print the same:

-0.75 -0.5 -0.25 0.0 0.25 0.5 0.75 1.0
-0.0 -0.0 -0.0 0.0 0.0 0.0 0.75 1.0
0.0 0.0 0.0 0.0 0.25 0.5 0.75 1.0
0 0 0 0 1 2 3 4
-3 0 -1 0 1 0 3 0

The levels

Level What it is Made by Listing
X_eTaL the program above a person threshold.xtl
Core the form X_eTaL’s evaluator runs X_eTaL’s loader walkthrough, level 1
XIR 19 lines of const, map and cast X_eTaL-gpu’s translator accel-threshold.xir
Mojo one fused kernel and a host function rust-demo’s emitter accel-threshold.mojo
PTX, NVIDIA NVIDIA’s portable assembly Mojo, through LLVM the build log’s PTX
AIR, Apple Apple’s IR for Metal Mojo stage0.air.ll
AMDGCN, AMD RDNA3 assembly, listing only Mojo stage0.gfx1100.amdgcn
arm64, CPU the M1 Max’s CPU Mojo closures.s
CUDA C++ → PTX → SASS the C++ road rust-demo’s second emitter, then nvcc .cu · .ptx · .sass
OpenCL C the run-time road X_eTaL-gpu’s emitter, then the driver threshold.cl

The top two levels are X_eTaL’s own. X_eTaL-gpu’s translator walks the Core form like a small evaluator: whatever it can compute now, such as r̲ange 8, becomes a constant, the library calls are inlined, and what has to run on the device becomes XIR, one operation per line:

%div = map div %l %r
%sub = map sub %div %r_2
%gt = map gt %sub %r_3
%f = cast f64 %gt
%mul = map mul %sub %f

Every operation is element by element, so the whole program is one pass. rust-demo’s emitter is a table from each XIR operation to one line of Mojo, %div becoming var v_div = v_l / Float32(4.0), fused into one GPU function with one thread per element, plus a host function that makes the buffers, copies, launches and copies back. Nothing in the Mojo file says which GPU it is for.

How Mojo uses CUDA, and goes around it

“CUDA” is three things usually sold together: a compiler, nvcc, from C++ to PTX; an assembler, ptxas, from PTX to the card’s machine code; and the driver, which allocates memory, copies and launches. Mojo replaces the first: mojo build compiles each kernel a second time, for the GPU, through an intrinsic in its standard library, kgen.compile_offload, and LLVM’s NVPTX back end, and stores the PTX in the library it builds. It brings its own copy of the second, a bundled PTX compiler that needs NVIDIA driver 580 or later (for older drivers, MODULAR_NVPTX_COMPILER_PATH points it at a system ptxas). The third it uses as it is: at run time the library loads the installed driver and hands it the kernel through Modular’s runtime library. No CUDA Toolkit was installed on the machine that built it. On a Mac the same file goes to Metal, and for AMD to AMD’s code, from the same compiler.

One line, six spellings

The first line of the program, x ← ((r̲ange 8) ÷ 4) − 1.0, as each road wrote it. Every one turned the divide into a multiply by 0.25; they differ in whether the multiply and the subtract become one instruction:

Road The instructions  
Mojo → PTX (NVIDIA) mul.rn.f32 by 0.25, add.f32 of -1.0 two
nvcc → PTX (NVIDIA) fma.rn.f32 … 0.25, -1.0 fused
OpenCL driver → PTX (NVIDIA) fma.rn.f32 … 0.25, -1.0 fused
Mojo → AIR (Apple) fdiv by 4.0, fsub contract 1.0 two, fusing left to Apple’s driver
Mojo → AMDGCN (AMD) v_mul_f32 by 0.25, v_add_f32 of -1.0 two
Mojo → arm64 (CPU) fmadd with 0.25 and -1.0 fused

A fused multiply-add rounds once instead of twice. Here it cannot matter, since 0.25 is exact, and every road that ran gave the same eight numbers; on other numbers it is a choice each compiler makes for you. (nvcc --fmad=false writes the two instructions separately; both PTX files are kept.)

The line where they differ most

The last line, (n m̲od 2) ᵃᶜk̲eep n, is where the targets part ways. m̲od is floored, so -3 mod 2 is 1, and Mojo’s road carries X_eTaL’s integers in 32-bit floats, so each target converts to a 64-bit integer, takes the remainder, fixes the sign, and converts back:

Target Instructions for the remainder Why
PTX (Mojo and nvcc) about 10 conversions, shifts and masks, a select for the sign
AIR (Apple) about 10 the same in LLVM IR, with @air.convert calls
AMDGCN (RDNA3) about 30 no instruction converts a float to a 64-bit integer or back, so each conversion is spelled out
arm64 9 64-bit conversions and conditional instructions in hardware

The OpenCL road, which keeps X_eTaL’s integers as 64-bit integers from end to end, needs no conversions at all: its PTX does n − 4, the ReLU and the remainder with integer instructions (add.s64, max.s64, shifts and masks). The same line of X_eTaL, from a few instructions to thirty, depending on the road and the chip. The counts come from the walkthrough, which reads each listing line by line.

Each road in brief

NVIDIA, through Mojo. The PTX for sm_120a, written by LLVM’s NVPTX back end, finished into machine code by the bundled compiler, and run on the RTX 5060 Ti: X_eTaL’s five lines exactly (the PTX).

NVIDIA, through CUDA C++ and nvcc. A second emitter in rust-demo writes the same XIR as CUDA C++, with the same one-line-per-operation table and a fix for C++’s %, which truncates where X_eTaL floors. nvcc 13.4 compiles it to PTX and, with ptxas, to the card’s machine code (SASS); built as a native program it gives X_eTaL’s five lines with zero mismatches. Building it needs the CUDA Toolkit; running it needs only the card and its driver (handoff).

Apple, through Mojo. The same .mojo file, built on the M1 Max, becomes AIR, Apple’s intermediate representation for Metal, which is LLVM’s own IR with Apple’s intrinsics. It runs on the M1 Max and matches exactly. The last step, from AIR to the GPU’s own instructions, happens inside Apple’s driver when the function is first loaded, and has no public listing (handoff).

AMD, through Mojo. No AMD card is installed, so this is a listing only: nothing ran. Mojo compiled the same file for gfx1100 (RDNA3) on the Mac, with no AMD device, and wrote its AMDGCN assembly: the same short float arithmetic, 32-wide waves like NVIDIA’s warps, and the expensive remainder above (handoff). The C++ road onto AMD, HIP with hipcc, would be the counterpart of the CUDA C++ road; it is not shown.

The CPU, through Mojo. The same file built with -D XIR_CPU=true hands each stage to the CPU as a parallel range over the elements instead of a grid of GPU threads. It ran on the M1 Max’s CPU and matched exactly; its arm64 body fuses the first line into one fmadd (handoff).

OpenCL, compiled at run time. X_eTaL-gpu’s first backend skips Mojo and writes OpenCL C, a dialect of C99, from the same XIR, and ships the text: the driver compiles it when the program runs (clBuildProgram), with no toolkit involved. Asked afterward for what it compiled (CL_PROGRAM_BINARIES), the two drivers answer differently. NVIDIA’s returns 2,252 bytes of PTX, written by the compiler inside the driver, with the fused fma and the integer remainder above (handoff). Apple’s returns 2,108 bytes that hold no compiled code at all: a property list, the format of macOS settings files, holding the compiler options and the source, the same 27 lines of OpenCL C it was given. Its compiled kernel never leaves the driver, and the one public trace of how the driver works is its error log, which names a translator from OpenCL C to Metal, cl2Metal (handoff). Both runs gave X_eTaL’s five lines exactly.

The toolchains side by side

  CUDA ROCm (HIP) Metal OpenCL Mojo
kernel language CUDA C++ HIP C++ Metal Shading Language OpenCL C Mojo
compiled by nvcc, ahead of time hipcc or amdclang++, ahead of time, for a named target xcrun metal, ahead of time the driver, at run time mojo build, ahead of time (LLVM)
what it produces PTX and/or machine code, in a fat binary an AMD GPU code object AIR, packed into a .metallib nothing shipped but the source PTX, AMD GPU code or AIR, in the library
last step to machine code ptxas at build time, or the driver at load done ahead of time Apple’s driver, at first load (no public listing) inside the driver NVIDIA: a bundled PTX compiler, or the driver
needed to build CUDA Toolkit ROCm Xcode’s Metal toolchain nothing Mojo
needed to run NVIDIA driver (the CUDA runtime is linked statically by default, except on Windows) ROCm’s HIP runtime (libamdhip64) macOS (Metal is built in) a driver with OpenCL the GPU’s driver: NVIDIA 580 or later, AMD 6.3.3 or later; macOS 15 or later

Sources: NVIDIA’s nvcc documentation, AMD’s HIP compilers page, Apple’s Metal tools, and Mojo’s requirements in the fork.

Next

A future post may investigate Qualcomm GPUs on a developer board, and other NPUs.