Benchmarking and Performance Analysis

To maintain its performance edge, sandalwood includes a comprehensive benchmarking suite. This toolkit allows developers to measure the overhead of the Python-Fortran bridge, compare backend implementations, and analyze scaling behavior across high-order Taylor series.

Quick Start

The easiest way to run benchmarks is using the included shell wrapper:

# Run a standard comparison of individual operations (Order 8, 4 Variables)
./scripts/benchmarks/benchmark.sh ops

# Run a three-way comparison including "Raw COSY" (direct Fortran timing)
./scripts/benchmarks/benchmark.sh raw

# Run a full parametric sweep with HTML report generation
./scripts/benchmarks/benchmark.sh full

Benchmarking Modes

The suite supports several specialized modes:

  1. Operations (`ops`): Compares the Python (Numba) backend against the COSY (Fortran) backend for basic arithmetic (+, -, *, /) and elementary functions (sin, exp, etc.).

  2. Three-Way Comparison (`raw`): Adds a third competitor: Raw COSY. This executes the logic inside a standalone Fortran binary, measuring the absolute minimum time required by the COSY core. This reveals the “Bridge Overhead” introduced by CTypes and Python marshalling.

    Note

    The raw and full benchmarking modes execute standalone COSY scripts and require the proprietary cosy.fox file (the main COSY macro package). Since this file is proprietary, it is not distributed with Sandalwood. You must place a copy of your licensed cosy.fox in scripts/benchmarks/COSY.fox or set the SANDALWOOD_COSY_SRC environment variable pointing to your COSY source directory before running these benchmarks.

  3. Batch Evaluation (`batch`): Tests the performance of the neval() method. This highlights the efficiency of the OpenMP-optimized batch evaluator in the COSY backend versus the vectorized NumPy/PyTorch implementations.

  4. Parametric Sweep (`full`): Automatically runs a battery of tests across varying orders (2 to 10) and dimensions (4 to 6). This is the primary tool for detecting performance regressions in high-order logic.

Methodology

Accurate benchmarking of Differential Algebra requires handling extremely small timescales and hardware jitter. sandalwood employs the following strategies:

Timing Strategy

  • CPU vs Wall Time: We use time.process_time() to measure only the time spent by the CPU on the current process, ignoring system-level context shifts.

  • Warmup & Repeats: Every benchmark includes a “Warmup” phase to trigger JIT compilation (Numba) or cache heating. Results are averaged over multiple repeats to filter out noise.

  • Internal Fortran Timing: For “Raw COSY”, we use the internal COSY CPUSEC function. This ensures we measure only the mathematical execution time, excluding binary startup and disk I/O.

Memory Profiling

When the --memory flag is used, the suite utilizes Python’s tracemalloc to capture Peak Memory Usage. This is critical for monitoring the “Combinatorial Explosion” of high-order Taylor series and ensuring that the Dense Mode tables don’t exceed available RAM.

Bridge Overhead Mitigation

To ensure fairness when comparing against Raw Fortran: * Expressions are pre-compiled into Python Bytecode (Lambdas) before the timing loop starts to eliminate eval() overhead. * Large arrays are pre-allocated and reused to avoid garbage collection interference.

Reporting Tools

The benchmarking suite generates multiple reports:

  • Markdown Tables: Simplified tables printed to the console and saved in scripts/benchmarks/artifacts/.

  • HTML Reports: The full mode generates a rich dashboard including:

    • System specifications (CPU, RAM, OS).

    • Interactive, sortable performance tables.

    • Speedup ratios (e.g., “COSY is 12.5x faster than Python”).

    • Embedded plots showing scaling behavior.

Extending Benchmarks

To add a new operation to the benchmark suite, modify the generate_benchmark_cases function in scripts/benchmarks/run.py. You must provide a Python expression and its equivalent COSY-compatible string.

cases.append(("NewOp", "mtf.special_op(x)", "SPECIAL(DA(1))"))