Heterogeneous Structured Pruning Under Resource Constraints for Hardware-Efficient Large Language Models

Author

Cameron Barker1,∗, Andreas Paraskeva2,∗, Jan van Rijn2, Elliot J. Crowley3, Henry Gouk1

Published

September 21, 2026

1 School of Informatics, University of Edinburgh
2 Leiden Institute of Advanced Computer Science, Leiden University
3 School of Engineering, University of Edinburgh

∗ Equal contribution

Overview

Pruning a language model reduces its number of weights, but the resulting latency and memory use depend on how those weights are stored and processed on the target hardware.

Heterogeneous Structured Pruning (HSP) chooses a sparsity structure and density for each sublayer under a latency or memory budget. It combines dense, unstructured, 2:4, block-sparse, head-pruned, and channel-pruned options in a single training-free search, using importance scores to minimise proxy pruning loss.

Diagram of the HSP framework: an importance metric, latency or memory constraint, and candidate sparsity structures feed into a solver that selects per-sublayer structures and densities.

The HSP framework combines an importance metric, a resource constraint, and a heterogeneous search space to select a pruning strategy for each sublayer.

Selected results

Six plots comparing pruning methods: WikiText-2 perplexity in the left column and mean zero-shot accuracy in the right column, against model density, profile-backed latency, and analytical memory.

Quality–resource trade-offs for OPT-2.7B on the Jetson Orin Nano Super GPU, with a 1024-token prefill and 128-token decode workload. The plots compare WikiText-2 perplexity and mean seven-task zero-shot accuracy against density, profile-backed latency, and analytical memory, normalised to the dense model.

Four panels showing per-layer pruning choices for GPU latency, CPU latency, GPU memory, and CPU memory objectives. Each model mixes sparsity structures and varies density across its projections.

Structures selected for OPT-2.7B on the Jetson Orin Nano Super at a budget ratio of 0.8 under the mixed workload. Columns compare GPU and CPU; rows compare latency and memory objectives. Bar height indicates retained density and colour indicates the selected structure.

Funding acknowledgements

This work was supported by the APRIL AI Hub. Cameron Barker is supported by the UKRI EPSRC Centre for Doctoral Training in Machine Learning Systems (EP/Y03516X/1).

This publication is part of the project LESSEN with project number NWA.1389.20.183 of the research program NWA ORC 2020/21 which is (partly) financed by the Dutch Research Council (NWO).

Henry Gouk was supported by the Royal Academy of Engineering under the Research Fellowship programme.