MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing

Nils Quetschlich1, Lukas Burgholzer1, and Robert Wille1,2

1Chair for Design Automation, Technical University of Munich, Germany
2Software Competence Center Hagenberg GmbH (SCCH), Austria

Find this paper interesting or want to discuss? Scite or leave a comment on SciRate.


Quantum software tools for a wide variety of design tasks on and across different levels of abstraction are crucial in order to eventually realize useful quantum applications. This requires practical and relevant benchmarks for new software tools to be empirically evaluated and compared to the current state of the art. Although benchmarks for specific design tasks are commonly available, the demand for an overarching cross-level benchmark suite has not yet been fully met and there is no mutual consolidation in how quantum software tools are evaluated thus far. In this work, we propose the $\textit{MQT Bench}$ benchmark suite (as part of the $\textit{Munich Quantum Toolkit}$, MQT) based on four core traits: (1) cross-level support for different abstraction levels, (2) accessibility via an easy-to-use web interface ( and a Python package, (3) provision of a broad selection of benchmarks to facilitate generalizability, as well as (4) extendability to future algorithms, gate-sets, and hardware architectures. By comprising more than 70,000 benchmark circuits ranging from 2 to 130 qubits on four abstraction levels, MQT Bench presents a first step towards benchmarking different abstraction levels with a single benchmark suite to increase comparability, reproducibility, and transparency.

MQT Bench comes as an easy-to-use website and a Python Package while its implementation is open-source available on GitHub.

To utilize quantum computers for the different application domains, the respective to-be-solved problem must be encoded into a quantum circuit. Afterwards, that circuit needs to be executed to determine the desired solution. For that, quantum software tools are essential, e.g., to classically simulate the considered quantum circuit or to compile it before it can be executed on a quantum computer. Whenever such a quantum software tool is proposed, it is important to empirically evaluate its performance and to compare it to the state of the art. For that purpose, MQT Bench (as part of the Munich Quantum Toolkit, MQT) is proposed. MQT Bench provides over 70,000 benchmarks on various abstraction levels (depending on what level the to-be-evaluated software tool operates on) ranging from 2 to 130 qubits aiming for increased comparability, reproducibility, and transparency.

► BibTeX data

► References

[1] Alwin Zulehnerand Robert Wille ``Advanced Simulation of Quantum Computations'' IEEE Trans. on CAD of Integrated Circuits and Systems (2019).

[2] D Michael Miller, Mitchell A Thornton, and David Goodman, ``A Decision Diagram Package for Reversible and Quantum Circuit Simulation'' Int'l Conf. on Evolutionary Computation (2006).

[3] Stefan Hillmich, Alwin Zulehner, Richard Kueng, Igor L. Markov, and Robert Wille, ``Approximating decision diagrams for quantum circuit simulation'' ACM Transactions on Quantum Computing 3, 1–21 (2022).

[4] Stefan Hillmich, Alwin Zulehner, and Robert Wille, ``Concurrency in DD-based quantum circuit simulation'' Asia and South Pacific Design Automation Conf. (2020).

[5] Lukas Burgholzer, Hartwig Bauer, and Robert Wille, ``Hybrid Schrödinger-Feynman simulation of quantum circuits with decision diagrams'' Int'l Conf. on Quantum Computing and Engineering (2021).

[6] Lukas Burgholzer, Alexander Ploier, and Robert Wille, ``Simulation paths for quantum circuit simulation with decision diagrams: What to learn from tensor networks, and what not'' IEEE Trans. on CAD of Integrated Circuits and Systems (2022).

[7] Aleks van de Kissingerand Renaud Vilmart ``Classical simulation of quantum circuits with partial and graphical stabiliser decompositions'' (2022).

[8] John Brennan, Momme Allalen, David Brayford, Kenneth Hanley, Luigi Iapichino, Lee J. O'Riordan, Myles Doyle, and Niall Moran, ``Tensor network circuit simulation at exascale'' (2021).

[9] Trevor Vincent, Lee J. O'Riordan, Mikhail Andrenkov, Jack Brown, Nathan Killoran, Haoyu Qi, and Ish Dhand, ``Jet: Fast quantum circuit simulations with parallel task-based tensor-network contraction'' (2021).

[10] Jaekyung Imand Seokhyeong Kang ``Graph Partitioning Approach for Fast Quantum Circuit Simulation'' Asia and South Pacific Design Automation Conf. 690–695 (2023).

[11] Danylo Lykov, Roman Schutski, Alexey Galda, Valerii Vinokur, and Yurii Alexeev, ``Tensor Network Quantum Simulator With Step-Dependent Parallelization'' (2020).

[12] Hans De Raedt, Fengping Jin, Dennis Willsch, Madita Willsch, Naoki Yoshioka, Nobuyasu Ito, Shengjun Yuan, and Kristel Michielsen, ``Massively parallel quantum computer simulator, eleven years later'' Computer Physics Communications 237, 47–61 (2019).

[13] Sergey Bravyiand David Gosset ``Improved classical simulation of quantum circuits dominated by Clifford gates'' Physical Review Letters 116, 250501 (2016).

[14] Thomas Häner, Damian S. Steiger, Krysta Svore, and Matthias Troyer, ``A software methodology for compiling quantum programs'' Quantum Science and Technology (2018).

[15] Matthew Amyand Vlad Gheorghiu ``staq—A full-stack quantum processing toolkit'' Quantum Science and Technology (2020).

[16] Alexander S. Green, Peter LeFanu Lumsdaine, Neil J. Ross, Peter Selinger, and Benoı̂t Valiron, ``Quipper: A Scalable Quantum Programming Language'' ACM SIGPLAN Not. (2013).

[17] N. Quetschlich, L. Burgholzer, and R. Wille, ``Predicting Good Quantum Circuit Compilation Options'' Int'l Conf. on Quantum Software (2023).

[18] N. Quetschlich, L. Burgholzer, and R. Wille, ``Compiler Optimization for Quantum Computing Using Reinforcement Learning'' Design Automation Conf. (2023).

[19] Tom Peham, Nina Brandl, Richard Kueng, Robert Wille, and Lukas Burgholzer, ``Depth-optimal synthesis of Clifford circuits with SAT solvers'' (2023).

[20] Lukas Burgholzer, Sarah Schneider, and Robert Wille, ``Limiting the search space in optimal quantum circuit mapping'' Asia and South Pacific Design Automation Conf. (2022).

[21] Robert Wille, Lukas Burgholzer, and Alwin Zulehner, ``Mapping quantum circuits to IBM QX architectures using the minimal number of SWAP and H operations'' Design Automation Conf. (2019).

[22] Stefan Hillmich, Alwin Zulehner, and Robert Wille, ``Exploiting Quantum Teleportation in Quantum Circuit Mapping'' Asia and South Pacific Design Automation Conf. 792–797 (2021).

[23] Alwin Zulehner, Alexandru Paler, and Robert Wille, ``An efficient methodology for mapping quantum circuits to the IBM QX architectures'' IEEE Trans. on CAD of Integrated Circuits and Systems (2019).

[24] Alwin Zulehnerand Robert Wille ``Compiling SU(4) quantum circuits to IBM QX architectures'' Asia and South Pacific Design Automation Conf. 185–190 (2019).

[25] Irfansha van de Shaik ``Optimal layout synthesis for quantum circuits as classical planning'' (2023).

[26] Ji Liu, Ed Younis, Mathias Weiden, Paul Hovland, John Kubiatowicz, and Costin Iancu, ``Tackling the Qubit Mapping Problem with Permutation-Aware Synthesis'' (2023).

[27] Robert Willeand Lukas Burgholzer ``MQT QMAP: Efficient quantum circuit mapping'' Int'l Symp. on Physical Design (2023).

[28] Chi Zhang, Ari B. Hayes, Longfei Qiu, Yuwei Jin, Yanhao Chen, and Eddy Z. Zhang, ``Time-optimal qubit mapping'' Int'l Conf. On Architectural Support for Programming Languages and Operating Systems (2021).

[29] Prakash Murali, Jonathan M. Baker, Ali Javadi-Abhari, Frederic T. Chong, and Margaret Martonosi, ``Noise-adaptive compiler mappings for noisy intermediate-scale quantum computers'' Int'l Conf. On Architectural Support for Programming Languages and Operating Systems 1015–1029 (2019).

[30] Alexander Cowtan, Silas Dilkes, Ross Duncan, Alexandre Krajenbrink, Will Simmons, and Seyon Sivarajah, ``On the qubit routing problem'' Theory of quantum computation, communication and cryptography (2019).

[31] Seyon Sivarajah, Silas Dilkes, Alexander Cowtan, Will Simmons, Alec Edgington, and Ross Duncan, ``t|ket⟩: a retargetable compiler for NISQ devices'' Quantum Science and Technology 6, 014003 (2021).

[32] Bochen Tanand Jason Cong ``Optimal layout synthesis for quantum computing'' Int'l Conf. on CAD (2020).

[33] Gushu Li, Yufei Ding, and Yuan Xie, ``Tackling the qubit mapping problem for NISQ-era quantum devices'' Int'l Conf. On Architectural Support for Programming Languages and Operating Systems (2019).

[34] Kaitlin N. Smithand Mitchell A. Thornton ``A Quantum Computational Compiler and Design Tool for Technology-Specific Targets'' Int'l Symp. on Computer Architecture (2019).

[35] S. Yamashitaand I. L. Markov ``Fast Equivalence-Checking for Quantum Circuits'' Int'l Symp. on Nanoscale Architectures (2010).

[36] Philipp Niemann, Robert Wille, David Michael Miller, Mitchell A. Thornton, and Rolf Drechsler, ``QMDDs: Efficient Quantum Function Representation and Manipulation'' IEEE Trans. on CAD of Integrated Circuits and Systems (2016).

[37] Ross Duncan, Aleks Kissinger, Simon Perdrix, and John van de Wetering, ``Graph-theoretic Simplification of Quantum Circuits with the ZX-calculus'' Quantum (2020).

[38] Lukas Burgholzer, Rudy Raymond, and Robert Wille, ``Verifying Results of the IBM Qiskit Quantum Circuit Compilation Flow'' Int'l Conf. on Quantum Computing and Engineering (2020).

[39] Tom Peham, Lukas Burgholzer, and Robert Wille, ``Equivalence checking of quantum circuits with the ZX-Calculus'' Journal of Emerging and Selected Topics in Circuits and Systems (2022).

[40] Tom Peham, Lukas Burgholzer, and Robert Wille, ``Equivalence checking of parameterized quantum circuits: Verifying the compilation of variational quantum algorithms'' Asia and South Pacific Design Automation Conf. (2023).

[41] Lukas Burgholzerand Robert Wille ``Advanced equivalence checking for quantum circuits'' IEEE Trans. on CAD of Integrated Circuits and Systems (2021).

[42] Lukas Burgholzer, Richard Kueng, and Robert Wille, ``Random stimuli generation for the verification of quantum circuits'' Asia and South Pacific Design Automation Conf. (2021).

[43] Lukas Burgholzerand Robert Wille ``Handling non-unitaries in quantum circuit equivalence checking'' Design Automation Conf. (2022).

[44] Wei Chun-Yu, Tsai Yuan-Hung, Jhang Chaio-Shan, and Jiang Jie-Hong, ``Accurate BDD-based Unitary Manipulation for Scalable and Robust Quantum Circuit Verification'' Design Automation Conf. (2022).

[45] Runzhou Tao, Yunong Shi, Jianan Yao, Xupeng Li, Ali Javadi-Abhari, Andrew W. Cross, Frederic T. Chong, and Ronghui Gu, ``Giallar: Push-button verification for the Qiskit quantum compiler'' Int'l Conf. on Programming Language Design and Implementation 641–656 (2022).

[46] Robert Willeand Lukas Burgholzer ``Verification of Quantum Circuits'' (2022).

[47] S.-A. Wang, C.-Y. Lu, I-M. Tsai, and S.-Y. Kuo, ``An XQDD-based verification method for quantum circuits'' IEICE Trans. Fundamentals 584–594 (2008).

[48] Xin Hong, Mingsheng Ying, Yuan Feng, Xiangzhen Zhou, and Sanjiang Li, ``Approximate Equivalence Checking of Noisy Quantum Circuits'' Design Automation Conf. 637–642 (2021).

[49] Yu-Fang Chen, Kai-Min Chung, Ondřej Lengál, Jyun-Ao Lin, Wei-Lun Tsai, and Di-De Yen, ``An automata-based framework for verification and bug hunting in quantum circuits'' Programming Languages (2023).

[50] Hsiao-Lun Liu, Yi-Ting Li, Yung-Chih Chen, and Chun-Yao Wang, ``A Robust Approach to Detecting Non-Equivalent Quantum Circuits Using Specially Designed Stimuli'' Asia and South Pacific Design Automation Conf. 696–701 (2023).

[51] Qiskit contributors ``Qiskit: An Open-source Framework for Quantum Computing'' (2023).

[52] Cirq Developers ``Cirq'' (2021) See full list of authors on Github: https:/​/​github .com/​quantumlib/​Cirq/​graphs/​contributor.

[53] Robert S. Smith, Michael J. Curtis, and William J. Zeng, ``A Practical Quantum Instruction Set Architecture'' (2016).

[54] Thomas Lubinski, Sonika Johri, Paul Varosy, Jeremiah Coleman, Luning Zhao, Jason Necaise, Charles H. Baldwin, Karl Mayer, and Timothy Proctor, ``Application-Oriented Performance Benchmarks for Quantum Computing'' IEEE Transactions on Quantum Engineering (2023).

[55] Teague Tomesh, Pranav Gokhale, Victory Omole, Gokul Subramanian Ravi, Kaitlin N. Smith, Joshua Viszlai, Xin-Chuan Wu, Nikos Hardavellas, Margaret R. Martonosi, and Frederic T. Chong, ``SupermarQ: A Scalable Quantum Benchmark Suite'' IEEE Int'l Symp. on High-Performance Computer Architecture (2022).

[56] Ang Li, Samuel Stein, Sriram Krishnamoorthy, and James Ang, ``QASMBench: A Low-Level Quantum Benchmark Suite for NISQ Evaluation and Simulation'' ACM Transactions on Quantum Computing (2022).

[57] R. Wille, D. Große, L. Teuber, G. W. Dueck, and R. Drechsler, ``RevLib: An Online Resource for Reversible Functions and Reversible Circuits'' Int'l Symp. on Multi-Valued Logic (2008) RevLib is available at http:/​/​

[58] Andrew Cross, Ali Javadi-Abhari, Thomas Alexander, Niel De Beaudrap, Lev S. Bishop, Steven Heidel, Colm A. Ryan, Prasahnt Sivarajah, John Smolin, Jay M. Gambetta, and Blake R. Johnson, ``OpenQASM 3: A Broader and Deeper Quantum Assembly Language'' ACM Transactions on Quantum Computing (2022).

[59] Thomas Grurl, Richard Kueng, Jürgen Fuß, and Robert Wille, ``Stochastic Quantum Circuit Simulation Using Decision Diagrams'' Design, Automation and Test in Europe (2021).

[60] Benjamin Villalonga, Sergio Boixo, Bron Nelson, Christopher Henze, Eleanor Rieffel, Rupak Biswas, and Salvatore Mandrà, ``A flexible high-performance simulator for verifying and benchmarking quantum circuits implemented on real hardware'' npj Quantum Information (2019).

[61] Tyson Jones, Anna Brown, Ian Bush, and Simon C. Benjamin, ``QuEST and High Performance Simulation of Quantum Computers'' Scientific Reports (2019).

[62] Thomas Grurl, Jurgen Fuß, and Robert Wille, ``Noise-aware quantum circuit simulation with decision diagrams'' IEEE Trans. on CAD of Integrated Circuits and Systems (2023).

[63] Marcos Yukio Siraichi, Vinícius Fernandes dos Santos, Sylvain Collange, and Fernando Magno Quintao Pereira, ``Qubit Allocation'' Int'l Symp. on Code Generation and Optimization (2018).

[64] Dominik Janzing, Pawel Wocjan, and Thomas Beth, ````Non-identity check'' is QMA-complete'' Int. J. Quantum Inform. (2005).

[65] Andrew W. Cross, Lev S. Bishop, John A. Smolin, and Jay M. Gambetta, ``Open Quantum Assembly Language'' (2017).

[66] Peter W Shor ``Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer'' SIAM review (1999).

[67] Lov K Grover ``A fast quantum mechanical algorithm for database search'' Symp. on Theory of Computing (1996).

[68] M. Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C. Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R. McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, and Patrick J. Coles, ``Variational Quantum Algorithms'' Nature Reviews Physics (2021).

[69] Boxi Li, Shahnawaz Ahmed, Sidhant Saraogi, Neill Lambert, Franco Nori, Alexander Pitchford, and Nathan Shammah, ``Pulse-level noisy quantum circuits with QuTiP'' Quantum (2022).

[70] Jernej Rudi Finžgar, Philipp Ross, Leonhard Hölscher, Johannes Klepsch, and Andre Luckow, ``QUARK: A Framework for Quantum Computing Application Benchmarking'' Int'l Conf. on Quantum Computing and Engineering (2022).

Cited by

[1] Waldemir Cambiucci, Regina Melo Silveira, and Wilson Vicente Ruggiero, 2023 IEEE International Conference on Quantum Computing and Engineering (QCE) 268 (2023) ISBN:979-8-3503-4323-6.

[2] Hoa T. Nguyen, Muhammad Usman, and Rajkumar Buyya, "QFaaS: A Serverless Function-as-a-Service framework for Quantum computing", Future Generation Computer Systems 154, 281 (2024).

[3] Guoqiang Shu, Yangyang Fei, Zongjiang Yi, Jinchen Xu, Fudong Liu, and Zheng Shan, "Design Method of Meta Operators Based on Equivalence Class Distribution", Journal of Physics: Conference Series 2620 1, 012009 (2023).

[4] WenBo Liu, Fei Wang, Han Lin, and JianDong Shang, "A user-centric quantum benchmarking test suite and evaluation framework", Quantum Information Processing 22 11, 397 (2023).

[5] Mirko Amico, Helena Zhang, Petar Jurcevic, Lev S. Bishop, Paul Nation, Andrew Wack, and David C. McKay, "Defining Standard Strategies for Quantum Benchmarks", arXiv:2303.02108, (2023).

[6] Nils Quetschlich, Lukas Burgholzer, and Robert Wille, "Predicting Good Quantum Circuit Compilation Options", arXiv:2210.08027, (2022).

[7] Tom Peham, Lukas Burgholzer, and Robert Wille, "Equivalence Checking of Quantum Circuits With the ZX-Calculus", IEEE Journal on Emerging and Selected Topics in Circuits and Systems 12 3, 662 (2022).

[8] Nils Quetschlich, Lukas Burgholzer, and Robert Wille, "Compiler Optimization for Quantum Computing Using Reinforcement Learning", arXiv:2212.04508, (2022).

[9] Konrad Jałowiecki, Paulina Lewandowska, and Łukasz Pawela, "PyQBench: A Python library for benchmarking gate-based quantum computers", SoftwareX 24, 101558 (2023).

[10] Tom Peham, Lukas Burgholzer, and Robert Wille, "On Optimal Subarchitectures for Quantum Circuit Mapping", arXiv:2210.09321, (2022).

[11] Robert Wille and Lukas Burgholzer, "MQT QMAP: Efficient Quantum Circuit Mapping", arXiv:2301.11935, (2023).

[12] Lukas Burgholzer, Alexander Ploier, and Robert Wille, "Simulation Paths for Quantum Circuit Simulation with Decision Diagrams", arXiv:2203.00703, (2022).

[13] Saikat Basu, Arnav Das, Amit Saha, Amlan Chakrabarti, and Susmita Sur-Kolay, "FragQC: An Efficient Quantum Error Reduction Technique using Quantum Circuit Fragmentation", arXiv:2310.00444, (2023).

[14] Philipp Seitz, Manuel Geiger, and Christian B. Mendl, "Multithreaded parallelism for heterogeneous clusters of QPUs", arXiv:2311.17490, (2023).

[15] Thomas Lubinski, Joshua J. Goings, Karl Mayer, Sonika Johri, Nithin Reddy, Aman Mehta, Niranjan Bhatia, Sonny Rappaport, Daniel Mills, Charles H. Baldwin, Luning Zhao, Aaron Barbosa, Smarak Maity, and Pranav S. Mundada, "Quantum Algorithm Exploration using Application-Oriented Performance Benchmarks", arXiv:2402.08985, (2024).

[16] Jingcheng Shen, Linbo Long, Masao Okita, and Fumihiko Ino, "A Reorder Trick for Decision Diagram Based Quantum Circuit Simulation", arXiv:2211.07110, (2022).

The above citations are from Crossref's cited-by service (last updated successfully 2024-03-02 12:39:17) and SAO/NASA ADS (last updated successfully 2024-03-02 12:39:17). The list may be incomplete as not all publishers provide suitable and complete citation data.