A volumetric framework for quantum computer benchmarks

Robin Blume-Kohout and Kevin C. Young

Quantum Performance Laboratory, Sandia National Laboratories, Albuquerque, NM 87185 and Livermore, California 94550

Find this paper interesting or want to discuss? Scite or leave a comment on SciRate.


We propose a very large family of benchmarks for probing the performance of quantum computers. We call them $\textit{volumetric benchmarks}$ (VBs) because they generalize IBM's benchmark for measuring quantum volume [1]. The quantum volume benchmark defines a family of $\textit{square}$ circuits whose depth $d$ and width $w$ are the same. A volumetric benchmark defines a family of $\textit{rectangular}$ quantum circuits, for which $d$ and $w$ are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — $(w,d)$ pairs — to test suites $\mathcal{C}$$\textit(w,d)$. A test suite is an ensemble of test circuits that share a common structure. The test suite $\mathcal{C}$ for a given circuit shape may be a single circuit $C$, a specific list of circuits $\{C_1\ldots C_N\}$ that must all be run, or a large set of possible circuits equipped with a distribution $Pr(C)$. The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the $d$ vs $w$ trade-off for the processor being benchmarked.

► BibTeX data

► References

[1] Andrew W Cross, Lev S Bishop, Sarah Sheldon, Paul D Nation, and Jay M Gambetta. Validating quantum computers using randomized model circuits. Phys. Rev. A, 100 (3): 032328, September 2019. ISSN 1050-2947. 10.1103/​PhysRevA.100.032328.

[2] Rami Barends, Julian Kelly, et al. Superconducting quantum circuits at the surface code threshold for fault tolerance. Nature, 508 (7497): 500, 2014. 10.1038/​nature13171.

[3] Simon J Devitt. Performing quantum computing experiments in the cloud. Phys. Rev. A, 94 (3): 032329, September 2016. ISSN 1050-2947. 10.1103/​PhysRevA.94.032329.

[4] S Debnath, N M Linke, C Figgatt, K A Landsman, K Wright, and C Monroe. Demonstration of a small programmable quantum computer with atomic qubits. Nature, 536 (7614): 63–66, August 2016. ISSN 0028-0836, 1476-4687. 10.1038/​nature18648.

[5] Norbert M Linke, Dmitri Maslov, Martin Roetteler, Shantanu Debnath, Caroline Figgatt, Kevin A Landsman, Kenneth Wright, and Christopher Monroe. Experimental comparison of two quantum computing architectures. Proceedings of the National Academy of Sciences, 114 (13): 3305–3310, 2017. 10.1073/​pnas.1618020114.

[6] James R Wootton. Benchmarking of quantum processors with random circuits. arXiv:1806.02736, 2018.

[7] Julian Kelly, Rami Barends, et al. State preservation by repetitive error detection in a superconducting quantum circuit. Nature, 519 (7541): 66, 2015. 10.1038/​nature14270.

[8] IonQ press release. [Online], 2018. https:/​/​ionq.co/​news/​december-11-2018 [2019-03-18].

[9] IBM Q ``Tokyo'' Specifications. [Online], 2019a. https:/​/​www.research.ibm.com/​ibm-q/​technology/​devices/​#ibmq-20-tokyo [2019-03-18].

[10] Frank Arute, Kunal Arya, et al. Quantum supremacy using a programmable superconducting processor. Nature, 574 (7779): 505–510, October 2019. ISSN 0028-0836, 1476-4687. 10.1038/​s41586-019-1666-5.

[11] IBM news room. [Online], 2019b. https:/​/​newsroom.ibm.com/​2019-09-18-IBM-Opens-Quantum-Computation-Center-in-New-York-Brings-Worlds-Largest-Fleet-of-Quantum-Computing-Systems-Online-Unveils-New-53-Qubit-Quantum-System-for-Broad-Use [2019-09-18].

[12] Fernando G S L Brandão, Aram W Harrow, and Michał Horodecki. Local random quantum circuits are approximate Polynomial-Designs. Commun. Math. Phys., 346 (2): 397–434, September 2016. ISSN 1432-0916. 10.1007/​s00220-016-2706-8.

[13] Joseph Emerson, Robert Alicki, and Karol Życzkowski. Scalable noise estimation with random unitary operators. Journal of Optics B: Quantum and Semiclassical Optics, 7 (10): S347, 2005. 10.1088/​1464-4266/​7/​10/​021.

[14] Joseph Emerson, Marcus Silva, Osama Moussa, Colm Ryan, Martin Laforest, Jonathan Baugh, David G Cory, and Raymond Laflamme. Symmetrized characterization of noisy quantum processes. Science, 317 (5846): 1893–1896, 2007. 10.1126/​science.1145699.

[15] Emanuel Knill, Dietrich Leibfried, et al. Randomized benchmarking of quantum gates. Physical Review A, 77 (1): 012307, 2008. 10.1103/​PhysRevA.77.012307.

[16] Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Scalable and robust randomized benchmarking of quantum processes. Physical Review Letters, 106 (18): 180504, 2011. 10.1103/​PhysRevLett.106.180504.

[17] Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Characterizing quantum gates via randomized benchmarking. Physical Review A, 85 (4): 042311, 2012. 10.1103/​PhysRevA.85.042311.

[18] Robin Blume-Kohout, John King Gamble, Erik Nielsen, Kenneth Rudinger, Jonathan Mizrahi, Kevin Fortier, and Peter Maunz. Demonstration of qubit operations below a rigorous fault tolerance threshold with gate set tomography. Nature Communications, 8: 14485, 2017. 10.1038/​ncomms16226.

[19] Daniel Greenbaum. Introduction to quantum gate set tomography. arXiv:1509.02921, 2015.

[20] Juan P Dehollain, Juha T Muhonen, Robin Blume-Kohout, Kenneth M Rudinger, John King Gamble, Erik Nielsen, Arne Laucht, Stephanie Simmons, Rachpon Kalra, Andrew S Dzurak, and Andrea Morello. Optimization of a solid-state electron spin qubit using gate set tomography. New J. Phys., 18 (10): 103018, October 2016. ISSN 1367-2630. 10.1088/​1367-2630/​18/​10/​103018.

[21] Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. The LINPACK benchmark: past, present and future. Concurrency and Computation: Practice and Experience, 15 (9): 803–820, 2003. 10.1002/​cpe.728.

[22] Joseph Emerson. Benchmarking into the future. IARPA LogiQ Principal Investigators Meeting, 6 September, 2017.

[23] Joseph Emerson. (private communication, 2019).

[24] Alexander Erhard, Joel J Wallman, Lukas Postler, Michael Meth, Roman Stricker, Esteban A Martinez, Philipp Schindler, Thomas Monz, Joseph Emerson, and Rainer Blatt. Characterizing large-scale quantum computers via cycle benchmarking. Nat. Commun., 10 (1): 5347, November 2019. ISSN 2041-1723. 10.1038/​s41467-019-13068-7.

[25] Lev Bishop. Is the `Quantum Volume' a fair metric for future, elaborate, high value quantum computations? Quantum Computing Stack Exchange, 2018. https:/​/​quantumcomputing.stackexchange.com/​a/​4001 [2019-03-21].

[26] Peter W Shor. Algorithms for quantum computation: Discrete logarithms and factoring. In Proceedings 35th Annual Symposium on Foundations of Computer Science, pages 124–134. IEEE, 1994. 10.1109/​SFCS.1994.365700.

[27] Lov K Grover. Quantum mechanics helps in searching for a needle in a haystack. Physical Review Letters, 79 (2): 325, 1997. 10.1103/​PhysRevLett.79.325.

[28] Richard Cleve and John Watrous. Fast parallel circuits for the quantum Fourier transform. In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 526–536. IEEE, 2000. 10.1109/​SFCS.2000.892140.

[29] Yasuhiro Sekino and Leonard Susskind. Fast scramblers. Journal of High Energy Physics, 2008 (10): 065, 2008. 10.1088/​1126-6708/​2008/​10/​065.

[30] Scott Aaronson and Lijie Chen. Complexity-Theoretic foundations of quantum supremacy experiments. December 2016. https:/​/​arxiv.org/​abs/​1612.05903.

[31] Steven J van Enk and Robin Blume-Kohout. When quantum tomography goes wrong: drift of quantum sources and other errors. New Journal of Physics, 15 (2): 025024, 2013. 10.1088/​1367-2630/​15/​2/​025024.

[32] Kenneth Rudinger, Timothy Proctor, Dylan Langharst, Mohan Sarovar, Kevin Young, and Robin Blume-Kohout. Probing Context-Dependent errors in quantum processors. Phys. Rev. X, 9 (2): 021045, June 2019. 10.1103/​PhysRevX.9.021045.

[33] Kristine Boone, Arnaud Carignan-Dugas, Joel J Wallman, and Joseph Emerson. Randomized benchmarking under different gate sets. Phys. Rev. A, 99 (3): 032329, March 2019. ISSN 1050-2947. 10.1103/​PhysRevA.99.032329.

[34] Timothy J Proctor, Arnaud Carignan-Dugas, Kenneth Rudinger, Erik Nielsen, Robin Blume-Kohout, and Kevin Young. Direct randomized benchmarking for multiqubit devices. Phys. Rev. Lett., 123 (3): 030503, July 2019. ISSN 0031-9007, 1079-7114. 10.1103/​PhysRevLett.123.030503.

[35] Jay M Gambetta, Antonio D Córcoles, et al. Characterization of addressability by simultaneous randomized benchmarking. Physical Review Letters, 109 (24): 240504, 2012. 10.1103/​PhysRevLett.109.240504.

[36] Sergio Boixo, Sergei V Isakov, Vadim N Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, Michael J Bremner, John M Martinis, and Hartmut Neven. Characterizing quantum supremacy in near-term devices. Nature Physics, 14 (6): 595, 2018. 10.1038/​s41567-018-0124-x.

[37] Charles Neill, Pedran Roushan, et al. A blueprint for demonstrating quantum supremacy with superconducting qubits. Science, 360 (6385): 195–199, 2018. 10.1126/​science.aao4309.

[38] Shelby Kimmel, Guang Hao Low, and Theodore J Yoder. Robust calibration of a universal single-qubit gate set via robust phase estimation. Physical Review A, 92 (6): 062315, 2015. 10.1103/​PhysRevA.92.062315.

[39] Richard Cleve, Artur Ekert, Chiara Macchiavello, and Michele Mosca. Quantum algorithms revisited. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 454 (1969): 339–354, 1998. 10.1098/​rspa.1998.0164.

[40] Alberto Peruzzo, Jarrod McClean, et al. A variational eigenvalue solver on a photonic quantum processor. Nature Communications, 5: 4213, 2014. 10.1038/​ncomms5213.

[41] David C McKay, Christopher J Wood, Sarah Sheldon, Jerry M Chow, and Jay M Gambetta. Efficient $Z$ gates for quantum computing. Phys. Rev. A, 96 (2): 022330, August 2017. ISSN 1050-2947. 10.1103/​PhysRevA.96.022330.

[42] M A Rol, L Ciorciaro, F K Malinowski, B M Tarasinski, R E Sagastizabal, C C Bultink, Y Salathe, N Haandbaek, J Sedivy, and L DiCarlo. Time-domain characterization and correction of on-chip distortion of control pulses in a quantum processor. Appl. Phys. Lett., 116 (5): 054001, February 2020. ISSN 0003-6951. 10.1063/​1.5133894.

[43] Adam Bouland, Bill Fefferman, Chinmay Nirkhe, and Umesh Vazirani. On the complexity and verification of quantum random circuit sampling. Nat. Phys., 15 (2): 159–163, February 2019. ISSN 1745-2473, 1745-2481. 10.1038/​s41567-018-0318-2.

[44] Robin Blume-Kohout et al. Idle Tomography. In preparation, 2020.

[45] Seth Lloyd. Universal quantum simulators. Science, 273: 1073–1078, 1996. 10.1126/​science.273.5278.1073.

Cited by

[1] Kübra Yeter‐Aydeniz, Bryan T. Gard, Jacek Jakowski, Swarnadeep Majumder, George S. Barron, George Siopsis, Travis S. Humble, and Raphael C. Pooser, "Benchmarking Quantum Chemistry Computations with Variational, Imaginary Time Evolution, and Krylov Space Solver Algorithms", Advanced Quantum Technologies 4 7, 2100012 (2021).

[2] Garrelt J. N. Alberts, M. Adriaan Rol, Thorsten Last, Benno W. Broer, Cornelis C. Bultink, Matthijs S. C. Rijlaarsdam, and Amber E. Van Hauwermeiren, "Accelerating quantum computer developments", EPJ Quantum Technology 8 1, 18 (2021).

[3] Daniel Mills, Seyon Sivarajah, Travis L. Scholten, and Ross Duncan, "Application-Motivated, Holistic Benchmarking of a Full Quantum Computing Stack", Quantum 5, 415 (2021).

[4] M. Morgado and S. Whitlock, "Quantum simulation and computing with Rydberg-interacting qubits", AVS Quantum Science 3 2, 023501 (2021).

[5] Daniel Stilck França, Sergii Strelchuk, and Michał Studziński, "Efficient Classical Simulation and Benchmarking of Quantum Processes in the Weyl Basis", Physical Review Letters 126 21, 210502 (2021).

[6] Tirthak Patel and Devesh Tiwari, Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems 443 (2021) ISBN:9781450383172.

[7] Michele Amoretti, "An Effective Framework for Full-Stack Benchmarking of Quantum Computers", Quantum Views 5, 52 (2021).

[8] Simon Martiel, Thomas Ayral, and Cyril Allouche, "Benchmarking Quantum Coprocessors in an Application-Centric, Hardware-Agnostic, and Scalable Way", IEEE Transactions on Quantum Engineering 2, 1 (2021).

[9] Seyon Sivarajah, Silas Dilkes, Alexander Cowtan, Will Simmons, Alec Edgington, and Ross Duncan, "t|ket⟩: a retargetable compiler for NISQ devices", Quantum Science and Technology 6 1, 014003 (2021).

[10] Erik Nielsen, John King Gamble, Kenneth Rudinger, Travis Scholten, Kevin Young, and Robin Blume-Kohout, "Gate Set Tomography", arXiv:2009.07301.

[11] Erik Nielsen, Kenneth Rudinger, Timothy Proctor, Antonio Russo, Kevin Young, and Robin Blume-Kohout, "Probing quantum processor performance with pyGSTi", Quantum Science and Technology 5 4, 044002 (2020).

[12] A. D. Corcoles, A. Kandala, A. Javadi-Abhari, D. T. McClure, A. W. Cross, K. Temme, P. D. Nation, M. Steffen, and J. M. Gambetta, "Challenges and Opportunities of Near-Term Quantum Computing Systems", arXiv:1910.02894.

[13] Peter J. Karalekas, Nikolas A. Tezak, Eric C. Peterson, Colm A. Ryan, Marcus P. da Silva, and Robert S. Smith, "A quantum-classical cloud platform optimized for variational hybrid algorithms", Quantum Science and Technology 5 2, 024003 (2020).

[14] Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen, and Robin Blume-Kohout, "Measuring the Capabilities of Quantum Computers", arXiv:2008.11294.

[15] Alba Cervera-Lierta, José Ignacio Latorre, and Dardo Goyeneche, "Quantum circuits for maximally entangled states", Physical Review A 100 2, 022342 (2019).

[16] Salonik Resch and Ulya R. Karpuzcu, "Benchmarking Quantum Computers and the Impact of Quantum Noise", arXiv:1912.00546.

[17] Travis L. Scholten, Yi-Kai Liu, Kevin Young, and Robin Blume-Kohout, "Classifying single-qubit noise using machine learning", arXiv:1908.11762.

[18] Erik Gustafson, Patrick Dreher, Zheyue Hang, and Yannick Meurice, "Indexed improvements for real-time trotter evolution of a (1 + 1) field theory using NISQ quantum computers", Quantum Science and Technology 6 4, 045020 (2021).

The above citations are from Crossref's cited-by service (last updated successfully 2021-09-23 13:57:09) and SAO/NASA ADS (last updated successfully 2021-09-23 13:57:10). The list may be incomplete as not all publishers provide suitable and complete citation data.