A volumetric framework for quantum computer benchmarks

Robin Blume-Kohout and Kevin C. Young

Quantum Performance Laboratory, Sandia National Laboratories, Albuquerque, NM 87185 and Livermore, California 94550

Find this paper interesting or want to discuss? Scite or leave a comment on SciRate.


We propose a very large family of benchmarks for probing the performance of quantum computers. We call them $\textit{volumetric benchmarks}$ (VBs) because they generalize IBM's benchmark for measuring quantum volume [1]. The quantum volume benchmark defines a family of $\textit{square}$ circuits whose depth $d$ and width $w$ are the same. A volumetric benchmark defines a family of $\textit{rectangular}$ quantum circuits, for which $d$ and $w$ are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — $(w,d)$ pairs — to test suites $\mathcal{C}$$\textit(w,d)$. A test suite is an ensemble of test circuits that share a common structure. The test suite $\mathcal{C}$ for a given circuit shape may be a single circuit $C$, a specific list of circuits $\{C_1\ldots C_N\}$ that must all be run, or a large set of possible circuits equipped with a distribution $Pr(C)$. The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the $d$ vs $w$ trade-off for the processor being benchmarked.

► BibTeX data

► References

[1] Andrew W Cross, Lev S Bishop, Sarah Sheldon, Paul D Nation, and Jay M Gambetta. Validating quantum computers using randomized model circuits. Phys. Rev. A, 100 (3): 032328, September 2019. ISSN 1050-2947. 10.1103/​PhysRevA.100.032328.

[2] Rami Barends, Julian Kelly, et al. Superconducting quantum circuits at the surface code threshold for fault tolerance. Nature, 508 (7497): 500, 2014. 10.1038/​nature13171.

[3] Simon J Devitt. Performing quantum computing experiments in the cloud. Phys. Rev. A, 94 (3): 032329, September 2016. ISSN 1050-2947. 10.1103/​PhysRevA.94.032329.

[4] S Debnath, N M Linke, C Figgatt, K A Landsman, K Wright, and C Monroe. Demonstration of a small programmable quantum computer with atomic qubits. Nature, 536 (7614): 63–66, August 2016. ISSN 0028-0836, 1476-4687. 10.1038/​nature18648.

[5] Norbert M Linke, Dmitri Maslov, Martin Roetteler, Shantanu Debnath, Caroline Figgatt, Kevin A Landsman, Kenneth Wright, and Christopher Monroe. Experimental comparison of two quantum computing architectures. Proceedings of the National Academy of Sciences, 114 (13): 3305–3310, 2017. 10.1073/​pnas.1618020114.

[6] James R Wootton. Benchmarking of quantum processors with random circuits. arXiv:1806.02736, 2018.

[7] Julian Kelly, Rami Barends, et al. State preservation by repetitive error detection in a superconducting quantum circuit. Nature, 519 (7541): 66, 2015. 10.1038/​nature14270.

[8] IonQ press release. [Online], 2018. https:/​/​ionq.co/​news/​december-11-2018 [2019-03-18].

[9] IBM Q ``Tokyo'' Specifications. [Online], 2019a. https:/​/​www.research.ibm.com/​ibm-q/​technology/​devices/​#ibmq-20-tokyo [2019-03-18].

[10] Frank Arute, Kunal Arya, et al. Quantum supremacy using a programmable superconducting processor. Nature, 574 (7779): 505–510, October 2019. ISSN 0028-0836, 1476-4687. 10.1038/​s41586-019-1666-5.

[11] IBM news room. [Online], 2019b. https:/​/​newsroom.ibm.com/​2019-09-18-IBM-Opens-Quantum-Computation-Center-in-New-York-Brings-Worlds-Largest-Fleet-of-Quantum-Computing-Systems-Online-Unveils-New-53-Qubit-Quantum-System-for-Broad-Use [2019-09-18].

[12] Fernando G S L Brandão, Aram W Harrow, and Michał Horodecki. Local random quantum circuits are approximate Polynomial-Designs. Commun. Math. Phys., 346 (2): 397–434, September 2016. ISSN 1432-0916. 10.1007/​s00220-016-2706-8.

[13] Joseph Emerson, Robert Alicki, and Karol Życzkowski. Scalable noise estimation with random unitary operators. Journal of Optics B: Quantum and Semiclassical Optics, 7 (10): S347, 2005. 10.1088/​1464-4266/​7/​10/​021.

[14] Joseph Emerson, Marcus Silva, Osama Moussa, Colm Ryan, Martin Laforest, Jonathan Baugh, David G Cory, and Raymond Laflamme. Symmetrized characterization of noisy quantum processes. Science, 317 (5846): 1893–1896, 2007. 10.1126/​science.1145699.

[15] Emanuel Knill, Dietrich Leibfried, et al. Randomized benchmarking of quantum gates. Physical Review A, 77 (1): 012307, 2008. 10.1103/​PhysRevA.77.012307.

[16] Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Scalable and robust randomized benchmarking of quantum processes. Physical Review Letters, 106 (18): 180504, 2011. 10.1103/​PhysRevLett.106.180504.

[17] Easwar Magesan, Jay M Gambetta, and Joseph Emerson. Characterizing quantum gates via randomized benchmarking. Physical Review A, 85 (4): 042311, 2012. 10.1103/​PhysRevA.85.042311.

[18] Robin Blume-Kohout, John King Gamble, Erik Nielsen, Kenneth Rudinger, Jonathan Mizrahi, Kevin Fortier, and Peter Maunz. Demonstration of qubit operations below a rigorous fault tolerance threshold with gate set tomography. Nature Communications, 8: 14485, 2017. 10.1038/​ncomms16226.

[19] Daniel Greenbaum. Introduction to quantum gate set tomography. arXiv:1509.02921, 2015.

[20] Juan P Dehollain, Juha T Muhonen, Robin Blume-Kohout, Kenneth M Rudinger, John King Gamble, Erik Nielsen, Arne Laucht, Stephanie Simmons, Rachpon Kalra, Andrew S Dzurak, and Andrea Morello. Optimization of a solid-state electron spin qubit using gate set tomography. New J. Phys., 18 (10): 103018, October 2016. ISSN 1367-2630. 10.1088/​1367-2630/​18/​10/​103018.

[21] Jack J Dongarra, Piotr Luszczek, and Antoine Petitet. The LINPACK benchmark: past, present and future. Concurrency and Computation: Practice and Experience, 15 (9): 803–820, 2003. 10.1002/​cpe.728.

[22] Joseph Emerson. Benchmarking into the future. IARPA LogiQ Principal Investigators Meeting, 6 September, 2017.

[23] Joseph Emerson. (private communication, 2019).

[24] Alexander Erhard, Joel J Wallman, Lukas Postler, Michael Meth, Roman Stricker, Esteban A Martinez, Philipp Schindler, Thomas Monz, Joseph Emerson, and Rainer Blatt. Characterizing large-scale quantum computers via cycle benchmarking. Nat. Commun., 10 (1): 5347, November 2019. ISSN 2041-1723. 10.1038/​s41467-019-13068-7.

[25] Lev Bishop. Is the `Quantum Volume' a fair metric for future, elaborate, high value quantum computations? Quantum Computing Stack Exchange, 2018. https:/​/​quantumcomputing.stackexchange.com/​a/​4001 [2019-03-21].

[26] Peter W Shor. Algorithms for quantum computation: Discrete logarithms and factoring. In Proceedings 35th Annual Symposium on Foundations of Computer Science, pages 124–134. IEEE, 1994. 10.1109/​SFCS.1994.365700.

[27] Lov K Grover. Quantum mechanics helps in searching for a needle in a haystack. Physical Review Letters, 79 (2): 325, 1997. 10.1103/​PhysRevLett.79.325.

[28] Richard Cleve and John Watrous. Fast parallel circuits for the quantum Fourier transform. In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 526–536. IEEE, 2000. 10.1109/​SFCS.2000.892140.

[29] Yasuhiro Sekino and Leonard Susskind. Fast scramblers. Journal of High Energy Physics, 2008 (10): 065, 2008. 10.1088/​1126-6708/​2008/​10/​065.

[30] Scott Aaronson and Lijie Chen. Complexity-Theoretic foundations of quantum supremacy experiments. December 2016. https:/​/​arxiv.org/​abs/​1612.05903.

[31] Steven J van Enk and Robin Blume-Kohout. When quantum tomography goes wrong: drift of quantum sources and other errors. New Journal of Physics, 15 (2): 025024, 2013. 10.1088/​1367-2630/​15/​2/​025024.

[32] Kenneth Rudinger, Timothy Proctor, Dylan Langharst, Mohan Sarovar, Kevin Young, and Robin Blume-Kohout. Probing Context-Dependent errors in quantum processors. Phys. Rev. X, 9 (2): 021045, June 2019. 10.1103/​PhysRevX.9.021045.

[33] Kristine Boone, Arnaud Carignan-Dugas, Joel J Wallman, and Joseph Emerson. Randomized benchmarking under different gate sets. Phys. Rev. A, 99 (3): 032329, March 2019. ISSN 1050-2947. 10.1103/​PhysRevA.99.032329.

[34] Timothy J Proctor, Arnaud Carignan-Dugas, Kenneth Rudinger, Erik Nielsen, Robin Blume-Kohout, and Kevin Young. Direct randomized benchmarking for multiqubit devices. Phys. Rev. Lett., 123 (3): 030503, July 2019. ISSN 0031-9007, 1079-7114. 10.1103/​PhysRevLett.123.030503.

[35] Jay M Gambetta, Antonio D Córcoles, et al. Characterization of addressability by simultaneous randomized benchmarking. Physical Review Letters, 109 (24): 240504, 2012. 10.1103/​PhysRevLett.109.240504.

[36] Sergio Boixo, Sergei V Isakov, Vadim N Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, Michael J Bremner, John M Martinis, and Hartmut Neven. Characterizing quantum supremacy in near-term devices. Nature Physics, 14 (6): 595, 2018. 10.1038/​s41567-018-0124-x.

[37] Charles Neill, Pedran Roushan, et al. A blueprint for demonstrating quantum supremacy with superconducting qubits. Science, 360 (6385): 195–199, 2018. 10.1126/​science.aao4309.

[38] Shelby Kimmel, Guang Hao Low, and Theodore J Yoder. Robust calibration of a universal single-qubit gate set via robust phase estimation. Physical Review A, 92 (6): 062315, 2015. 10.1103/​PhysRevA.92.062315.

[39] Richard Cleve, Artur Ekert, Chiara Macchiavello, and Michele Mosca. Quantum algorithms revisited. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 454 (1969): 339–354, 1998. 10.1098/​rspa.1998.0164.

[40] Alberto Peruzzo, Jarrod McClean, et al. A variational eigenvalue solver on a photonic quantum processor. Nature Communications, 5: 4213, 2014. 10.1038/​ncomms5213.

[41] David C McKay, Christopher J Wood, Sarah Sheldon, Jerry M Chow, and Jay M Gambetta. Efficient $Z$ gates for quantum computing. Phys. Rev. A, 96 (2): 022330, August 2017. ISSN 1050-2947. 10.1103/​PhysRevA.96.022330.

[42] M A Rol, L Ciorciaro, F K Malinowski, B M Tarasinski, R E Sagastizabal, C C Bultink, Y Salathe, N Haandbaek, J Sedivy, and L DiCarlo. Time-domain characterization and correction of on-chip distortion of control pulses in a quantum processor. Appl. Phys. Lett., 116 (5): 054001, February 2020. ISSN 0003-6951. 10.1063/​1.5133894.

[43] Adam Bouland, Bill Fefferman, Chinmay Nirkhe, and Umesh Vazirani. On the complexity and verification of quantum random circuit sampling. Nat. Phys., 15 (2): 159–163, February 2019. ISSN 1745-2473, 1745-2481. 10.1038/​s41567-018-0318-2.

[44] Robin Blume-Kohout et al. Idle Tomography. In preparation, 2020.

[45] Seth Lloyd. Universal quantum simulators. Science, 273: 1073–1078, 1996. 10.1126/​science.273.5278.1073.

Cited by

[1] Alba Cervera-Lierta, José Ignacio Latorre, and Dardo Goyeneche, "Quantum circuits for maximally entangled states", Physical Review A 100 2, 022342 (2019).

[2] A. D. Corcoles, A. Kandala, A. Javadi-Abhari, D. T. McClure, A. W. Cross, K. Temme, P. D. Nation, M. Steffen, and J. M. Gambetta, "Challenges and Opportunities of Near-Term Quantum Computing Systems", arXiv:1910.02894.

[3] Seyon Sivarajah, Silas Dilkes, Alexander Cowtan, Will Simmons, Alec Edgington, and Ross Duncan, "t|ket⟩: a retargetable compiler for NISQ devices", Quantum Science and Technology 6 1, 014003 (2021).

[4] Peter J. Karalekas, Nikolas A. Tezak, Eric C. Peterson, Colm A. Ryan, Marcus P. da Silva, and Robert S. Smith, "A quantum-classical cloud platform optimized for variational hybrid algorithms", Quantum Science and Technology 5 2, 024003 (2020).

[5] Erik Nielsen, Kenneth Rudinger, Timothy Proctor, Antonio Russo, Kevin Young, and Robin Blume-Kohout, "Probing quantum processor performance with pyGSTi", Quantum Science and Technology 5 4, 044002 (2020).

[6] Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen, and Robin Blume-Kohout, "Measuring the Capabilities of Quantum Computers", arXiv:2008.11294.

[7] Erik Nielsen, John King Gamble, Kenneth Rudinger, Travis Scholten, Kevin Young, and Robin Blume-Kohout, "Gate Set Tomography", arXiv:2009.07301.

[8] Travis L. Scholten, Yi-Kai Liu, Kevin Young, and Robin Blume-Kohout, "Classifying single-qubit noise using machine learning", arXiv:1908.11762.

[9] Salonik Resch and Ulya R. Karpuzcu, "Benchmarking Quantum Computers and the Impact of Quantum Noise", arXiv:1912.00546.

[10] Daniel Stilck França, Sergii Strelchuk, and Michał Studziński, "Efficient benchmarking and classical simulation of quantum processes in the Weyl basis", arXiv:2008.12250.

[11] Daniel Mills, Seyon Sivarajah, Travis L. Scholten, and Ross Duncan, "Application-Motivated, Holistic Benchmarking of a Full Quantum Computing Stack", arXiv:2006.01273.

The above citations are from SAO/NASA ADS (last updated successfully 2021-03-06 13:37:21). The list may be incomplete as not all publishers provide suitable and complete citation data.

On Crossref's cited-by service no data on citing works was found (last attempt 2021-03-06 13:37:19).