Re-examining the quantum volume test: Ideal distributions, compiler optimizations, confidence intervals, and scalable resource estimations

Charles H. Baldwin, Karl Mayer, Natalie C. Brown, Ciarán Ryan-Anderson, and David Hayes

Quantinuum, 303 S. Technology Ct, Broomfield, Colorado 80021, USA

Find this paper interesting or want to discuss? Scite or leave a comment on SciRate.


The quantum volume test is a full-system benchmark for quantum computers that is sensitive to qubit number, fidelity, connectivity, and other quantities believed to be important in building useful devices. The test was designed to produce a single-number measure of a quantum computer's general capability, but a complete understanding of its limitations and operational meaning is still missing. We explore the quantum volume test to better understand its design aspects, sensitivity to errors, passing criteria, and what passing implies about a quantum computer. We elucidate some transient behaviors the test exhibits for small qubit number including the ideal measurement output distributions and the efficacy of common compiler optimizations. We then present an efficient algorithm for estimating the expected heavy output probability under different error models and compiler optimization options, which predicts performance goals for future systems. Additionally, we explore the original confidence interval construction and show that it underachieves the desired coverage level for single shot experiments and overachieves for more typical number of shots. We propose a new confidence interval construction that reaches the specified coverage for typical number of shots and is more efficient in the number of circuits needed to pass the test. We demonstrate these savings with a $QV=2^{10}$ experimental dataset collected from Quantinuum System Model H1-1. Finally, we discuss what the quantum volume test implies about a quantum computer's practical or operational abilities especially in terms of quantum error correction.

Repository of code used in simulation and analysis:

The quantum volume test is a previously proposed benchmark for quantum computers that is sensitive to qubit number and system fidelity. Quantum volume is measured by running a set of complicated random circuits on a quantum computer, measuring the outputs, and then comparing those outputs to a classical simulation. The comparison method yields a single-number measure of a quantum computer’s general capability, but a complete understanding of the tests limitations and operational meaning is still missing. We explore the quantum volume test to better understand its design aspects, sensitivity to errors, passing criteria, and what passing implies about a quantum computer.

► BibTeX data

► References

[1] Lev S Bishop, Sergey Bravyi, Andrew Cross, Jay M Gambetta, and John Smolin. ``Quantum volume''. Quantum Volume. Technical Report (2017).

[2] Nikolaj Moll, Panagiotis Barkoutsos, Lev S Bishop, Jerry M Chow, Andrew Cross, Daniel J Egger, Stefan Filipp, Andreas Fuhrer, Jay M Gambetta, Marc Ganzhorn, Abhinav Kandala, Antonio Mezzacapo, Peter Müller, Walter Riess, Gian Salis, John Smolin, Ivano Tavernelli, and Kristan Temme. ``Quantum optimization using variational algorithms on near-term quantum devices''. Quantum Science and Technology 3, 030503 (2018).

[3] Andrew W. Cross, Lev S. Bishop, Sarah Sheldon, Paul D. Nation, and Jay M. Gambetta. ``Validating quantum computers using randomized model circuits''. Phys. Rev. A 100, 032328 (2019).

[4] J. M. Pino, J. M. Dreiling, C. Figgatt, J. P. Gaebler, S. A. Moses, M. S. Allman, C. H. Baldwin, M. Foss-Feig, D. Hayes, K. Mayer, C. Ryan-Anderson, and B. Neyenhuis. ``Demonstration of the trapped-ion quantum ccd computer architecture''. Nature 559, 209–213 (2021).

[5] Neereja Sundaresan, Isaac Lauer, Emily Pritchett, Easwar Magesan, Petar Jurcevic, and Jay M. Gambetta. ``Reducing unitary and spectator errors in cross resonance with optimized rotary echoes''. PRX Quantum 1, 020318 (2020).

[6] Petar Jurcevic, Ali Javadi-Abhari, Lev S Bishop, Isaac Lauer, Daniela F Bogorin, Markus Brink, Lauren Capelluto, Oktay Günlük, Toshinari Itoko, Naoki Kanazawa, Abhinav Kandala, George A Keefe, Kevin Krsulich, William Landers, Eric P Lewandowski, Douglas T McClure, Giacomo Nannicini, Adinath Narasgond, Hasan M Nayfeh, Emily Pritchett, Mary Beth Rothwell, Srikanth Srinivasan, Neereja Sundaresan, Cindy Wang, Ken X Wei, Christopher J Wood, Jeng-Bang Yau, Eric J Zhang, Oliver E Dial, Jerry M Chow, and Jay M Gambetta. ``Demonstration of quantum volume 64 on a superconducting quantum computing system''. Quantum Science and Technology 6, 025020 (2021).

[7] ``Achieving quantum volume 128 on the Honeywell quantum computer'' (2020).

[8] ``IBM achieves a new quantum volume level of 128'' (2021).

[9] ``Honeywell sets new record for quantum computing performance'' (2021).

[10] ``Honeywell sets another record for quantum computing performance'' (2021).

[11] Isaac L. Chuang and M. A. Nielsen. ``Prescription for experimental determination of the dynamics of a quantum black box''. Journal of Modern Optics 44, 2455–2467 (1997).

[12] E. Knill, D. Leibfried, R. Reichle, J. Britton, R. B. Blakestad, J. D. Jost, C. Langer, R. Ozeri, S. Seidelin, and D. J. Wineland. ``Randomized benchmarking of quantum gates''. Phys. Rev. A 77, 012307 (2008).

[13] Easwar Magesan, Jay M. Gambetta, and Joseph Emerson. ``Characterizing quantum gates via randomized benchmarking''. Phys. Rev. A 85, 042311 (2012).

[14] Thomas Lubinski, Sonika Johri, Paul Varosy, Jeremiah Coleman, Luning Zhao, Jason Necaise, Charles H. Baldwin, Karl Mayer, and Timothy Proctor. ``Application-oriented performance benchmarks for quantum computing'' (2021). arXiv:2110.03137.

[15] Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen, and Robin Blume-Kohout. ``Measuring the capabilities of quantum computers''. Nature Physics 18, 75–79 (2021).

[16] Mohan Sarovar, Timothy Proctor, Kenneth Rudinger, Kevin Young, Erik Nielsen, and Robin Blume-Kohout. ``Detecting crosstalk errors in quantum information processors''. Quantum 4, 321 (2020).

[17] Sergio Boixo, Sergei V. Isakov, Vadim N. Smelyanskiy, Ryan Babbush, Nan Ding, Zhang Jiang, Michael J. Bremner, John M. Martinis, and Hartmut Neven. ``Characterizing quantum supremacy in near-term devices''. Nature Physics 14, 595–600 (2018).

[18] Alexander Erhard, Joel J. Wallman, Lukas Postler, Michael Meth, Roman Stricker, Esteban A. Martinez, Philipp Schindler, Thomas Monz, Joseph Emerson, and Rainer Blatt. ``Characterizing large-scale quantum computers via cycle benchmarking''. Nat. Comm. 10 (2019).

[19] Timothy J. Proctor, Arnaud Carignan-Dugas, Kenneth Rudinger, Erik Nielsen, Robin Blume-Kohout, and Kevin Young. ``Direct randomized benchmarking for multiqubit devices''. Phys. Rev. Lett. 123 (2019).

[20] Robin Harper, Steven T. Flammia, and Joel J. Wallman. ``Efficient learning of quantum noise''. Nature Physics 16, 1184–1188 (2020).

[21] K. Wright, K. M. Beck, S. Debnath, J. M. Amini, Y. Nam, N. Grzesiak, J.-S. Chen, N. C. Pisenti, M. Chmielewski, C. Collins, K. M. Hudek, J. Mizrahi, J. D. Wong-Campos, S. Allen, J. Apisdorf, P. Solomon, M. Williams, A. M. Ducore, A. Blinov, S. M. Kreikemeier, V. Chaplin, M. Keesan, C. Monroe, and J. Kim. ``Benchmarking an 11-qubit quantum computer''. Nat. Comm. 10 (2019).

[22] Arjan Cornelissen, Johannes Bausch, and András Gilyén. ``Scalable benchmarks for gate-based quantum computers'' (2021). arXiv:2104.10698.

[23] Scott Aaronson and Lijie Chen. ``Complexity-theoretic foundations of quantum supremacy experiments'' (2016). arXiv:1612.05903.

[24] Frank Arute, Kunal Arya, Ryan Babbush, Dave Bacon, Joseph C. Bardin, Rami Barends, Rupak Biswas, Sergio Boixo, Fernando G. S. L. Brandao, David A. Buell, and et al. ``Quantum supremacy using a programmable superconducting processor''. Nature 574, 505–510 (2019).

[25] R. Oliveira, O. C. O. Dahlsten, and M. B. Plenio. ``Generic entanglement can be generated efficiently''. Phys. Rev. Lett. 98 (2007).

[26] Aram W. Harrow and Richard A. Low. ``Random quantum circuits are approximate 2-designs''. Commun. Math. Phys. 291, 257–302 (2009).

[27] F. G. S. L. Brandão, A. W. Harrow, and M. Horodecki. ``Local random quantum circuits are approximate polynomial-designs''. Commun. Math. Phys. 346, 397–434 (2016).

[28] Christoph Dankert, Richard Cleve, Joseph Emerson, and Etera Livine. ``Exact and approximate unitary 2-designs and their application to fidelity estimation''. Phys. Rev. A 80, 012304 (2009).

[29] Sean Mullane. ``Sampling random quantum circuits: a pedestrian's guide'' (2020). arXiv:2007.07872.

[30] Don N. Page. ``Average entropy of a subsystem''. Phys. Rev. Lett. 71, 1291–1294 (1993).

[31] ``Nist/​sematech e-handbook of statistical methods''. Accessed: 2022-03-14.

[32] David Callan. ``A combinatorial survey of identities for the double factorial'' (2009). arXiv:0906.1317.

[33] Fred S. Roberts and Barry Tesman. ``Applied combinatorics 2nd ed.''. CRC Press. (2009).

[34] J. R. McClean, J. Romero, R. Babbush, and A. Aspuru-Guzik. ``The theory of variational hybrid quantum-classical algorithms''. New J. Phys. 18, 023023 (2016).

[35] E. Farhi and J. Goldstone. ``A quantum approximate optimization algorithm'' (2014). arXiv:1411.4028.

[36] D. Coppersmith. ``An approximate fourier transform useful in quantum factoring'' (2002). arXiv:quant-ph/​0201067.

[37] Navin Khaneja, Roger Brockett, and Steffen J. Glaser. ``Time optimal control in spin systems''. Phys. Rev. A 63, 032308 (2001).

[38] Farrokh Vatan and Colin Williams. ``Optimal quantum circuits for general two-qubit gates''. Phys. Rev. A 69, 032315 (2004).

[39] Anders Sørensen and Klaus Mølmer. ``Entanglement and quantum computation with ions in thermal motion''. Phys. Rev. A 62 (2000).

[40] H. Abraham and et al. ``Qiskit: An open-source framework for quantum computing'' (2019).

[41] Jun Zhang, Jiri Vala, Shankar Sastry and K. Birgitta Whaley. ``Geometric theory of nonlocal two-qubit operations''. Phys. Rev. A 67, 042313 (2003).

[42] C. H. Baldwin ``qvtsim GitHub repository.'' Accessed: 2022-03-22.

[43] Michael A. Nielsen and Isaac L. Chuang. ``Quantum computation and quantum information''. Cambridge University Press. (2000).

[44] Alexei Gilchrist, Nathan K. Langford, and Michael A. Nielsen. ``Distance measures to compare real and ideal quantum processes''. Phys. Rev. A 71, 062310 (2005).

[45] Arnaud Carignan-Dugas, Joel J Wallman, and Joseph Emerson. ``Bounding the average gate fidelity of composite channels using the unitarity''. New Journal of Physics 21, 053016 (2019).

[46] Daniel Greenbaum. ``Introduction to quantum gate set tomography'' (2015). arXiv:1509.02921.

[47] C. H. Baldwin, B. J. Bjork, J. P. Gaebler, D. Hayes, and D. Stack. ``Subspace benchmarking high-fidelity entangling operations with trapped ions''. Phys. Rev. Research 2, 013317 (2020).

[48] Filip B. Maciejewski, Zoltán Zimborás, and Michał Oszmaniec. ``Mitigation of readout noise in near-term quantum devices by classical post-processing based on detector tomography''. Quantum 4, 257 (2020).

[49] Wassily Hoeffding. ``On the Distribution of the Number of Successes in Independent Trials''. The Annals of Mathematical Statistics 27, 713 – 721 (1956).

[50] Adam M. Meier. ``Randomized benchmarking of clifford operators''. PhD thesis. University of Colorado. (2006).

[51] Bradley Efron and Robert J. Tibshirani. ``An introduciton to the bootstrap''. CRC Press. (1994).

[52] Robin Blume-Kohout, Marcus P. da Silva, Erik Nielsen, Timothy Proctor, Kenneth Rudinger, Mohan Sarovar, and Kevin Young. ``A taxonomy of small markovian errors'' (2021). arXiv:2103.01928.

[53] Jin-Sung Kim, Lev S. Bishop, Antonio D. Córcoles, Seth Merkel, John A. Smolin, and Sarah Sheldon. ``Hardware-efficient random circuits to classify noise in a multiqubit system''. Phys. Rev. A 104, 022609 (2021).

[54] Yunchao Liu, Matthew Otten, Roozbeh Bassirianjahromi, Liang Jiang, and Bill Fefferman. ``Benchmarking near-term quantum computers via random circuit sampling'' (2021). arXiv:2105.05232.

[55] Simon J Devitt, William J Munro, and Kae Nemoto. ``Quantum error correction for beginners''. Reports on Progress in Physics 76, 076001 (2013).

[56] A. M. Steane. ``Simple quantum error-correcting codes''. Physical Review A 54, 4741–4751 (1996).

[57] Austin G. Fowler, Matteo Mariantoni, John M. Martinis, and Andrew N. Cleland. ``Surface codes: Towards practical large-scale quantum computation''. Phys. Rev. A 86 (2012).

[58] Sergey B Bravyi and A Yu Kitaev. ``Quantum codes on a lattice with boundary'' (1998). arXiv:quant-ph/​9811052.

[59] Ciaran Ryan-Anderson ``PECOS: Performance estimator of codes on surfaces''. pecos GitHub repository. Accessed: 2021-10-01.

[60] Jack Edmonds. ``Paths, trees, and flowers''. Canadian Journal of mathematics 17, 449–467 (1965).

[61] Austin G. Fowler. ``Minimum weight perfect matching of fault-tolerant topological quantum error correction in average o(1) parallel time''. Quantum Info. Comput. 15, 145–158 (2015).

[62] Zunaira Babar, Panagiotis Botsinis, Dimitrios Alanis, Soon Xin Ng, and Lajos Hanzo. ``Fifteen years of quantum ldpc coding and improved decoding strategies''. IEEE Access 3, 2492–2519 (2015).

[63] Michael J. Gullans, Stefan Krastanov, David A. Huse, Liang Jiang, and Steven T. Flammia. ``Quantum coding with low-depth random circuits''. Phys. Rev. X 11, 031066 (2021).

[64] C. Ryan-Anderson, J. G. Bohnet, K. Lee, D. Gresh, A. Hankin, J. P. Gaebler, D. Francois, A. Chernoguzov, D. Lucchetti, N. C. Brown, T. M. Gatterman, S. K. Halit, K. Gilmore, J. A. Gerber, B. Neyenhuis, D. Hayes, and R. P. Stutz. ``Realization of real-time fault-tolerant quantum error correction''. Phys. Rev. X 11, 041058 (2021).

[65] Martin Suchara, Andrew W. Cross, and Jay M. Gambetta. ``Leakage suppression in the toric code''. Quantum Info. Comput. 15, 997–1016 (2015).

[66] Natalie C. Brown and Kenneth R. Brown. ``Comparing zeeman qubits to hyperfine qubits in the context of the surface code: $^{174}\mathrm{Yb}^{+}$ and $^{171}\mathrm{Yb}^{+}$''. Phys. Rev. A 97, 052301 (2018).

[67] Joseph K Iverson and John Preskill. ``Coherence in logical quantum channels''. New Journal of Physics 22, 073066 (2020).

[68] Daniel Gottesman. ``Quantum fault tolerance in small experiments'' (2016). arXiv:1610.03507.

[69] Natalie C. Brown, Andrew Cross, and Kenneth R. Brown. ``Critical faults of leakage errors on the surface code''. In 2020 IEEE International Conference on Quantum Computing and Engineering (QCE). Pages 286–294. (2020).

[70] Natalie C Brown, Michael Newman, and Kenneth R Brown. ``Handling leakage with subsystem codes''. New Journal of Physics 21, 073055 (2019).

[71] Bichen Zhang, Swarnadeep Majumder, Pak Hong Leung, Stephen Crain, Ye Wang, Chao Fang, Dripto M. Debroy, Jungsang Kim, and Kenneth R. Brown. ``Hidden inverses: Coherent error cancellation at the circuit level''. Phys. Rev. Applied 17, 034074 (2022).

[72] Pedro Parrado-Rodríguez, Ciarán Ryan-Anderson, Alejandro Bermudez, and Markus Müller. ``Crosstalk suppression for fault-tolerant quantum error correction with trapped ions''. Quantum 5, 487 (2021).

[73] David K. Tuckett, Stephen D. Bartlett, and Steven T. Flammia. ``Ultrahigh error threshold for surface codes with biased noise''. Phys. Rev. Lett. 120, 050505 (2018).

[74] Joel J. Wallman and Joseph Emerson. ``Noise tailoring for scalable quantum computation via randomized compiling''. Phys. Rev. A 94 (2016).

[75] Dripto M. Debroy, Muyuan Li, Michael Newman, and Kenneth R. Brown. ``Stabilizer slicing: Coherent error cancellations in low-density parity-check stabilizer codes''. Phys. Rev. Lett. 121, 250502 (2018).

[76] D. Hayes, D. Stack, B. Bjork, A. C. Potter, C. H. Baldwin, and R. P. Stutz. ``Eliminating leakage errors in hyperfine qubits''. Phys. Rev. Lett. 124, 170501 (2020).

Cited by

[1] Adrian Ortega, Orsolya Kálmán, and Tamás Kiss, "Testing quantum computers with the protocol of quantum state matching", Physica Scripta 98 2, 024006 (2023).

[2] Benedikt Fauseweh, "Quantum many-body simulations on digital quantum computers: State-of-the-art and future challenges", Nature Communications 15 1, 2123 (2024).

[3] Chris N. Self, Marcello Benedetti, and David Amaro, "Protecting expressive circuits with a quantum error detection code", Nature Physics (2024).

[4] Sascha Heußen, Lukas Postler, Manuel Rispler, Ivan Pogorelov, Christian D. Marciniak, Thomas Monz, Philipp Schindler, and Markus Müller, "Strategies for a practical advantage of fault-tolerant circuit design in noisy trapped-ion quantum computers", Physical Review A 107 4, 042422 (2023).

[5] S. A. Moses, C. H. Baldwin, M. S. Allman, R. Ancona, L. Ascarrunz, C. Barnes, J. Bartolotta, B. Bjork, P. Blanchard, M. Bohn, J. G. Bohnet, N. C. Brown, N. Q. Burdick, W. C. Burton, S. L. Campbell, J. P. Campora, C. Carron, J. Chambers, J. W. Chan, Y. H. Chen, A. Chernoguzov, E. Chertkov, J. Colina, J. P. Curtis, R. Daniel, M. DeCross, D. Deen, C. Delaney, J. M. Dreiling, C. T. Ertsgaard, J. Esposito, B. Estey, M. Fabrikant, C. Figgatt, C. Foltz, M. Foss-Feig, D. Francois, J. P. Gaebler, T. M. Gatterman, C. N. Gilbreth, J. Giles, E. Glynn, A. Hall, A. M. Hankin, A. Hansen, D. Hayes, B. Higashi, I. M. Hoffman, B. Horning, J. J. Hout, R. Jacobs, J. Johansen, L. Jones, J. Karcz, T. Klein, P. Lauria, P. Lee, D. Liefer, S. T. Lu, D. Lucchetti, C. Lytle, A. Malm, M. Matheny, B. Mathewson, K. Mayer, D. B. Miller, M. Mills, B. Neyenhuis, L. Nugent, S. Olson, J. Parks, G. N. Price, Z. Price, M. Pugh, A. Ransford, A. P. Reed, C. Roman, M. Rowe, C. Ryan-Anderson, S. Sanders, J. Sedlacek, P. Shevchuk, P. Siegfried, T. Skripka, B. Spaun, R. T. Sprenkle, R. P. Stutz, M. Swallows, R. I. Tobey, A. Tran, T. Tran, E. Vogt, C. Volin, J. Walker, A. M. Zolot, and J. M. Pino, "A Race-Track Trapped-Ion Quantum Processor", Physical Review X 13 4, 041052 (2023).

[6] Anita Gehlot and Ankita Joshi, 2023 IEEE International Conference on Integrated Circuits and Communication Systems (ICICACS) 1 (2023) ISBN:979-8-3503-9846-5.

[7] Yuxuan Zhang, Daoheng Niu, Alireza Shabani, and Hassan Shapourian, CLEO 2023 FM3A.2 (2023) ISBN:978-1-957171-25-8.

[8] Elijah Pelofske, "Analysis of a Programmable Quantum Annealer as a Random Number Generator", IEEE Transactions on Information Forensics and Security 19, 3636 (2024).

[9] Yuxuan Zhang, Daoheng Niu, Alireza Shabani, and Hassan Shapourian, "Quantum Volume for Photonic Quantum Processors", Physical Review Letters 130 11, 110602 (2023).

[10] Edwin Tham, Ilia Khait, and Aharon Brodutch, 2022 IEEE International Conference on Quantum Computing and Engineering (QCE) 476 (2022) ISBN:978-1-6654-9113-6.

[11] Elijah Pelofske, Andreas Bärtschi, and Stephan Eidenbenz, "Quantum Volume in Practice: What Users Can Expect From NISQ Devices", IEEE Transactions on Quantum Engineering 3, 1 (2022).

[12] Anita Gehlot and Ankita Joshi, 2023 IEEE International Conference on Integrated Circuits and Communication Systems (ICICACS) 1 (2023) ISBN:979-8-3503-9846-5.

[13] LeeAnn M. Sager-Smith, Scott E. Smart, and David A. Mazziotti, "Qubit Condensation for Assessing Efficacy of Molecular Simulation on Quantum Computers", The Journal of Physical Chemistry A 127 29, 6032 (2023).

[14] Mirko Amico, Helena Zhang, Petar Jurcevic, Lev S. Bishop, Paul Nation, Andrew Wack, and David C. McKay, 2023 IEEE International Conference on Quantum Computing and Engineering (QCE) 692 (2023) ISBN:979-8-3503-4323-6.

[15] Travis L. Scholten, Carl J. Williams, Dustin Moody, Michele Mosca, William Hurley, William J. Zeng, Matthias Troyer, and Jay M. Gambetta, "Assessing the Benefits and Risks of Quantum Computers", arXiv:2401.16317, (2024).

[16] Ryan LaRose, Andrea Mari, Vincent Russo, Dan Strano, and William J. Zeng, "Error mitigation increases the effective quantum volume of quantum computers", arXiv:2203.05489, (2022).

[17] Elijah Pelofske, Andreas Bärtschi, and Stephan Eidenbenz, "Quantum Volume in Practice: What Users Can Expect from NISQ Devices", arXiv:2203.03816, (2022).

[18] Eric C. Peterson, Lev S. Bishop, and Ali Javadi-Abhari, "Optimal synthesis into fixed XX interactions", Quantum 6, 696 (2022).

[19] Steven Herbert, Ifan Williams, Roland Guichard, and Darren Ng, "Noise-Aware Quantum Amplitude Estimation", arXiv:2109.04840, (2021).

[20] Elijah Pelofske, "Analysis of a Programmable Quantum Annealer as a Random Number Generator", arXiv:2307.02573, (2023).

[21] Aliza U. Siddiqui, Kaitlin Gili, and Chris Ballance, "Stressing Out Modern Quantum Hardware: Performance Evaluation and Execution Insights", arXiv:2401.13793, (2024).

The above citations are from Crossref's cited-by service (last updated successfully 2024-05-25 02:42:12) and SAO/NASA ADS (last updated successfully 2024-05-25 02:42:13). The list may be incomplete as not all publishers provide suitable and complete citation data.