# Quantum error correction for the toric code using deep reinforcement learning

Philip Andreasson, Joel Johansson, Simon Liljestrand, and Mats Granath

Department of Physics, University of Gothenburg, SE-41296 Gothenburg, Sweden

### Abstract

We implement a quantum error correction algorithm for bit-flip errors on the topological toric code using deep reinforcement learning. An action-value Q-function encodes the discounted value of moving a defect to a neighboring site on the square grid (the action) depending on the full set of defects on the torus (the syndrome or state). The Q-function is represented by a deep convolutional neural network. Using the translational invariance on the torus allows for viewing each defect from a central perspective which significantly simplifies the state space representation independently of the number of defect pairs. The training is done using experience replay, where data from the algorithm being played out is stored and used for mini-batch upgrade of the Q-network. We find performance which is close to, and for small error rates asymptotically equivalent to, that achieved by the Minimum Weight Perfect Matching algorithm for code distances up to $d=7$. Our results show that it is possible for a self-trained agent without supervision or support algorithms to find a decoding scheme that performs on par with hand-made algorithms, opening up for future machine engineered decoders for more general error models and error correcting codes.

Quantum computers are much more susceptible to noise than present day classical computers. To construct a universal quantum computer it will be necessary to incorporate an auxiliary system for error correction, otherwise errors would quickly accumulate and ruin the calculation. Errors of the quantum bits, or qubits, are the quantum analogs of bit flip errors that also occur in a classical computer. However, in contrast to the classical bits, it is not possible to get an exact diagnosis of qubit errors without destroying the stored quantum information. Instead the error correction has to rely on partial information known as the syndrome and based on this suggest the best way to correct errors. Because of the incomplete information this is a very challenging problem requiring sophisticated algorithms known as error decoders.
In this paper we develop an error decoder based on artificial intelligence. We use deep reinforcement learning, which is the same framework that has recently achieved super-human performance in playing computer and board games. By exploration, experience is gathered and used to train an artificial neural network that can suggest the best error correction to perform for any given syndrome. Our results show that it is possible for a self-trained agent without supervision or support algorithms to find a decoding scheme that performs on par with hand-made algorithms, opening up for future machine engineered decoders for more general types of noise and error correcting codes.

### ► References

[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems 25, pages 1097–1105, 2012.

[2] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 (7553): 436, 2015. 10.1038/​nature14539.
https:/​/​doi.org/​10.1038/​nature14539

[3] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:/​/​www.deeplearningbook.org.
http:/​/​www.deeplearningbook.org

[4] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.

[5] Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38 (3): 58–68, 1995. URL https:/​/​link.galegroup.com/​apps/​doc/​A16764437/​AONE?u=googlescholar&sid=AONE&xid=f888cd62.

[6] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. URL https:/​/​arxiv.org/​abs/​1312.5602.
arXiv:1312.5602

[7] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 (7540): 529, 2015. 10.1038/​nature14236.
https:/​/​doi.org/​10.1038/​nature14236

[8] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550 (7676): 354, 2017. 10.1038/​nature24270.
https:/​/​doi.org/​10.1038/​nature24270

[9] Louis-François Arsenault, Alejandro Lopez-Bezanilla, O Anatole von Lilienfeld, and Andrew J Millis. Machine learning for many-body physics: the case of the anderson impurity model. Physical Review B, 90 (15): 155136, 2014. 10.1103/​PhysRevB.90.155136.
https:/​/​doi.org/​10.1103/​PhysRevB.90.155136

[10] Evert PL Van Nieuwenburg, Ye-Hua Liu, and Sebastian D Huber. Learning phase transitions by confusion. Nature Physics, 13 (5): 435, 2017. 10.1038/​nphys4037.
https:/​/​doi.org/​10.1038/​nphys4037

[11] Juan Carrasquilla and Roger G Melko. Machine learning phases of matter. Nature Physics, 13 (5): 431, 2017. 10.1038/​nphys4035.
https:/​/​doi.org/​10.1038/​nphys4035

[12] Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355 (6325): 602–606, 2017. 10.1126/​science.aag2302.
https:/​/​doi.org/​10.1126/​science.aag2302

[13] Xun Gao and Lu-Ming Duan. Efficient representation of quantum many-body states with deep neural networks. Nature communications, 8 (1): 662, 2017. 10.1038/​s41467-017-00705-2.
https:/​/​doi.org/​10.1038/​s41467-017-00705-2

[14] Peter W. Shor. Scheme for reducing decoherence in quantum computer memory. Phys. Rev. A, 52: R2493–R2496, Oct 1995. 10.1103/​PhysRevA.52.R2493.
https:/​/​doi.org/​10.1103/​PhysRevA.52.R2493

[15] A. M. Steane. Error correcting codes in quantum theory. Phys. Rev. Lett., 77: 793–797, Jul 1996. 10.1103/​PhysRevLett.77.793.
https:/​/​doi.org/​10.1103/​PhysRevLett.77.793

[16] Michael A Nielsen and Isaac Chuang. Quantum computation and quantum information, 2002.

[17] Barbara M Terhal. Quantum error correction for quantum memories. Reviews of Modern Physics, 87 (2): 307, 2015. 10.1103/​RevModPhys.87.307.
https:/​/​doi.org/​10.1103/​RevModPhys.87.307

[18] Alexey A Melnikov, Hendrik Poulsen Nautrup, Mario Krenn, Vedran Dunjko, Markus Tiersch, Anton Zeilinger, and Hans J Briegel. Active learning machine learns to create new quantum experiments. Proceedings of the National Academy of Sciences, 115 (6): 1221–1226, 2018. 10.1073/​pnas.1714936115.
https:/​/​doi.org/​10.1073/​pnas.1714936115

[19] Thomas Fösel, Petru Tighineanu, Talitha Weiss, and Florian Marquardt. Reinforcement learning with neural networks for quantum feedback. Phys. Rev. X, 8: 031084, Sep 2018. 10.1103/​PhysRevX.8.031084.
https:/​/​doi.org/​10.1103/​PhysRevX.8.031084

[20] Marin Bukov, Alexandre G. R. Day, Dries Sels, Phillip Weinberg, Anatoli Polkovnikov, and Pankaj Mehta. Reinforcement learning in different phases of quantum control. Phys. Rev. X, 8: 031086, Sep 2018. 10.1103/​PhysRevX.8.031086.
https:/​/​doi.org/​10.1103/​PhysRevX.8.031086

[21] Jacob Biamonte, Peter Wittek, Nicola Pancotti, Patrick Rebentrost, Nathan Wiebe, and Seth Lloyd. Quantum machine learning. Nature, 549 (7671): 195, 2017. 10.1038/​nature23474.
https:/​/​doi.org/​10.1038/​nature23474

[22] A Yu Kitaev. Fault-tolerant quantum computation by anyons. Annals of Physics, 303 (1): 2–30, 2003. 10.1016/​S0003-4916(02)00018-0.
https:/​/​doi.org/​10.1016/​S0003-4916(02)00018-0

[23] Eric Dennis, Alexei Kitaev, Andrew Landahl, and John Preskill. Topological quantum memory. Journal of Mathematical Physics, 43 (9): 4452–4505, 2002. 10.1063/​1.1499754.
https:/​/​doi.org/​10.1063/​1.1499754

[24] Robert Raussendorf, Jim Harrington, and Kovid Goyal. Topological fault-tolerance in cluster state quantum computation. New Journal of Physics, 9 (6): 199, 2007. 10.1088/​1367-2630/​9/​6/​199.
https:/​/​doi.org/​10.1088/​1367-2630/​9/​6/​199

[25] Austin G Fowler, Matteo Mariantoni, John M Martinis, and Andrew N Cleland. Surface codes: Towards practical large-scale quantum computation. Physical Review A, 86 (3): 032324, 2012. 10.1103/​PhysRevA.86.032324.
https:/​/​doi.org/​10.1103/​PhysRevA.86.032324

[26] Julian Kelly, Rami Barends, Austin G Fowler, Anthony Megrant, Evan Jeffrey, Theodore C White, Daniel Sank, Josh Y Mutus, Brooks Campbell, Yu Chen, et al. State preservation by repetitive error detection in a superconducting quantum circuit. Nature, 519 (7541): 66, 2015. 10.1038/​nature14270.
https:/​/​doi.org/​10.1038/​nature14270

[27] Jack Edmonds. Paths, trees, and flowers. Canadian Journal of mathematics, 17 (3): 449–467, 1965. 10.4153/​CJM-1965-045-4.
https:/​/​doi.org/​10.4153/​CJM-1965-045-4

[28] Austin G Fowler. Minimum weight perfect matching of fault-tolerant topological quantum error correction in average o(1) parallel time. Quantum Information and Computation, 15 (1&2): 0145–0158, 2015. URL http:/​/​dl.acm.org/​citation.cfm?id=2685188.2685197.
http:/​/​dl.acm.org/​citation.cfm?id=2685188.2685197

[29] Sergey Bravyi, Martin Suchara, and Alexander Vargo. Efficient algorithms for maximum likelihood decoding in the surface code. Phys. Rev. A, 90: 032326, Sep 2014. 10.1103/​PhysRevA.90.032326.
https:/​/​doi.org/​10.1103/​PhysRevA.90.032326

[30] Ryan Sweke, Markus S Kesselring, Evert PL van Nieuwenburg, and Jens Eisert. Reinforcement learning decoders for fault-tolerant quantum computation. arXiv preprint arXiv:1810.07207, 2018. URL https:/​/​arxiv.org/​abs/​1810.07207.
arXiv:1810.07207

[31] Guillaume Duclos-Cianci and David Poulin. Fast decoders for topological quantum codes. Physical review letters, 104 (5): 050504, 2010. 10.1103/​PhysRevLett.104.050504.
https:/​/​doi.org/​10.1103/​PhysRevLett.104.050504

[32] Guillaume Duclos-Cianci and David Poulin. Fault-tolerant renormalization group decoder for abelian topological codes. Quantum Info. Comput., 14 (9&10): 721–740, July 2014. ISSN 1533-7146. URL http:/​/​dl.acm.org/​citation.cfm?id=2638670.2638671.
http:/​/​dl.acm.org/​citation.cfm?id=2638670.2638671

[33] Michael Herold, Earl T Campbell, Jens Eisert, and Michael J Kastoryano. Cellular-automaton decoders for topological quantum memories. npj Quantum Information, 1: 15010, 2015. 10.1038/​npjqi.2015.10.
https:/​/​doi.org/​10.1038/​npjqi.2015.10

[34] Aleksander Kubica and John Preskill. Cellular-automaton decoders with provable thresholds for topological codes. arXiv preprint arXiv:1809.10145, 2018. URL https:/​/​arxiv.org/​abs/​1809.10145. 10.1103/​PhysRevLett.123.020501.
https:/​/​doi.org/​10.1103/​PhysRevLett.123.020501
arXiv:1809.10145

[35] Giacomo Torlai and Roger G. Melko. Neural decoder for topological codes. Phys. Rev. Lett., 119: 030501, Jul 2017. 10.1103/​PhysRevLett.119.030501.
https:/​/​doi.org/​10.1103/​PhysRevLett.119.030501

[36] Stefan Krastanov and Liang Jiang. Deep neural network probabilistic decoder for stabilizer codes. Scientific reports, 7 (1): 11003, 2017. 10.1038/​s41598-017-11266-1.
https:/​/​doi.org/​10.1038/​s41598-017-11266-1

[37] Savvas Varsamopoulos, Ben Criger, and Koen Bertels. Decoding small surface codes with feedforward neural networks. Quantum Science and Technology, 3 (1): 015004, 2017. 10.1088/​2058-9565/​aa955a.
https:/​/​doi.org/​10.1088/​2058-9565/​aa955a

[38] Paul Baireuther, Thomas E O'Brien, Brian Tarasinski, and Carlo WJ Beenakker. Machine-learning-assisted correction of correlated qubit errors in a topological code. Quantum, 2: 48, 2018. 10.22331/​q-2018-01-29-48.
https:/​/​doi.org/​10.22331/​q-2018-01-29-48

[39] Nikolas P Breuckmann and Xiaotong Ni. Scalable neural network decoders for higher dimensional quantum codes. Quantum, 2: 68, 2018. 10.22331/​q-2018-05-24-68.
https:/​/​doi.org/​10.22331/​q-2018-05-24-68

[40] Christopher Chamberland and Pooya Ronagh. Deep neural decoders for near term fault-tolerant experiments. Quantum Sci. Technol., 3: 044002, 2018. 10.1088/​2058-9565/​aad1f7.

[41] Nishad Maskara, Aleksander Kubica, and Tomas Jochym-O'Connor. Advantages of versatile neural-network decoding for topological codes. Phys. Rev. A, 99: 052351, May 2019. 10.1103/​PhysRevA.99.052351.
https:/​/​doi.org/​10.1103/​PhysRevA.99.052351

[42] Xiaotong Ni. Neural network decoders for large-distance 2d toric codes. arXiv preprint arXiv:1809.06640, 2018. URL https:/​/​arxiv.org/​abs/​1809.06640.
arXiv:1809.06640

[43] Ye-Hua Liu and David Poulin. Neural belief-propagation decoders for quantum error-correcting codes. Phys. Rev. Lett., 122: 200501, May 2019. 10.1103/​PhysRevLett.122.200501.
https:/​/​doi.org/​10.1103/​PhysRevLett.122.200501

[44] Dan Browne. Topological codes and computation a lecture course given at the university of innsbruck. 2014. URL http:/​/​bit.do/​topological.
http:/​/​bit.do/​topological

[45] David K. Tuckett, Stephen D. Bartlett, and Steven T. Flammia. Ultrahigh error threshold for surface codes with biased noise. Phys. Rev. Lett., 120: 050505, Jan 2018. 10.1103/​PhysRevLett.120.050505.
https:/​/​doi.org/​10.1103/​PhysRevLett.120.050505

[46] Vladimir Kolmogorov. Blossom v: a new implementation of a minimum cost perfect matching algorithm. Mathematical Programming Computation, 1 (1): 43–67, 2009. 10.1007/​s12532-009-0002-8.
https:/​/​doi.org/​10.1007/​s12532-009-0002-8

[47] Austin G Fowler. Optimal complexity correction of correlated errors in the surface code. arXiv preprint arXiv:1310.0863, 2013. URL https:/​/​arxiv.org/​abs/​1310.0863.
arXiv:1310.0863

[48] Mattias Eliasson, David Fitzek, and Mats Granath. In preperation, 2019.

[49] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018. URL https:/​/​arxiv.org/​abs/​1803.00933.
arXiv:1803.00933

### Cited by

[1] Agnes Valenti, Evert van Nieuwenburg, Sebastian Huber, and Eliska Greplova, "Hamiltonian learning for quantum error correction", Physical Review Research 1 3, 033092 (2019).

[2] S. Varona and M. A. Martin-Delgado, "Determination of the semion code threshold using neural decoders", Physical Review A 102 3, 032411 (2020).

[3] Xiao-Ming Zhang, Zezhu Wei, Raza Asad, Xu-Chen Yang, and Xin Wang, "When does reinforcement learning stand out in quantum control? A comparative study on state preparation", npj Quantum Information 5 1, 85 (2019).

[4] Laia Domingo Colomer, Michalis Skotiniotis, and Ramon Muñoz-Tapia, "Reinforcement learning for optimal error correction of toric codes", Physics Letters A 384 17, 126353 (2020).

[5] Chae-Yeun Park and Michael J. Kastoryano, "Geometry of learning neural quantum states", Physical Review Research 2 2, 023232 (2020).

[6] Oleksandr Balabanov and Mats Granath, "Unsupervised learning using topological data augmentation", Physical Review Research 2 1, 013354 (2020).

[7] Juan Carrasquilla, "Machine learning for quantum matter", Advances in Physics: X 5 1, 1797528 (2020).

[8] David Fitzek, Mattias Eliasson, Anton Frisk Kockum, and Mats Granath, "Deep Q-learning decoder for depolarizing noise on the toric code", Physical Review Research 2 2, 023230 (2020).

[9] Hendrik Poulsen Nautrup, Nicolas Delfosse, Vedran Dunjko, Hans J. Briegel, and Nicolai Friis, "Optimizing Quantum Error Correction Codes with Reinforcement Learning", arXiv:1812.08451, Quantum 3, 215 (2019).

[10] Tomi Ohtsuki and Tomohiro Mano, "Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions", Journal of the Physical Society of Japan 89 2, 022001 (2020).

[11] Jelena Mackeprang, Durga B. Rao Dasari, and Jörg Wrachtrup, "A reinforcement learning approach for quantum state engineering", Quantum Machine Intelligence 2 1, 5 (2020).

[12] Samuel Yen-Chi Chen, Chao-Han Huck Yang, Jun Qi, Pin-Yu Chen, Xiaoli Ma, and Hsi-Sheng Goan, "Variational Quantum Circuits for Deep Reinforcement Learning", IEEE Access 8, 141007 (2020).

[13] Hiroki Saito, "Creation and Manipulation of Quantized Vortices in Bose–Einstein Condensates Using Reinforcement Learning", Journal of the Physical Society of Japan 89 7, 074006 (2020).

[14] Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn, Lecture Notes in Computer Science 12109, 55 (2020) ISBN:978-3-030-47357-0.

[15] Jonas Schuff, Lukas J Fiderer, and Daniel Braun, "Improving the dynamics of quantum sensors with reinforcement learning", New Journal of Physics 22 3, 035001 (2020).

[16] Jun-Jie Chen and Ming Xue, "Manipulation of Spin Dynamics by Deep Reinforcement Learning Agent", arXiv:1901.08748.

[17] Chaitanya Chinni, Abhishek Kulkarni, Dheeraj M. Pai, Kaushik Mitra, and Pradeep Kiran Sarvepalli, "Neural Decoder for Topological Codes using Pseudo-Inverse of Parity Check Matrix", arXiv:1901.07535.

The above citations are from Crossref's cited-by service (last updated successfully 2020-10-28 09:25:13) and SAO/NASA ADS (last updated successfully 2020-10-28 09:25:14). The list may be incomplete as not all publishers provide suitable and complete citation data.