References

  1. Agnihotri, A. & Batra, N. (2020). Exploring Bayesian Optimization. Distill, 5(5), e26.
  2. Ainsworth, S. K., Hayase, J. & Srinivasa, S. (2023). Git Re-Basin: Merging Models modulo Permutation Symmetries. International Conference on Learning Representations (ICLR).
  3. Ainsworth, S. K., Hayase, J. & Srinivasa, S. (n.d.). Git Re-Basin: Official implementation. GitHub [software repository]. Accessed 20 November 2026.
  4. Amsel, N., Persson, D., Musco, C. & Gower, R. M. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm. International Conference on Learning Representations (ICLR).
  5. Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research, 13, 281–305.
  6. Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. Proceedings of COMPSTAT'2010, 177–186.
  7. c-lasso contributors (n.d.). c-lasso: Constrained sparse regression. bio-datascience [software repository]. Accessed 20 November 2026.
  8. Cevher, V., Becker, S. & Schmidt, M. (2014). Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics. IEEE Signal Processing Magazine, 31(5), 32–43.
  9. Combettes, P. L. & Pesquet, J. C. (2011). Proximal Splitting Methods in Signal Processing. In Fixed-Point Algorithms for Inverse Problems in Science and Engineering. Springer, 185–212.
  10. Combettes, P. L. & Müller, C. L. (2020). Perspective maximum likelihood-type estimation via proximal decomposition. Electronic Journal of Statistics, 14(1), 207–238.
  11. CVXPYlayers contributors (n.d.). CVXPYlayers: Differentiable convex optimization layers. CVXPY [software repository]. Accessed 20 November 2026.
  12. Davidon, W. C. (1959). Variable Metric Method for Minimization. Argonne National Laboratory Report ANL-5990.
  13. Davis, D. (n.d.). Teaching and research resources. Damek Davis, personal academic website. Accessed 20 November 2026.
  14. Dawkins, R. (1996). Climbing Mount Improbable. W. W. Norton & Company.
  15. Dial, L. (2026). Architecture Decisions for Marin 535B-A23B. Open Athena Blog, 9 November 2026.
  16. Draxler, F., Veschgini, K., Salmhofer, M. & Hamprecht, F. A. (2018). Essentially No Barriers in Neural Network Energy Landscape. Proceedings of Machine Learning Research, 80, 1309–1318.
  17. Duchi, J., Shalev-Shwartz, S., Singer, Y. & Chandra, T. (2008). Efficient projections onto the ℓ₁-ball for learning in high dimensions. Proceedings of the 25th international conference on Machine learning - ICML '08, 272–279.
  18. Duchi, J., Hazan, E. & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12, 2121–2159.
  19. Edureka (n.d.). What is Deep Learning?. Edureka blog. Accessed 20 November 2026.
  20. Eyring, H. & Polanyi, M. (1931). Über einfache Gasreaktionen. Zeitschrift für Physikalische Chemie, Abteilung B, 12, 279–311.
  21. Fort, S. & Jastrzębski, S. (2019). Large Scale Structure of Neural Network Loss Landscapes. Advances in Neural Information Processing Systems, 32.
  22. Fort, S. (2022). Post on Git Re-Basin, replication, and loss-landscape claims. X, 15 September 2022 [social media post].
  23. Frank, M. & Wolfe, P. (1956). An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(1-2), 95–110.
  24. Frauenfelder, H., Sligar, S. G. & Wolynes, P. G. (1991). The Energy Landscapes and Motions of Proteins. Science, 254(5038), 1598–1603.
  25. Grant, M. & Boyd, S. (n.d.). CVX: Matlab Software for Disciplined Convex Programming. CVX Research [software and user guide]. Accessed 20 November 2026.
  26. Gurobi Optimization, LLC (n.d.). Gurobi Optimizer. Gurobi [optimization software]. Accessed 20 November 2026.
  27. Hall, D. (2026). Initial Pre-Training Checkpoints for Marin “Mochi” 535B-A23B. Open Athena Blog, 17 November 2026.
  28. Hansen, N. (2016). The CMA Evolution Strategy: A Tutorial. arXiv:1604.00772.
  29. Hazan, E. & Kakade, S. (2019). Revisiting the Polyak step size. arXiv:1905.00313.
  30. He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
  31. Hutson, M. (2019). AI protein-folding algorithms solve structures faster than ever. Nature.
  32. INFORMS Computing Society (n.d.). The Nature of Mathematical Programming. INFORMS Computing Society glossary [archived 5 March 2014]. Accessed 20 November 2026.
  33. Jaggi, M. (2013). Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Proceedings of Machine Learning Research, 28(1), 427–435.
  34. Jaggi, M. (n.d.). Optimization for Machine Learning, Lecture 2. EPFL Machine Learning and Optimization Laboratory [lecture slides]. Accessed 20 November 2026.
  35. JAX authors (n.d.). JAX documentation. JAX [software documentation]. Accessed 20 November 2026.
  36. Johnson, S. G. (n.d.). The NLopt nonlinear-optimization package. NLopt [software documentation]. Accessed 20 November 2026.
  37. Jones, D. R., Perttunen, C. D. & Stuckman, B. E. (1993). Lipschitzian optimization without the Lipschitz constant. Journal of Optimization Theory and Applications, 79(1), 157–181.
  38. Jones, D. R., Schonlau, M. & Welch, W. J. (1998). Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization, 13(4), 455–492.
  39. Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L. & Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks. Keller Jordan blog.
  40. Jordan, K., and Muon contributors (n.d.). Muon: PyTorch implementation. GitHub [software; commit f98f1cacc0263b04290753e32be8d498c1efc806]. Accessed 20 November 2026.
  41. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., et al. (2021). Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning, 14(1-2), 1–210.
  42. Karmarkar, N. (1984). A new polynomial-time algorithm for linear programming. Combinatorica, 4(4), 373–395.
  43. Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (ICLR).
  44. Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. (1983). Optimization by Simulated Annealing. Science, 220(4598), 671–680.
  45. Kjellström, G. (1969). Network Optimization by Random Variation of Component Values. Ericsson Technics, 25(3), 133–151.
  46. Kjellström, G. & Taxén, L. (1981). Stochastic optimization in system design. IEEE Transactions on Circuits and Systems, 28(7), 702–715.
  47. Kjellström, G. (1991). On the efficiency of Gaussian adaptation. Journal of Optimization Theory and Applications, 71(3), 589–597.
  48. Kjellström, G. & Taxén, L. (1992). Gaussian Adaptation, an evolution-based efficient global optimizer. Computational and Applied Mathematics. Elsevier Science, 267–276.
  49. Kolb, C., Müller, C. L., Bischl, B. & Rügamer, D. (2026). Smoothing the Edges: Smooth Optimization for Sparse Regularization Using Hadamard Overparametrization. Machine Learning, 115(4), Article 87.
  50. LearnThermo (n.d.). Thermodynamics tutorial, Chapter 3, Lesson A, page 1. LearnThermo [archived 2 March 2024]. Accessed 20 November 2026.
  51. Li, H., Xu, Z., Taylor, G., Studer, C. & Goldstein, T. (2018). Visualizing the Loss Landscape of Neural Nets. Advances in Neural Information Processing Systems, 31.
  52. Liu, D. C. & Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3), 503–528.
  53. Loshchilov, I. & Hutter, F. (2016). CMA-ES for Hyperparameter Optimization of Deep Neural Networks. ICLR Workshop.
  54. Loshchilov, I. & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR).
  55. Maheswaranathan, N., Metz, L., Tucker, G., Choi, D. & Sohl-Dickstein, J. (2019). Guided evolutionary strategies: augmenting random search with surrogate gradients. Proceedings of Machine Learning Research, 97, 4264–4273.
  56. Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C. & Rudolph, G. (2011). Exploratory landscape analysis. Proceedings of the 13th annual conference on Genetic and evolutionary computation, 829–836.
  57. Mockus, J. (1989). Bayesian Approach to Global Optimization: Theory and Applications. Kluwer Academic Publishers.
  58. MOSEK ApS (n.d.). MOSEK Optimization Suite. MOSEK [optimization software]. Accessed 20 November 2026.
  59. Müller, C. L. (2010). Black-box Landscapes: Characterization, Optimization, Sampling, and Application to Geometric Configuration Problems. PhD thesis, ETH Zürich, Dissertation No. 19438.
  60. Müller, C. L. & Sbalzarini, I. F. (2010). Gaussian Adaptation Revisited—An Entropic View on Covariance Matrix Adaptation. Applications of Evolutionary Computation, LNCS, 6024, 432–441.
  61. Müller, C. L. (2014). Stochastic Methods for Single Objective Global Optimization. Computational Intelligence in Aerospace Sciences, 63–112.
  62. Neri, F. & Cotta, C. (2012). Memetic algorithms and memetic computing optimization: A literature review. Swarm and Evolutionary Computation, 2, 1–14.
  63. Parikh, N. & Boyd, S. (2014). Proximal Algorithms. Foundations and Trends in Optimization, 1(3), 127–239.
  64. PETSc development team (n.d.). SNES: Nonlinear Solvers. PETSc Manual [software documentation]. Accessed 20 November 2026.
  65. PyTorch contributors (n.d.). torch.optim. PyTorch documentation. Accessed 20 November 2026.
  66. Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B. & Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. International Conference on Machine Learning (ICML).
  67. Rastrigin, L. A. (1963). The convergence of the random search method in the extremal control of a many-parameter system. Automation and Remote Control, 24, 1337–1342.
  68. Robbins, H. & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407.
  69. Rockafellar, R. T. (1993). Lagrange Multipliers and Optimality. SIAM Review, 35(2), 183–238.
  70. Romero, P. A. & Arnold, F. H. (2009). Exploring protein fitness landscapes by directed evolution. Nature Reviews Molecular Cell Biology, 10(12), 866–876.
  71. Romero, P. A., Krause, A. & Arnold, F. H. (2013). Navigating the protein fitness landscape with Gaussian processes. Proceedings of the National Academy of Sciences, 110(3), E193–E201.
  72. Schaipp, F. (2023). Decay No More. ICLR Blogposts 2023.
  73. Schaipp, F., Hägele, A., Taylor, A., Simsekli, U. & Bach, F. (2025). The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training. arXiv:2501.18965.
  74. Schmidt, R. M., Schneider, F. & Hennig, P. (2021). Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers. International Conference on Learning Representations (ICLR).
  75. Singhee, A. & Rutenbar, R. A. (2009). Novel Algorithms for Fast Statistical Analysis of Scaled Circuits. Lecture Notes in Electrical Engineering, vol. 46. Springer.
  76. Snoek, J., Larochelle, H. & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959.
  77. Sobol', I. M. (1967). On the distribution of points in a cube and the approximate evaluation of integrals. USSR Computational Mathematics and Mathematical Physics, 7(4), 86–112.
  78. Stillinger, F. H. & Weber, T. A. (1984). Packing Structures and Transitions in Liquids and Solids. Science, 225(4666), 983–989.
  79. The SciPy community (n.d.). Optimization (scipy.optimize). SciPy User Guide [software documentation]. Accessed 20 November 2026.
  80. Vandenberghe, L. (n.d.). Gradient methods. UCLA EE236C [lecture notes]. Accessed 20 November 2026.
  81. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 5998–6008.
  82. Venturi, L., Bandeira, A. S. & Bruna, J. (2019). Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes. Journal of Machine Learning Research, 20(133), 1–34.
  83. Verpoort, P. C., Lee, A. A. & Wales, D. J. (2020). Archetypal landscapes for deep neural networks. Proceedings of the National Academy of Sciences, 117(36), 21857–21864.
  84. Wales, D. J. & Doye, J. P. K. (1997). Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms. The Journal of Physical Chemistry A, 101(28), 5111–5116.
  85. Wales, D. J. (2003). Energy Landscapes: Applications to Clusters, Biomolecules and Glasses. Cambridge University Press.
  86. Watson, L. T. & He, J. (2008). The Deterministic Global Optimization Algorithm DIRECT. HPCS/BIS Tutorial, 16 April 2008, Ottawa, Canada.
  87. Wikipedia contributors (n.d.). Ellipsoid method. Wikipedia. Accessed 20 November 2026.
  88. Wikipedia contributors (n.d.). Karmarkar’s algorithm. Wikipedia. Accessed 20 November 2026.
  89. Wikipedia contributors (n.d.). Leonid Khachiyan. Wikipedia. Accessed 20 November 2026.
  90. Wikipedia contributors (n.d.). Mathematical optimization. Wikipedia. Accessed 20 November 2026.
  91. Wikipedia contributors (n.d.). Narendra Karmarkar. Wikipedia. Accessed 20 November 2026.
  92. Wikipedia contributors (n.d.). Quasi-Newton method. Wikipedia. Accessed 20 November 2026.
  93. Wright, S. (1932). The roles of mutation, inbreeding, crossbreeding and selection in evolution. Proceedings of the 6th International Congress of Genetics, 1, 356–366.
  94. Wu, L., Zhu, Z. & E, W. (2017). Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes. arXiv:1706.10239.
  95. Zhou, Z., Zhu, Z., Li, X., Galkin, M., Feng, X., Koyejo, S., Tang, J. & Han, B. (2026). Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models. International Conference on Learning Representations (ICLR).