References Return to the slides · Bibliographic data Agnihotri, A. & Batra, N. (2020). Exploring Bayesian Optimization . Distill , 5(5), e26. Ainsworth, S. K., Hayase, J. & Srinivasa, S. (2023). Git Re-Basin: Merging Models modulo Permutation Symmetries . International Conference on Learning Representations (ICLR) . Ainsworth, S. K., Hayase, J. & Srinivasa, S. (n.d.). Git Re-Basin: Official implementation . GitHub [software repository] . Accessed 20 November 2026. Amsel, N., Persson, D., Musco, C. & Gower, R. M. (2026). The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm . International Conference on Learning Representations (ICLR) . Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization . Journal of Machine Learning Research , 13, 281–305. Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent . Proceedings of COMPSTAT'2010 , 177–186. c-lasso contributors (n.d.). c-lasso: Constrained sparse regression . bio-datascience [software repository] . Accessed 20 November 2026. Cevher, V., Becker, S. & Schmidt, M. (2014). Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics . IEEE Signal Processing Magazine , 31(5), 32–43. Combettes, P. L. & Pesquet, J. C. (2011). Proximal Splitting Methods in Signal Processing . In Fixed-Point Algorithms for Inverse Problems in Science and Engineering. Springer , 185–212. Combettes, P. L. & Müller, C. L. (2020). Perspective maximum likelihood-type estimation via proximal decomposition . Electronic Journal of Statistics , 14(1), 207–238. CVXPYlayers contributors (n.d.). CVXPYlayers: Differentiable convex optimization layers . CVXPY [software repository] . Accessed 20 November 2026. Davidon, W. C. (1959). Variable Metric Method for Minimization . Argonne National Laboratory Report ANL-5990 . Davis, D. (n.d.). Teaching and research resources . Damek Davis, personal academic website . Accessed 20 November 2026. Dawkins, R. (1996). Climbing Mount Improbable . W. W. Norton & Company . Dial, L. (2026). Architecture Decisions for Marin 535B-A23B . Open Athena Blog, 9 November 2026 . Draxler, F., Veschgini, K., Salmhofer, M. & Hamprecht, F. A. (2018). Essentially No Barriers in Neural Network Energy Landscape . Proceedings of Machine Learning Research , 80, 1309–1318. Duchi, J., Shalev-Shwartz, S., Singer, Y. & Chandra, T. (2008). Efficient projections onto the ℓ₁-ball for learning in high dimensions . Proceedings of the 25th international conference on Machine learning - ICML '08 , 272–279. Duchi, J., Hazan, E. & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization . Journal of Machine Learning Research , 12, 2121–2159. Edureka (n.d.). What is Deep Learning? . Edureka blog . Accessed 20 November 2026. Eyring, H. & Polanyi, M. (1931). Über einfache Gasreaktionen . Zeitschrift für Physikalische Chemie, Abteilung B , 12, 279–311. Fort, S. & Jastrzębski, S. (2019). Large Scale Structure of Neural Network Loss Landscapes . Advances in Neural Information Processing Systems , 32. Fort, S. (2022). Post on Git Re-Basin, replication, and loss-landscape claims . X, 15 September 2022 [social media post] . Frank, M. & Wolfe, P. (1956). An algorithm for quadratic programming . Naval Research Logistics Quarterly , 3(1-2), 95–110. Frauenfelder, H., Sligar, S. G. & Wolynes, P. G. (1991). The Energy Landscapes and Motions of Proteins . Science , 254(5038), 1598–1603. Grant, M. & Boyd, S. (n.d.). CVX: Matlab Software for Disciplined Convex Programming . CVX Research [software and user guide] . Accessed 20 November 2026. Gurobi Optimization, LLC (n.d.). Gurobi Optimizer . Gurobi [optimization software] . Accessed 20 November 2026. Hall, D. (2026). Initial Pre-Training Checkpoints for Marin “Mochi” 535B-A23B . Open Athena Blog, 17 November 2026 . Hansen, N. (2016). The CMA Evolution Strategy: A Tutorial . arXiv:1604.00772 . Hazan, E. & Kakade, S. (2019). Revisiting the Polyak step size . arXiv:1905.00313 . He, K., Zhang, X., Ren, S. & Sun, J. (2016). Deep Residual Learning for Image Recognition . Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 770–778. Hutson, M. (2019). AI protein-folding algorithms solve structures faster than ever . Nature . INFORMS Computing Society (n.d.). The Nature of Mathematical Programming . INFORMS Computing Society glossary [archived 5 March 2014] . Accessed 20 November 2026. Jaggi, M. (2013). Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization . Proceedings of Machine Learning Research , 28(1), 427–435. Jaggi, M. (n.d.). Optimization for Machine Learning, Lecture 2 . EPFL Machine Learning and Optimization Laboratory [lecture slides] . Accessed 20 November 2026. JAX authors (n.d.). JAX documentation . JAX [software documentation] . Accessed 20 November 2026. Johnson, S. G. (n.d.). The NLopt nonlinear-optimization package . NLopt [software documentation] . Accessed 20 November 2026. Jones, D. R., Perttunen, C. D. & Stuckman, B. E. (1993). Lipschitzian optimization without the Lipschitz constant . Journal of Optimization Theory and Applications , 79(1), 157–181. Jones, D. R., Schonlau, M. & Welch, W. J. (1998). Efficient Global Optimization of Expensive Black-Box Functions . Journal of Global Optimization , 13(4), 455–492. Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L. & Bernstein, J. (2024). Muon: An optimizer for hidden layers in neural networks . Keller Jordan blog . Jordan, K., and Muon contributors (n.d.). Muon: PyTorch implementation . GitHub [software; commit f98f1cacc0263b04290753e32be8d498c1efc806] . Accessed 20 November 2026. Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., et al. (2021). Advances and Open Problems in Federated Learning . Foundations and Trends in Machine Learning , 14(1-2), 1–210. Karmarkar, N. (1984). A new polynomial-time algorithm for linear programming . Combinatorica , 4(4), 373–395. Kingma, D. P. & Ba, J. (2015). Adam: A Method for Stochastic Optimization . International Conference on Learning Representations (ICLR) . Kirkpatrick, S., Gelatt, C. D. & Vecchi, M. P. (1983). Optimization by Simulated Annealing . Science , 220(4598), 671–680. Kjellström, G. (1969). Network Optimization by Random Variation of Component Values . Ericsson Technics , 25(3), 133–151. Kjellström, G. & Taxén, L. (1981). Stochastic optimization in system design . IEEE Transactions on Circuits and Systems , 28(7), 702–715. Kjellström, G. (1991). On the efficiency of Gaussian adaptation . Journal of Optimization Theory and Applications , 71(3), 589–597. Kjellström, G. & Taxén, L. (1992). Gaussian Adaptation, an evolution-based efficient global optimizer . Computational and Applied Mathematics. Elsevier Science , 267–276. Kolb, C., Müller, C. L., Bischl, B. & Rügamer, D. (2026). Smoothing the Edges: Smooth Optimization for Sparse Regularization Using Hadamard Overparametrization . Machine Learning , 115(4), Article 87. LearnThermo (n.d.). Thermodynamics tutorial, Chapter 3, Lesson A, page 1 . LearnThermo [archived 2 March 2024] . Accessed 20 November 2026. Li, H., Xu, Z., Taylor, G., Studer, C. & Goldstein, T. (2018). Visualizing the Loss Landscape of Neural Nets . Advances in Neural Information Processing Systems , 31. Liu, D. C. & Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization . Mathematical Programming , 45(1-3), 503–528. Loshchilov, I. & Hutter, F. (2016). CMA-ES for Hyperparameter Optimization of Deep Neural Networks . ICLR Workshop . Loshchilov, I. & Hutter, F. (2019). Decoupled Weight Decay Regularization . International Conference on Learning Representations (ICLR) . Maheswaranathan, N., Metz, L., Tucker, G., Choi, D. & Sohl-Dickstein, J. (2019). Guided evolutionary strategies: augmenting random search with surrogate gradients . Proceedings of Machine Learning Research , 97, 4264–4273. Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C. & Rudolph, G. (2011). Exploratory landscape analysis . Proceedings of the 13th annual conference on Genetic and evolutionary computation , 829–836. Mockus, J. (1989). Bayesian Approach to Global Optimization: Theory and Applications . Kluwer Academic Publishers . MOSEK ApS (n.d.). MOSEK Optimization Suite . MOSEK [optimization software] . Accessed 20 November 2026. Müller, C. L. (2010). Black-box Landscapes: Characterization, Optimization, Sampling, and Application to Geometric Configuration Problems . PhD thesis, ETH Zürich, Dissertation No. 19438 . Müller, C. L. & Sbalzarini, I. F. (2010). Gaussian Adaptation Revisited—An Entropic View on Covariance Matrix Adaptation . Applications of Evolutionary Computation, LNCS , 6024, 432–441. Müller, C. L. (2014). Stochastic Methods for Single Objective Global Optimization . Computational Intelligence in Aerospace Sciences , 63–112. Neri, F. & Cotta, C. (2012). Memetic algorithms and memetic computing optimization: A literature review . Swarm and Evolutionary Computation , 2, 1–14. Parikh, N. & Boyd, S. (2014). Proximal Algorithms . Foundations and Trends in Optimization , 1(3), 127–239. PETSc development team (n.d.). SNES: Nonlinear Solvers . PETSc Manual [software documentation] . Accessed 20 November 2026. PyTorch contributors (n.d.). torch.optim . PyTorch documentation . Accessed 20 November 2026. Qiu, X., Gan, Y., Hayes, C. F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B. & Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning . International Conference on Machine Learning (ICML) . Rastrigin, L. A. (1963). The convergence of the random search method in the extremal control of a many-parameter system . Automation and Remote Control , 24, 1337–1342. Robbins, H. & Monro, S. (1951). A Stochastic Approximation Method . The Annals of Mathematical Statistics , 22(3), 400–407. Rockafellar, R. T. (1993). Lagrange Multipliers and Optimality . SIAM Review , 35(2), 183–238. Romero, P. A. & Arnold, F. H. (2009). Exploring protein fitness landscapes by directed evolution . Nature Reviews Molecular Cell Biology , 10(12), 866–876. Romero, P. A., Krause, A. & Arnold, F. H. (2013). Navigating the protein fitness landscape with Gaussian processes . Proceedings of the National Academy of Sciences , 110(3), E193–E201. Schaipp, F. (2023). Decay No More . ICLR Blogposts 2023 . Schaipp, F., Hägele, A., Taylor, A., Simsekli, U. & Bach, F. (2025). The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training . arXiv:2501.18965 . Schmidt, R. M., Schneider, F. & Hennig, P. (2021). Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers . International Conference on Learning Representations (ICLR) . Singhee, A. & Rutenbar, R. A. (2009). Novel Algorithms for Fast Statistical Analysis of Scaled Circuits . Lecture Notes in Electrical Engineering, vol. 46. Springer . Snoek, J., Larochelle, H. & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms . Advances in Neural Information Processing Systems , 25, 2951–2959. Sobol', I. M. (1967). On the distribution of points in a cube and the approximate evaluation of integrals . USSR Computational Mathematics and Mathematical Physics , 7(4), 86–112. Stillinger, F. H. & Weber, T. A. (1984). Packing Structures and Transitions in Liquids and Solids . Science , 225(4666), 983–989. The SciPy community (n.d.). Optimization (scipy.optimize) . SciPy User Guide [software documentation] . Accessed 20 November 2026. Vandenberghe, L. (n.d.). Gradient methods . UCLA EE236C [lecture notes] . Accessed 20 November 2026. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. (2017). Attention Is All You Need . Advances in Neural Information Processing Systems , 30, 5998–6008. Venturi, L., Bandeira, A. S. & Bruna, J. (2019). Spurious Valleys in One-hidden-layer Neural Network Optimization Landscapes . Journal of Machine Learning Research , 20(133), 1–34. Verpoort, P. C., Lee, A. A. & Wales, D. J. (2020). Archetypal landscapes for deep neural networks . Proceedings of the National Academy of Sciences , 117(36), 21857–21864. Wales, D. J. & Doye, J. P. K. (1997). Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms . The Journal of Physical Chemistry A , 101(28), 5111–5116. Wales, D. J. (2003). Energy Landscapes: Applications to Clusters, Biomolecules and Glasses . Cambridge University Press . Watson, L. T. & He, J. (2008). The Deterministic Global Optimization Algorithm DIRECT . HPCS/BIS Tutorial, 16 April 2008, Ottawa, Canada . Wikipedia contributors (n.d.). Ellipsoid method . Wikipedia . Accessed 20 November 2026. Wikipedia contributors (n.d.). Karmarkar’s algorithm . Wikipedia . Accessed 20 November 2026. Wikipedia contributors (n.d.). Leonid Khachiyan . Wikipedia . Accessed 20 November 2026. Wikipedia contributors (n.d.). Mathematical optimization . Wikipedia . Accessed 20 November 2026. Wikipedia contributors (n.d.). Narendra Karmarkar . Wikipedia . Accessed 20 November 2026. Wikipedia contributors (n.d.). Quasi-Newton method . Wikipedia . Accessed 20 November 2026. Wright, S. (1932). The roles of mutation, inbreeding, crossbreeding and selection in evolution . Proceedings of the 6th International Congress of Genetics , 1, 356–366. Wu, L., Zhu, Z. & E, W. (2017). Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes . arXiv:1706.10239 . Zhou, Z., Zhu, Z., Li, X., Galkin, M., Feng, X., Koyejo, S., Tang, J. & Han, B. (2026). Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models . International Conference on Learning Representations (ICLR) .