Incremental Neural Generalized Non-Metric Multidimensional Scaling
Abstract
The paper considers an incremental version of neural generalized non-metric multidimensional scaling intended for extending a representation space after the appearance of new objects. Unlike classical GNMDS, where the entire set of objects is jointly optimized according to ordinal constraints, the incremental scenario separates training into old and new portions of data. The modes Old Objects Only, Full Space Retraining, Fixed Old Coordinates, and Expanded Old Coordinates are studied on the MEN, SimLex-999, and WordSim-353 datasets. It is shown that Fixed Old Coordinates worsens the quality metric on old-new pairs, whereas Space Expansion with L2 regularization of the additional block of old coordinates substantially improves the rank-based quality metric and at the same time limits the deformation of old objects.
References
2. Kruskal J.B. Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis. Psychometrika. 1964;29(1):1-27. https://doi.org/10.1007/BF02289565
3. Kruskal J.B. Nonmetric Multidimensional Scaling: A Numerical Method. Psychometrika. 1964;29(2):115-129. https://doi.org/10.1007/BF02289694
4. Agarwal S., Wills J., Cayton L., Lanckriet G., Kriegman D., Belongie S. Generalized Non-metric Multidimensional Scaling. In: Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR. 2007;2:11-18. Available at: https://proceedings.mlr.press/v2/agarwal07a.html (accessed 20.02.2026).
5. Schroff F., Kalenichenko D., Philbin J. FaceNet: A Unified Embedding for Face Recognition and Clustering. In: Proceedings of CVPR, 2015. p. 815-823. Available at: https://doi.org/10.1109/CVPR.2015.7298682 (accessed 20.02.2026).
6. Weinberger K.Q., Saul L.K. Distance Metric Learning for Large Margin Nearest Neighbor Classification. Journal of Machine Learning Research. 2009;10(9):207-244. Available at: https://jmlr.org/papers/v10/weinberger09a.html (accessed 20.02.2026).
7. Harris Z.S. Distributional Structure. WORD. 1954;10(2-3):146-162. https://doi.org/10.1080/00437956.1954.11659520
8. Mikolov T., Chen K., Corrado G., Dean J. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781, 2013. https://doi.org/10.48550/arXiv.1301.3781
9. Mikolov T., Sutskever I., Chen K., Corrado G., Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Advances in Neural Information Processing Systems (NIPS 2013). Vol. 26. Curran Associates, Inc.; 2013. 9 p. Available at: https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality (accessed 20.02.2026).
10. Pennington J., Socher R., Manning C. GloVe: Global Vectors for Word Representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics; 2014. p. 1532-1543. https://doi.org/10.3115/v1/D14-1162
11. Finkelstein L., Gabrilovich E., Matias Y., Rivlin E., Solan Z., Wolfman G., Ruppin E. Placing Search in Context: The Concept Revisited. ACM Transactions on Information Systems. 2002;20(1):116-131. https://doi.org/10.1145/503104.503110
12. Hill F., Reichart R., Korhonen A. SimLex-999: Evaluating Semantic Models With Genuine Similarity Estimation. Computational Linguistics. 2015;41(4):665-695. https://doi.org/10.1162/COLI_a_00237
13. Bruni E., Tran N.K., Baroni M. Multimodal Distributional Semantics. Journal of Artificial Intelligence Research. 2014;49:1-47. https://doi.org/10.1613/jair.4135
14. Terada Y., von Luxburg U. Local Ordinal Embedding. In: Proceedings of the 31st International Conference on Machine Learning, PMLR. 2014;32(2):847-855. Available at: https://proceedings.mlr.press/v32/terada14.html (accessed 20.02.2026).
15. Jamieson K.G., Nowak R.D. Low-dimensional embedding using adaptively selected ordinal data. In: 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton). Monticello, IL, USA: IEEE Press; 2011. p. 1077-1084. https://doi.org/10.1109/Allerton.2011.6120287
16. Nocedal J., Wright S.J. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer New York, NY; 2006. 664 p. https://doi.org/10.1007/978-0-387-40065-5
17. Dauphin Y.N., Pascanu R., Gulcehre C., Cho K., Ganguli S., Bengio Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In: Advances in Neural Information Processing Systems (NIPS 2014). Vol. 27. Curran Associates, Inc.; 2014. 9 p. Available at: https://papers.nips.cc/paper/5486-identifying-and-attacking-the-saddle-point-problem-in-high-dimensional-non-convex-optimization (accessed 20.02.2026).
18. Robbins H., Monro S. A Stochastic Approximation Method. Annals of Mathematical Statistics. 1951;22(3):400-407. https://doi.org/10.1214/aoms/1177729586
19. Bottou L. Large-Scale Machine Learning with Stochastic Gradient Descent. In: Lechevallier Y., Saporta G. (eds) Proceedings of COMPSTAT'2010. Physica-Verlag HD; 2010. p. 177-186. https://doi.org/10.1007/978-3-7908-2604-3_16
20. Ghadimi S., Lan G. Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming. SIAM Journal on Optimization. 2013;23(4):2341-2368. https://doi.org/10.1137/120880811
21. Tikhonov A.N., Arsenin V.Y. Solutions of Ill-posed Problems. V.H. Winston and Son, Inc.; 1977. 258 p.
22. Goodfellow I., Bengio Y., Courville A. Deep Learning. MIT Press; 2016. Available at: http://www.deeplearningbook.org (accessed 20.02.2026).
23. Hinton G.E., Salakhutdinov R.R. Reducing the Dimensionality of Data with Neural Networks. Science. 2006;313(5786):504-507. https://doi.org/10.1126/science.1127647
24. Belkin M., Niyogi P. Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering. In: Advances in Neural Information Processing Systems (NIPS 2001). Vol. 14. MIT Press; 2001. 7 p. Available at: https://proceedings.neurips.cc/paper/2001/hash/f106b7f99d2cb30c3db1c3cc0fde9ccb-Abstract.html (accessed 20.02.2026).
25. GNMDS Implementation Repository: obj2vec/gnmds. Available at: https://github.com/obj2vec/gnmds (accessed 20.02.2026).
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication policy of the journal is based on traditional ethical principles of the Russian scientific periodicals and is built in terms of ethical norms of editors and publishers work stated in Code of Conduct and Best Practice Guidelines for Journal Editors and Code of Conduct for Journal Publishers, developed by the Committee on Publication Ethics (COPE). In the course of publishing editorial board of the journal is led by international rules for copyright protection, statutory regulations of the Russian Federation as well as international standards of publishing.
Authors publishing articles in this journal agree to the following: They retain copyright and grant the journal right of first publication of the work, which is automatically licensed under the Creative Commons Attribution License (CC BY license). Users can use, reuse and build upon the material published in this journal provided that such uses are fully attributed.
