Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

Felix Petersen; Christian Borgelt; Hilde Kühne; Oliver Deussen

Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

Felix Petersen, Christian Borgelt, Hilde Kühne, Oliver Deussen

Artificial Intelligence and Human Interfaces

Publikation: Beitrag in Buch/Bericht/Konferenzband/Gesetzeskommentar › Konferenzbeitrag › Peer-reviewed

Abstract

Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achieve
stable training on large input sets of up to 1024 elements.

Originalsprache	Englisch
Titel	Proceedings of the 38th International Conference on Machine Learning
Untertitel	Proceedings of Machine Learning Research 139
Seiten	8546-8555
Seitenumfang	10
Band	139
Publikationsstatus	Veröffentlicht - 18 Juli 2021
Veranstaltung	International Conference on Machine Learning 2021 - Online Dauer: 18 Juli 2021 → 24 Juli 2021 https://icml.cc/Conferences/2021

Publikationsreihe

Name	Proceedings of Machine Learning Research
Herausgeber (Verlag)	Proceedings of Machine Learning Research
ISSN (elektronisch)	1938-7228

Konferenz

Konferenz	International Conference on Machine Learning 2021
Kurztitel	ICML 2021
Zeitraum	18/07/21 → 24/07/21
Internetadresse	https://icml.cc/Conferences/2021

Schlagwörter

neural networks
sorting networks

Systematik der Wissenschaftszweige 2012

102 Informatik

Andere Dateien und Links

https://proceedings.mlr.press/v139/petersen21a

Dieses zitieren

@inproceedings{1e69e20792b84bb585f3612bf8d5b3d9,

title = "Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision",

abstract = "Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achievestable training on large input sets of up to 1024 elements.",

keywords = "neural networks, sorting networks",

author = "Felix Petersen and Christian Borgelt and Hilde K{\"u}hne and Oliver Deussen",

year = "2021",

month = jul,

day = "18",

language = "English",

volume = "139",

series = "Proceedings of Machine Learning Research",

publisher = "Proceedings of Machine Learning Research",

pages = "8546--8555",

booktitle = "Proceedings of the 38th International Conference on Machine Learning",

note = "International Conference on Machine Learning 2021, ICML 2021 ; Conference date: 18-07-2021 Through 24-07-2021",

url = "https://icml.cc/Conferences/2021",

}

Petersen, F, Borgelt, C, Kühne, H & Deussen, O 2021, Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision. in Proceedings of the 38th International Conference on Machine Learning: Proceedings of Machine Learning Research 139. Bd. 139, Proceedings of Machine Learning Research, S. 8546-8555, International Conference on Machine Learning 2021, 18/07/21.

Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision. / Petersen, Felix; Borgelt, Christian; Kühne, Hilde et al.
Proceedings of the 38th International Conference on Machine Learning: Proceedings of Machine Learning Research 139. Band 139 2021. S. 8546-8555 (Proceedings of Machine Learning Research).

Publikation: Beitrag in Buch/Bericht/Konferenzband/Gesetzeskommentar › Konferenzbeitrag › Peer-reviewed

TY - GEN

T1 - Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

AU - Petersen, Felix

AU - Borgelt, Christian

AU - Kühne, Hilde

AU - Deussen, Oliver

PY - 2021/7/18

Y1 - 2021/7/18

N2 - Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achievestable training on large input sets of up to 1024 elements.

AB - Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achievestable training on large input sets of up to 1024 elements.

KW - neural networks

KW - sorting networks

UR - https://proceedings.mlr.press/v139/petersen21a

M3 - Conference contribution

VL - 139

T3 - Proceedings of Machine Learning Research

SP - 8546

EP - 8555

BT - Proceedings of the 38th International Conference on Machine Learning

T2 - International Conference on Machine Learning 2021

Y2 - 18 July 2021 through 24 July 2021

ER -