HeeSeung Jung

IEEE Transaction on Neural Networks and Learning Systems (Waiting for the acceptance)

The flexibility of decision boundaries in neural networks that are unguided by training data is a well-known problem typically resolved with generalization methods. A surprising result from recent knowledge distillation (KD) literature is that random, untrained, and equally structured teacher networks can also vastly improve generalization performance. It raises the possibility of existence of undiscovered assumptions useful for generalization on an uncertain region. In this paper, we shed light on the assumptions by analyzing decision boundaries and confidence distributions of both simple and KD-based generalization methods. Assuming that a decision boundary exists to represent the most general tendency of distinction on an input sample space (i.e., the simplest hypothesis), we show the various limitations of methods when using the hypothesis. To resolve these limitations, we propose matured dumb teacher based KD, conservatively transferring the hypothesis for generalization of the student without massive destruction of trained information. In practical experiments on feed-forward and convolution neural networks for image classification tasks on MNIST, CIFAR-10, and CIFAR-100 datasets, the proposed method shows stable improvement to the best test performance in the grid search of hyperparameters. The analysis and results imply that the proposed method can provide finer generalization than existing methods.

Applied Science

Most neural machine translation models are implemented as a conditional language model framework composed of encoder and decoder models. This framework learns complex and long-distant dependencies, but its deep structure causes inefficiency in training. Matching vector representations of source and target sentences improves the inefficiency by shortening the depth from parameters to costs and generalizes NMTs with different perspective to cross-entropy loss. In this paper, we propose matching methods to derive the cost based on constant word embedding vectors of source and target sentences. To find the best method, we analyze impact of the methods with varying structures, distance metrics, and model capacity in a French to English translation task. An optimally configured method is applied to English from and to French, Spanish, and German translation tasks. In the tasks, the method showed performance improvement by 3.23 BLEU in maximum, 0.71 in average. We evaluated the robustness of this method to various embedding distributions and models as conventional gated structures and transformer network, and empirical results showed that it has higher chance to improve performance in those variety.

[The bachelor thesis] The Korean Institute of Electrical Engineers

Recently, machine learning has been applied in a variety of fields. Speaker recognition is one of attractive applications of machine learning. In this paper, we propose a convolutional Siamese neural network for speaker recognition. The proposed model generates feature vectors through the identical two convolutional neural networks for speech data of two speakers. The similarity is measured by calculating the Euclidean distance of two output feature vectors. If the calculated similarity is less than the threshold, it is judged that two speakers are the same.
The experimental result of the proposed speaker recognition based on the convolutional Siamese neural network showed its accuracy was achieved up to 96%. The accuracy of one-shot classification using the trained convolutional Siamese neural network was evaluated also.
For the evaluation, the 10-way one-shot classification for 10 speakers not used for learning stages were tested, resulting in 92% accuracy.