2021-03-14 08:18:14.413: INFO @api : Created output directory with name log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:18:14.413: INFO @api : Creating dataset 2021-03-14 08:18:15.525: WARNING @lazy_loader : The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. 2021-03-14 08:18:15.526: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/utils/profiler.py:12: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2021-03-14 08:18:15.550: INFO @api : EmbeddingDatasetWrapper created 2021-03-14 08:18:15.550: INFO @api : Creating a model 2021-03-14 08:18:15.564: INFO @model : Creating TF model on 1 GPU devices 2021-03-14 08:18:15.565: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:427: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. 2021-03-14 08:18:15.899: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2021-03-14 08:18:15.899: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead. 2021-03-14 08:18:15.900: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. 2021-03-14 08:18:15.900: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:143: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-03-14 08:18:15.902: WARNING @module_wrapper: From /home/kopalja/faces/models/embedding_body.py:59: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-03-14 08:18:15.911: WARNING @deprecation : From /usr/lib/python3.8/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. 2021-03-14 08:18:16.249: WARNING @deprecation : From /home/kopalja/faces/models/embedding_body.py:72: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.flatten instead. 2021-03-14 08:18:16.253: WARNING @deprecation : From /home/kopalja/faces/models/embedding_body.py:75: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. 2021-03-14 08:18:16.294: WARNING @deprecation : From /usr/lib/python3.8/site-packages/tensorflow_core/contrib/losses/python/metric_learning/metric_loss_ops.py:219: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where 2021-03-14 08:18:16.305: WARNING @module_wrapper: From /home/kopalja/faces/models/embedding_body.py:18: The name tf.diag_part is deprecated. Please use tf.linalg.tensor_diag_part instead. 2021-03-14 08:18:16.325: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:149: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead. 2021-03-14 08:18:16.325: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/graph_tower.py:80: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-03-14 08:18:17.117: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:182: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead. 2021-03-14 08:18:17.164: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:516: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead. 2021-03-14 08:18:18.085: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:517: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2021-03-14 08:18:18.158: INFO @model : Number of parameters: 1847248 2021-03-14 08:18:18.158: INFO @api : BodyEmbeddingModel created 2021-03-14 08:18:18.158: INFO @api : Creating hooks 2021-03-14 08:18:18.168: INFO @api : ComputeStats created 2021-03-14 08:18:18.168: INFO @api : LogVariables created 2021-03-14 08:18:18.169: INFO @api : LogProfile created 2021-03-14 08:18:18.169: INFO @api : LogDir created 2021-03-14 08:18:18.338: INFO @loader : Loading faiss with AVX2 support. 2021-03-14 08:18:18.338: INFO @loader : Loading faiss. 2021-03-14 08:18:18.373: INFO @api : ComputeMatchingMetrics created 2021-03-14 08:18:18.374: WARNING @abstract_hook: Argument `streams` was not recognized by `WriteTensorBoard`. Recognized arguments are `['model', 'output_dir', 'image_variables', 'flush_secs', 'visualize_graph', 'on_unknown_type', 'on_missing_variable', 'kwargs']`. 2021-03-14 08:18:18.374: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/write_tensorboard.py:68: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead. 2021-03-14 08:18:18.375: INFO @api : WriteTensorBoard created 2021-03-14 08:18:18.783: INFO @api : EvaluateEmbedding created 2021-03-14 08:18:18.784: INFO @api : DecayLR created 2021-03-14 08:18:18.785: INFO @api : SaveEvery created 2021-03-14 08:18:18.785: INFO @api : StopAfter created 2021-03-14 08:18:18.786: INFO @api : ShowProgress created 2021-03-14 08:18:18.786: WARNING @api : TrainingTrace hook added between hooks. Add it to your config.yaml to suppress this warning. 2021-03-14 08:18:18.786: INFO @api : Creating main loop 2021-03-14 08:18:18.786: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:18:18.787: INFO @main_loop : Training epoch 1 2021-03-14 08:18:20.901: WARNING @main_loop : Some sources provided by the stream do not match model placeholders. Set `main_loop.on_unused_sources` to `ignore` in order to suppress this warning. Extra sources: ['filenames'] 2021-03-14 08:21:48.661: INFO @log_variables: train loss nanmean: 0.485039 2021-03-14 08:21:48.661: INFO @log_variables: train positive_distance nanmean: 0.611778 2021-03-14 08:21:48.661: INFO @log_variables: train negative_distance nanmean: 1.413177 2021-03-14 08:21:48.661: INFO @log_variables: train distance nanmean: 0.801398 2021-03-14 08:21:48.661: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:23:04.371: INFO @metrics_hook: train matching accuracy: 0.8708889833343884, N=110587, DB_SIZE=110592, N_IDS=11058 2021-03-14 08:23:04.394: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/write_tensorboard.py:102: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead. 2021-03-14 08:23:04.466: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/decay_lr.py:89: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2021-03-14 08:23:04.541: INFO @decay_lr : LR updated to `0.000995` 2021-03-14 08:23:04.543: INFO @log_profile : T train: 206.319995 2021-03-14 08:23:04.543: INFO @log_profile : T read data: 3.203551 2021-03-14 08:23:04.543: INFO @log_profile : T hooks: 76.185914 2021-03-14 08:23:04.543: INFO @main_loop : Epoch 1 done 2021-03-14 08:23:04.543: INFO @main_loop : Training epoch 2 2021-03-14 08:25:34.776: INFO @log_variables: train loss nanmean: 0.309431 2021-03-14 08:25:34.776: INFO @log_variables: train positive_distance nanmean: 0.641748 2021-03-14 08:25:34.776: INFO @log_variables: train negative_distance nanmean: 1.417616 2021-03-14 08:25:34.776: INFO @log_variables: train distance nanmean: 0.775868 2021-03-14 08:25:34.776: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:26:51.743: INFO @metrics_hook: train matching accuracy: 0.9504647882229537, N=110588, DB_SIZE=110592, N_IDS=11072 2021-03-14 08:26:51.888: INFO @decay_lr : LR updated to `0.000990025` 2021-03-14 08:26:51.889: INFO @log_profile : T train: 147.766417 2021-03-14 08:26:51.889: INFO @log_profile : T read data: 2.092591 2021-03-14 08:26:51.889: INFO @log_profile : T hooks: 77.437203 2021-03-14 08:26:51.889: INFO @main_loop : Epoch 2 done 2021-03-14 08:26:51.889: INFO @main_loop : Training epoch 3 2021-03-14 08:29:04.970: INFO @log_variables: train loss nanmean: 0.254701 2021-03-14 08:29:04.970: INFO @log_variables: train positive_distance nanmean: 0.638114 2021-03-14 08:29:04.970: INFO @log_variables: train negative_distance nanmean: 1.418108 2021-03-14 08:29:04.970: INFO @log_variables: train distance nanmean: 0.779994 2021-03-14 08:29:04.970: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:30:20.945: INFO @metrics_hook: train matching accuracy: 0.968703373814282, N=110587, DB_SIZE=110592, N_IDS=11070 2021-03-14 08:30:21.090: INFO @decay_lr : LR updated to `0.0009850749` 2021-03-14 08:30:21.091: INFO @log_profile : T train: 130.698605 2021-03-14 08:30:21.091: INFO @log_profile : T read data: 1.990308 2021-03-14 08:30:21.091: INFO @log_profile : T hooks: 76.461786 2021-03-14 08:30:21.091: INFO @main_loop : Epoch 3 done 2021-03-14 08:30:21.091: INFO @main_loop : Training epoch 4 2021-03-14 08:32:27.515: INFO @log_variables: train loss nanmean: 0.221405 2021-03-14 08:32:27.515: INFO @log_variables: train positive_distance nanmean: 0.632257 2021-03-14 08:32:27.515: INFO @log_variables: train negative_distance nanmean: 1.418376 2021-03-14 08:32:27.515: INFO @log_variables: train distance nanmean: 0.786118 2021-03-14 08:32:27.515: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:33:42.565: INFO @metrics_hook: train matching accuracy: 0.9766695302256183, N=110585, DB_SIZE=110592, N_IDS=11077 2021-03-14 08:33:42.708: INFO @decay_lr : LR updated to `0.0009801495` 2021-03-14 08:33:42.709: INFO @log_profile : T train: 123.867471 2021-03-14 08:33:42.710: INFO @log_profile : T read data: 2.163755 2021-03-14 08:33:42.710: INFO @log_profile : T hooks: 75.535030 2021-03-14 08:33:42.710: INFO @main_loop : Epoch 4 done 2021-03-14 08:33:42.710: INFO @main_loop : Training epoch 5 2021-03-14 08:35:49.094: INFO @log_variables: train loss nanmean: 0.203679 2021-03-14 08:35:49.095: INFO @log_variables: train positive_distance nanmean: 0.629517 2021-03-14 08:35:49.095: INFO @log_variables: train negative_distance nanmean: 1.418481 2021-03-14 08:35:49.095: INFO @log_variables: train distance nanmean: 0.788964 2021-03-14 08:35:49.095: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:37:04.883: INFO @metrics_hook: train matching accuracy: 0.9786960846369472, N=110590, DB_SIZE=110592, N_IDS=11102 2021-03-14 08:37:05.024: INFO @decay_lr : LR updated to `0.00097524875` 2021-03-14 08:37:05.026: INFO @log_profile : T train: 123.790350 2021-03-14 08:37:05.026: INFO @log_profile : T read data: 2.204093 2021-03-14 08:37:05.026: INFO @log_profile : T hooks: 76.269671 2021-03-14 08:37:05.026: INFO @main_loop : Epoch 5 done 2021-03-14 08:37:05.026: INFO @main_loop : Training epoch 6 2021-03-14 08:39:11.656: INFO @log_variables: train loss nanmean: 0.183087 2021-03-14 08:39:11.657: INFO @log_variables: train positive_distance nanmean: 0.622967 2021-03-14 08:39:11.657: INFO @log_variables: train negative_distance nanmean: 1.418612 2021-03-14 08:39:11.657: INFO @log_variables: train distance nanmean: 0.795646 2021-03-14 08:39:11.657: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:40:26.283: INFO @metrics_hook: train matching accuracy: 0.9805042138387529, N=110588, DB_SIZE=110592, N_IDS=11080 2021-03-14 08:40:26.425: INFO @decay_lr : LR updated to `0.0009703725` 2021-03-14 08:40:26.426: INFO @log_profile : T train: 124.114490 2021-03-14 08:40:26.426: INFO @log_profile : T read data: 2.119026 2021-03-14 08:40:26.426: INFO @log_profile : T hooks: 75.114578 2021-03-14 08:40:26.426: INFO @main_loop : Epoch 6 done 2021-03-14 08:40:26.426: INFO @main_loop : Training epoch 7 2021-03-14 08:42:32.515: INFO @log_variables: train loss nanmean: 0.168241 2021-03-14 08:42:32.516: INFO @log_variables: train positive_distance nanmean: 0.617468 2021-03-14 08:42:32.516: INFO @log_variables: train negative_distance nanmean: 1.418740 2021-03-14 08:42:32.516: INFO @log_variables: train distance nanmean: 0.801272 2021-03-14 08:42:32.516: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:43:48.753: INFO @metrics_hook: train matching accuracy: 0.982439324338988, N=110588, DB_SIZE=110592, N_IDS=11076 2021-03-14 08:43:48.897: INFO @decay_lr : LR updated to `0.0009655207` 2021-03-14 08:43:48.898: INFO @log_profile : T train: 123.737019 2021-03-14 08:43:48.898: INFO @log_profile : T read data: 1.956709 2021-03-14 08:43:48.898: INFO @log_profile : T hooks: 76.726612 2021-03-14 08:43:48.898: INFO @main_loop : Epoch 7 done 2021-03-14 08:43:48.898: INFO @main_loop : Training epoch 8 2021-03-14 08:45:55.025: INFO @log_variables: train loss nanmean: 0.158788 2021-03-14 08:45:55.025: INFO @log_variables: train positive_distance nanmean: 0.614251 2021-03-14 08:45:55.025: INFO @log_variables: train negative_distance nanmean: 1.418787 2021-03-14 08:45:55.025: INFO @log_variables: train distance nanmean: 0.804536 2021-03-14 08:45:55.026: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:47:11.178: INFO @metrics_hook: train matching accuracy: 0.9831895538314073, N=110586, DB_SIZE=110592, N_IDS=11104 2021-03-14 08:47:11.327: INFO @decay_lr : LR updated to `0.00096069305` 2021-03-14 08:47:11.328: INFO @log_profile : T train: 123.673362 2021-03-14 08:47:11.328: INFO @log_profile : T read data: 2.067960 2021-03-14 08:47:11.328: INFO @log_profile : T hooks: 76.637416 2021-03-14 08:47:11.328: INFO @main_loop : Epoch 8 done 2021-03-14 08:47:11.328: INFO @main_loop : Training epoch 9 2021-03-14 08:49:17.040: INFO @log_variables: train loss nanmean: 0.151328 2021-03-14 08:49:17.040: INFO @log_variables: train positive_distance nanmean: 0.611973 2021-03-14 08:49:17.040: INFO @log_variables: train negative_distance nanmean: 1.418818 2021-03-14 08:49:17.040: INFO @log_variables: train distance nanmean: 0.806845 2021-03-14 08:49:17.041: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:50:33.159: INFO @metrics_hook: train matching accuracy: 0.9847269570564352, N=110587, DB_SIZE=110592, N_IDS=11066 2021-03-14 08:50:33.297: INFO @decay_lr : LR updated to `0.0009558896` 2021-03-14 08:50:33.298: INFO @log_profile : T train: 123.349416 2021-03-14 08:50:33.298: INFO @log_profile : T read data: 1.971663 2021-03-14 08:50:33.298: INFO @log_profile : T hooks: 76.597840 2021-03-14 08:50:33.298: INFO @main_loop : Epoch 9 done 2021-03-14 08:50:33.298: INFO @main_loop : Training epoch 10 2021-03-14 08:52:37.817: INFO @log_variables: train loss nanmean: 0.142985 2021-03-14 08:52:37.817: INFO @log_variables: train positive_distance nanmean: 0.609638 2021-03-14 08:52:37.817: INFO @log_variables: train negative_distance nanmean: 1.418889 2021-03-14 08:52:37.817: INFO @log_variables: train distance nanmean: 0.809251 2021-03-14 08:52:37.817: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:53:54.791: INFO @metrics_hook: train matching accuracy: 0.9852693831045521, N=110586, DB_SIZE=110592, N_IDS=11078 2021-03-14 08:53:54.934: INFO @decay_lr : LR updated to `0.00095111015` 2021-03-14 08:53:54.936: INFO @log_profile : T train: 123.374936 2021-03-14 08:53:54.936: INFO @log_profile : T read data: 0.756582 2021-03-14 08:53:54.936: INFO @log_profile : T hooks: 77.455391 2021-03-14 08:53:54.936: INFO @main_loop : Epoch 10 done 2021-03-14 08:53:54.936: INFO @main_loop : Training epoch 11 2021-03-14 08:55:59.695: INFO @log_variables: train loss nanmean: 0.135697 2021-03-14 08:55:59.695: INFO @log_variables: train positive_distance nanmean: 0.607683 2021-03-14 08:55:59.695: INFO @log_variables: train negative_distance nanmean: 1.418885 2021-03-14 08:55:59.695: INFO @log_variables: train distance nanmean: 0.811202 2021-03-14 08:55:59.695: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 08:57:16.784: INFO @metrics_hook: train matching accuracy: 0.9856947797701399, N=110589, DB_SIZE=110592, N_IDS=11082 2021-03-14 08:57:16.929: INFO @decay_lr : LR updated to `0.0009463546` 2021-03-14 08:57:16.931: INFO @log_profile : T train: 122.137266 2021-03-14 08:57:16.931: INFO @log_profile : T read data: 2.224250 2021-03-14 08:57:16.931: INFO @log_profile : T hooks: 77.581944 2021-03-14 08:57:16.931: INFO @main_loop : Epoch 11 done 2021-03-14 08:57:16.931: INFO @main_loop : Training epoch 12 2021-03-14 08:59:23.170: INFO @log_variables: train loss nanmean: 0.129824 2021-03-14 08:59:23.171: INFO @log_variables: train positive_distance nanmean: 0.605296 2021-03-14 08:59:23.171: INFO @log_variables: train negative_distance nanmean: 1.418937 2021-03-14 08:59:23.171: INFO @log_variables: train distance nanmean: 0.813641 2021-03-14 08:59:23.171: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:00:39.756: INFO @metrics_hook: train matching accuracy: 0.9856491780153003, N=110586, DB_SIZE=110592, N_IDS=11092 2021-03-14 09:00:39.920: INFO @decay_lr : LR updated to `0.0009416228` 2021-03-14 09:00:39.921: INFO @log_profile : T train: 123.746764 2021-03-14 09:00:39.921: INFO @log_profile : T read data: 2.114454 2021-03-14 09:00:39.921: INFO @log_profile : T hooks: 77.077346 2021-03-14 09:00:39.921: INFO @main_loop : Epoch 12 done 2021-03-14 09:00:39.921: INFO @main_loop : Training epoch 13 2021-03-14 09:02:46.299: INFO @log_variables: train loss nanmean: 0.124797 2021-03-14 09:02:46.300: INFO @log_variables: train positive_distance nanmean: 0.604355 2021-03-14 09:02:46.300: INFO @log_variables: train negative_distance nanmean: 1.418975 2021-03-14 09:02:46.300: INFO @log_variables: train distance nanmean: 0.814620 2021-03-14 09:02:46.300: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:04:03.153: INFO @metrics_hook: train matching accuracy: 0.9865446522226643, N=110588, DB_SIZE=110592, N_IDS=11086 2021-03-14 09:04:03.300: INFO @decay_lr : LR updated to `0.0009369147` 2021-03-14 09:04:03.301: INFO @log_profile : T train: 123.854087 2021-03-14 09:04:03.301: INFO @log_profile : T read data: 2.147969 2021-03-14 09:04:03.301: INFO @log_profile : T hooks: 77.325861 2021-03-14 09:04:03.302: INFO @main_loop : Epoch 13 done 2021-03-14 09:04:03.302: INFO @main_loop : Training epoch 14 2021-03-14 09:06:04.546: INFO @log_variables: train loss nanmean: 0.119183 2021-03-14 09:06:04.546: INFO @log_variables: train positive_distance nanmean: 0.603051 2021-03-14 09:06:04.546: INFO @log_variables: train negative_distance nanmean: 1.418993 2021-03-14 09:06:04.546: INFO @log_variables: train distance nanmean: 0.815943 2021-03-14 09:06:04.546: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:07:19.835: INFO @metrics_hook: train matching accuracy: 0.9867707165334394, N=110588, DB_SIZE=110592, N_IDS=11078 2021-03-14 09:07:19.982: INFO @decay_lr : LR updated to `0.00093223015` 2021-03-14 09:07:19.983: INFO @log_profile : T train: 118.736798 2021-03-14 09:07:19.983: INFO @log_profile : T read data: 2.115146 2021-03-14 09:07:19.983: INFO @log_profile : T hooks: 75.777480 2021-03-14 09:07:19.983: INFO @main_loop : Epoch 14 done 2021-03-14 09:07:19.983: INFO @main_loop : Training epoch 15 2021-03-14 09:09:25.761: INFO @log_variables: train loss nanmean: 0.116685 2021-03-14 09:09:25.762: INFO @log_variables: train positive_distance nanmean: 0.601240 2021-03-14 09:09:25.762: INFO @log_variables: train negative_distance nanmean: 1.419012 2021-03-14 09:09:25.762: INFO @log_variables: train distance nanmean: 0.817772 2021-03-14 09:09:25.762: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:10:40.176: INFO @metrics_hook: train matching accuracy: 0.9859660544900487, N=110589, DB_SIZE=110592, N_IDS=11070 2021-03-14 09:10:40.326: INFO @decay_lr : LR updated to `0.000927569` 2021-03-14 09:10:40.328: INFO @log_profile : T train: 123.205079 2021-03-14 09:10:40.328: INFO @log_profile : T read data: 2.185886 2021-03-14 09:10:40.328: INFO @log_profile : T hooks: 74.901955 2021-03-14 09:10:40.328: INFO @main_loop : Epoch 15 done 2021-03-14 09:10:40.328: INFO @main_loop : Training epoch 16 2021-03-14 09:12:46.785: INFO @log_variables: train loss nanmean: 0.111222 2021-03-14 09:12:46.785: INFO @log_variables: train positive_distance nanmean: 0.599655 2021-03-14 09:12:46.785: INFO @log_variables: train negative_distance nanmean: 1.419035 2021-03-14 09:12:46.785: INFO @log_variables: train distance nanmean: 0.819379 2021-03-14 09:12:46.785: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:14:03.692: INFO @metrics_hook: train matching accuracy: 0.987132070353122, N=110585, DB_SIZE=110592, N_IDS=11125 2021-03-14 09:14:03.842: INFO @decay_lr : LR updated to `0.0009229311` 2021-03-14 09:14:03.844: INFO @log_profile : T train: 123.858907 2021-03-14 09:14:03.844: INFO @log_profile : T read data: 2.232040 2021-03-14 09:14:03.844: INFO @log_profile : T hooks: 77.373692 2021-03-14 09:14:03.844: INFO @main_loop : Epoch 16 done 2021-03-14 09:14:03.844: INFO @main_loop : Training epoch 17 2021-03-14 09:16:08.718: INFO @log_variables: train loss nanmean: 0.105313 2021-03-14 09:16:08.718: INFO @log_variables: train positive_distance nanmean: 0.597773 2021-03-14 09:16:08.718: INFO @log_variables: train negative_distance nanmean: 1.419082 2021-03-14 09:16:08.719: INFO @log_variables: train distance nanmean: 0.821309 2021-03-14 09:16:08.719: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:17:24.984: INFO @metrics_hook: train matching accuracy: 0.9876388034868159, N=110588, DB_SIZE=110592, N_IDS=11077 2021-03-14 09:17:25.138: INFO @decay_lr : LR updated to `0.0009183165` 2021-03-14 09:17:25.139: INFO @log_profile : T train: 122.498514 2021-03-14 09:17:25.139: INFO @log_profile : T read data: 1.981602 2021-03-14 09:17:25.139: INFO @log_profile : T hooks: 76.764676 2021-03-14 09:17:25.139: INFO @main_loop : Epoch 17 done 2021-03-14 09:17:25.139: INFO @main_loop : Training epoch 18 2021-03-14 09:19:31.765: INFO @log_variables: train loss nanmean: 0.103204 2021-03-14 09:19:31.765: INFO @log_variables: train positive_distance nanmean: 0.595904 2021-03-14 09:19:31.766: INFO @log_variables: train negative_distance nanmean: 1.419086 2021-03-14 09:19:31.766: INFO @log_variables: train distance nanmean: 0.823182 2021-03-14 09:19:31.766: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:20:46.152: INFO @metrics_hook: train matching accuracy: 0.988217208482163, N=110585, DB_SIZE=110592, N_IDS=11074 2021-03-14 09:20:46.298: INFO @decay_lr : LR updated to `0.0009137249` 2021-03-14 09:20:46.299: INFO @log_profile : T train: 124.207679 2021-03-14 09:20:46.299: INFO @log_profile : T read data: 2.029159 2021-03-14 09:20:46.299: INFO @log_profile : T hooks: 74.870525 2021-03-14 09:20:46.299: INFO @main_loop : Epoch 18 done 2021-03-14 09:20:46.299: INFO @main_loop : Training epoch 19 2021-03-14 09:22:50.697: INFO @log_variables: train loss nanmean: 0.099982 2021-03-14 09:22:50.697: INFO @log_variables: train positive_distance nanmean: 0.596799 2021-03-14 09:22:50.697: INFO @log_variables: train negative_distance nanmean: 1.419082 2021-03-14 09:22:50.697: INFO @log_variables: train distance nanmean: 0.822283 2021-03-14 09:22:50.697: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:24:06.460: INFO @metrics_hook: train matching accuracy: 0.9880181581422267, N=110584, DB_SIZE=110592, N_IDS=11092 2021-03-14 09:24:06.611: INFO @decay_lr : LR updated to `0.0009091563` 2021-03-14 09:24:06.613: INFO @log_profile : T train: 123.254420 2021-03-14 09:24:06.613: INFO @log_profile : T read data: 0.749555 2021-03-14 09:24:06.613: INFO @log_profile : T hooks: 76.259441 2021-03-14 09:24:06.613: INFO @main_loop : Epoch 19 done 2021-03-14 09:24:06.613: INFO @main_loop : Training epoch 20 2021-03-14 09:26:13.043: INFO @log_variables: train loss nanmean: 0.093939 2021-03-14 09:26:13.043: INFO @log_variables: train positive_distance nanmean: 0.592808 2021-03-14 09:26:13.043: INFO @log_variables: train negative_distance nanmean: 1.419119 2021-03-14 09:26:13.043: INFO @log_variables: train distance nanmean: 0.826311 2021-03-14 09:26:13.043: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:27:28.698: INFO @metrics_hook: train matching accuracy: 0.9878192143528901, N=110584, DB_SIZE=110592, N_IDS=11081 2021-03-14 09:27:28.855: INFO @decay_lr : LR updated to `0.0009046105` 2021-03-14 09:27:28.857: INFO @log_profile : T train: 123.940711 2021-03-14 09:27:28.857: INFO @log_profile : T read data: 2.086241 2021-03-14 09:27:28.857: INFO @log_profile : T hooks: 76.164968 2021-03-14 09:27:28.857: INFO @main_loop : Epoch 20 done 2021-03-14 09:27:28.857: INFO @main_loop : Training epoch 21 2021-03-14 09:29:34.989: INFO @log_variables: train loss nanmean: 0.093612 2021-03-14 09:29:34.990: INFO @log_variables: train positive_distance nanmean: 0.593249 2021-03-14 09:29:34.990: INFO @log_variables: train negative_distance nanmean: 1.419125 2021-03-14 09:29:34.990: INFO @log_variables: train distance nanmean: 0.825876 2021-03-14 09:29:34.990: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:30:49.717: INFO @metrics_hook: train matching accuracy: 0.9886246496066552, N=110590, DB_SIZE=110592, N_IDS=11066 2021-03-14 09:30:49.870: INFO @decay_lr : LR updated to `0.00090008747` 2021-03-14 09:30:49.872: INFO @log_profile : T train: 123.663453 2021-03-14 09:30:49.872: INFO @log_profile : T read data: 2.078636 2021-03-14 09:30:49.872: INFO @log_profile : T hooks: 75.221253 2021-03-14 09:30:49.872: INFO @main_loop : Epoch 21 done 2021-03-14 09:30:49.872: INFO @main_loop : Training epoch 22 2021-03-14 09:32:56.102: INFO @log_variables: train loss nanmean: 0.091809 2021-03-14 09:32:56.103: INFO @log_variables: train positive_distance nanmean: 0.592150 2021-03-14 09:32:56.103: INFO @log_variables: train negative_distance nanmean: 1.419115 2021-03-14 09:32:56.103: INFO @log_variables: train distance nanmean: 0.826965 2021-03-14 09:32:56.103: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:34:11.110: INFO @metrics_hook: train matching accuracy: 0.9881454019350755, N=110590, DB_SIZE=110592, N_IDS=11088 2021-03-14 09:34:11.258: INFO @decay_lr : LR updated to `0.000895587` 2021-03-14 09:34:11.259: INFO @log_profile : T train: 123.692409 2021-03-14 09:34:11.259: INFO @log_profile : T read data: 2.150812 2021-03-14 09:34:11.259: INFO @log_profile : T hooks: 75.492984 2021-03-14 09:34:11.260: INFO @main_loop : Epoch 22 done 2021-03-14 09:34:11.260: INFO @main_loop : Training epoch 23 2021-03-14 09:36:17.288: INFO @log_variables: train loss nanmean: 0.088674 2021-03-14 09:36:17.288: INFO @log_variables: train positive_distance nanmean: 0.591418 2021-03-14 09:36:17.288: INFO @log_variables: train negative_distance nanmean: 1.419152 2021-03-14 09:36:17.288: INFO @log_variables: train distance nanmean: 0.827734 2021-03-14 09:36:17.288: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:37:34.110: INFO @metrics_hook: train matching accuracy: 0.9884163599699783, N=110587, DB_SIZE=110592, N_IDS=11079 2021-03-14 09:37:34.263: INFO @decay_lr : LR updated to `0.0008911091` 2021-03-14 09:37:34.264: INFO @log_profile : T train: 123.571169 2021-03-14 09:37:34.264: INFO @log_profile : T read data: 2.057367 2021-03-14 09:37:34.264: INFO @log_profile : T hooks: 77.322044 2021-03-14 09:37:34.264: INFO @main_loop : Epoch 23 done 2021-03-14 09:37:34.264: INFO @main_loop : Training epoch 24 2021-03-14 09:39:40.332: INFO @log_variables: train loss nanmean: 0.084564 2021-03-14 09:39:40.332: INFO @log_variables: train positive_distance nanmean: 0.588740 2021-03-14 09:39:40.332: INFO @log_variables: train negative_distance nanmean: 1.419151 2021-03-14 09:39:40.332: INFO @log_variables: train distance nanmean: 0.830411 2021-03-14 09:39:40.332: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:40:56.247: INFO @metrics_hook: train matching accuracy: 0.9881269950355829, N=110587, DB_SIZE=110592, N_IDS=11065 2021-03-14 09:40:56.397: INFO @decay_lr : LR updated to `0.00088665355` 2021-03-14 09:40:56.398: INFO @log_profile : T train: 123.690966 2021-03-14 09:40:56.398: INFO @log_profile : T read data: 1.976382 2021-03-14 09:40:56.398: INFO @log_profile : T hooks: 76.415800 2021-03-14 09:40:56.398: INFO @main_loop : Epoch 24 done 2021-03-14 09:40:56.398: INFO @main_loop : Training epoch 25 2021-03-14 09:43:02.369: INFO @log_variables: train loss nanmean: 0.083301 2021-03-14 09:43:02.369: INFO @log_variables: train positive_distance nanmean: 0.589523 2021-03-14 09:43:02.369: INFO @log_variables: train negative_distance nanmean: 1.419145 2021-03-14 09:43:02.369: INFO @log_variables: train distance nanmean: 0.829622 2021-03-14 09:43:02.369: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:44:18.386: INFO @metrics_hook: train matching accuracy: 0.9892210587235274, N=110586, DB_SIZE=110592, N_IDS=11095 2021-03-14 09:44:18.537: INFO @decay_lr : LR updated to `0.0008822203` 2021-03-14 09:44:18.539: INFO @log_profile : T train: 123.516273 2021-03-14 09:44:18.539: INFO @log_profile : T read data: 2.081076 2021-03-14 09:44:18.539: INFO @log_profile : T hooks: 76.492666 2021-03-14 09:44:18.540: INFO @main_loop : Epoch 25 done 2021-03-14 09:44:18.540: INFO @main_loop : Training epoch 26 2021-03-14 09:46:25.051: INFO @log_variables: train loss nanmean: 0.082132 2021-03-14 09:46:25.051: INFO @log_variables: train positive_distance nanmean: 0.590507 2021-03-14 09:46:25.051: INFO @log_variables: train negative_distance nanmean: 1.419173 2021-03-14 09:46:25.051: INFO @log_variables: train distance nanmean: 0.828667 2021-03-14 09:46:25.051: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:47:39.956: INFO @metrics_hook: train matching accuracy: 0.987873910370022, N=110588, DB_SIZE=110592, N_IDS=11088 2021-03-14 09:47:40.099: INFO @decay_lr : LR updated to `0.0008778092` 2021-03-14 09:47:40.101: INFO @log_profile : T train: 124.093073 2021-03-14 09:47:40.101: INFO @log_profile : T read data: 2.051373 2021-03-14 09:47:40.101: INFO @log_profile : T hooks: 75.365408 2021-03-14 09:47:40.101: INFO @main_loop : Epoch 26 done 2021-03-14 09:47:40.101: INFO @main_loop : Training epoch 27 2021-03-14 09:49:46.286: INFO @log_variables: train loss nanmean: 0.080392 2021-03-14 09:49:46.286: INFO @log_variables: train positive_distance nanmean: 0.588624 2021-03-14 09:49:46.287: INFO @log_variables: train negative_distance nanmean: 1.419189 2021-03-14 09:49:46.287: INFO @log_variables: train distance nanmean: 0.830564 2021-03-14 09:49:46.287: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:51:01.028: INFO @metrics_hook: train matching accuracy: 0.989022217801369, N=110587, DB_SIZE=110592, N_IDS=11064 2021-03-14 09:51:01.179: INFO @decay_lr : LR updated to `0.00087342015` 2021-03-14 09:51:01.180: INFO @log_profile : T train: 123.717407 2021-03-14 09:51:01.180: INFO @log_profile : T read data: 2.079458 2021-03-14 09:51:01.180: INFO @log_profile : T hooks: 75.230180 2021-03-14 09:51:01.180: INFO @main_loop : Epoch 27 done 2021-03-14 09:51:01.180: INFO @main_loop : Training epoch 28 2021-03-14 09:53:06.042: INFO @log_variables: train loss nanmean: 0.077687 2021-03-14 09:53:06.042: INFO @log_variables: train positive_distance nanmean: 0.587815 2021-03-14 09:53:06.043: INFO @log_variables: train negative_distance nanmean: 1.419201 2021-03-14 09:53:06.043: INFO @log_variables: train distance nanmean: 0.831386 2021-03-14 09:53:06.043: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:54:21.314: INFO @metrics_hook: train matching accuracy: 0.9893477533525641, N=110587, DB_SIZE=110592, N_IDS=11071 2021-03-14 09:54:21.465: INFO @decay_lr : LR updated to `0.00086905307` 2021-03-14 09:54:21.467: INFO @log_profile : T train: 123.699766 2021-03-14 09:54:21.467: INFO @log_profile : T read data: 0.768464 2021-03-14 09:54:21.467: INFO @log_profile : T hooks: 75.767536 2021-03-14 09:54:21.467: INFO @main_loop : Epoch 28 done 2021-03-14 09:54:21.467: INFO @main_loop : Training epoch 29 2021-03-14 09:56:28.091: INFO @log_variables: train loss nanmean: 0.075905 2021-03-14 09:56:28.091: INFO @log_variables: train positive_distance nanmean: 0.587205 2021-03-14 09:56:28.091: INFO @log_variables: train negative_distance nanmean: 1.419181 2021-03-14 09:56:28.091: INFO @log_variables: train distance nanmean: 0.831977 2021-03-14 09:56:28.091: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 09:57:43.348: INFO @metrics_hook: train matching accuracy: 0.9884795543739714, N=110586, DB_SIZE=110592, N_IDS=11109 2021-03-14 09:57:43.497: INFO @decay_lr : LR updated to `0.0008647078` 2021-03-14 09:57:43.498: INFO @log_profile : T train: 124.121317 2021-03-14 09:57:43.498: INFO @log_profile : T read data: 2.116465 2021-03-14 09:57:43.498: INFO @log_profile : T hooks: 75.740451 2021-03-14 09:57:43.498: INFO @main_loop : Epoch 29 done 2021-03-14 09:57:43.499: INFO @main_loop : Training epoch 30 2021-03-14 09:59:49.753: INFO @log_variables: train loss nanmean: 0.074595 2021-03-14 09:59:49.753: INFO @log_variables: train positive_distance nanmean: 0.586083 2021-03-14 09:59:49.753: INFO @log_variables: train negative_distance nanmean: 1.419204 2021-03-14 09:59:49.753: INFO @log_variables: train distance nanmean: 0.833120 2021-03-14 09:59:49.753: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:01:06.460: INFO @metrics_hook: train matching accuracy: 0.9885879640095854, N=110585, DB_SIZE=110592, N_IDS=11065 2021-03-14 10:01:06.617: INFO @decay_lr : LR updated to `0.00086038426` 2021-03-14 10:01:06.618: INFO @log_profile : T train: 123.744362 2021-03-14 10:01:06.618: INFO @log_profile : T read data: 2.112851 2021-03-14 10:01:06.619: INFO @log_profile : T hooks: 77.211215 2021-03-14 10:01:06.619: INFO @main_loop : Epoch 30 done 2021-03-14 10:01:06.619: INFO @main_loop : Training epoch 31 2021-03-14 10:03:12.788: INFO @log_variables: train loss nanmean: 0.071637 2021-03-14 10:03:12.788: INFO @log_variables: train positive_distance nanmean: 0.585173 2021-03-14 10:03:12.788: INFO @log_variables: train negative_distance nanmean: 1.419239 2021-03-14 10:03:12.788: INFO @log_variables: train distance nanmean: 0.834066 2021-03-14 10:03:12.788: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:04:28.456: INFO @metrics_hook: train matching accuracy: 0.9892756060729367, N=110589, DB_SIZE=110592, N_IDS=11098 2021-03-14 10:04:28.619: INFO @decay_lr : LR updated to `0.00085608236` 2021-03-14 10:04:28.620: INFO @log_profile : T train: 123.740287 2021-03-14 10:04:28.620: INFO @log_profile : T read data: 2.029273 2021-03-14 10:04:28.620: INFO @log_profile : T hooks: 76.180473 2021-03-14 10:04:28.620: INFO @main_loop : Epoch 31 done 2021-03-14 10:04:28.620: INFO @main_loop : Training epoch 32 2021-03-14 10:06:34.588: INFO @log_variables: train loss nanmean: 0.070940 2021-03-14 10:06:34.589: INFO @log_variables: train positive_distance nanmean: 0.585523 2021-03-14 10:06:34.589: INFO @log_variables: train negative_distance nanmean: 1.419212 2021-03-14 10:06:34.589: INFO @log_variables: train distance nanmean: 0.833689 2021-03-14 10:06:34.589: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:07:50.242: INFO @metrics_hook: train matching accuracy: 0.989067332212034, N=110586, DB_SIZE=110592, N_IDS=11042 2021-03-14 10:07:50.394: INFO @decay_lr : LR updated to `0.00085180195` 2021-03-14 10:07:50.395: INFO @log_profile : T train: 123.336863 2021-03-14 10:07:50.395: INFO @log_profile : T read data: 2.248769 2021-03-14 10:07:50.395: INFO @log_profile : T hooks: 76.137528 2021-03-14 10:07:50.395: INFO @main_loop : Epoch 32 done 2021-03-14 10:07:50.396: INFO @main_loop : Training epoch 33 2021-03-14 10:09:56.198: INFO @log_variables: train loss nanmean: 0.071067 2021-03-14 10:09:56.198: INFO @log_variables: train positive_distance nanmean: 0.586011 2021-03-14 10:09:56.198: INFO @log_variables: train negative_distance nanmean: 1.419219 2021-03-14 10:09:56.198: INFO @log_variables: train distance nanmean: 0.833208 2021-03-14 10:09:56.198: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:11:11.274: INFO @metrics_hook: train matching accuracy: 0.9888954198128137, N=110585, DB_SIZE=110592, N_IDS=11080 2021-03-14 10:11:11.423: INFO @decay_lr : LR updated to `0.00084754295` 2021-03-14 10:11:11.424: INFO @log_profile : T train: 123.156054 2021-03-14 10:11:11.424: INFO @log_profile : T read data: 2.282239 2021-03-14 10:11:11.424: INFO @log_profile : T hooks: 75.540042 2021-03-14 10:11:11.424: INFO @main_loop : Epoch 33 done 2021-03-14 10:11:11.425: INFO @main_loop : Training epoch 34 2021-03-14 10:13:17.932: INFO @log_variables: train loss nanmean: 0.067019 2021-03-14 10:13:17.932: INFO @log_variables: train positive_distance nanmean: 0.583582 2021-03-14 10:13:17.932: INFO @log_variables: train negative_distance nanmean: 1.419241 2021-03-14 10:13:17.932: INFO @log_variables: train distance nanmean: 0.835659 2021-03-14 10:13:17.933: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:14:33.103: INFO @metrics_hook: train matching accuracy: 0.9898271060151191, N=110588, DB_SIZE=110592, N_IDS=11049 2021-03-14 10:14:33.254: INFO @decay_lr : LR updated to `0.00084330526` 2021-03-14 10:14:33.255: INFO @log_profile : T train: 123.887233 2021-03-14 10:14:33.255: INFO @log_profile : T read data: 2.227971 2021-03-14 10:14:33.255: INFO @log_profile : T hooks: 75.663820 2021-03-14 10:14:33.255: INFO @main_loop : Epoch 34 done 2021-03-14 10:14:33.255: INFO @main_loop : Training epoch 35 2021-03-14 10:16:39.615: INFO @log_variables: train loss nanmean: 0.066026 2021-03-14 10:16:39.615: INFO @log_variables: train positive_distance nanmean: 0.582521 2021-03-14 10:16:39.615: INFO @log_variables: train negative_distance nanmean: 1.419233 2021-03-14 10:16:39.615: INFO @log_variables: train distance nanmean: 0.836712 2021-03-14 10:16:39.615: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:17:55.068: INFO @metrics_hook: train matching accuracy: 0.9893658386609637, N=110587, DB_SIZE=110592, N_IDS=11103 2021-03-14 10:17:55.224: INFO @decay_lr : LR updated to `0.00083908875` 2021-03-14 10:17:55.225: INFO @log_profile : T train: 123.803150 2021-03-14 10:17:55.225: INFO @log_profile : T read data: 2.157639 2021-03-14 10:17:55.225: INFO @log_profile : T hooks: 75.956782 2021-03-14 10:17:55.225: INFO @main_loop : Epoch 35 done 2021-03-14 10:17:55.225: INFO @main_loop : Training epoch 36 2021-03-14 10:20:01.685: INFO @log_variables: train loss nanmean: 0.065805 2021-03-14 10:20:01.686: INFO @log_variables: train positive_distance nanmean: 0.583183 2021-03-14 10:20:01.686: INFO @log_variables: train negative_distance nanmean: 1.419248 2021-03-14 10:20:01.686: INFO @log_variables: train distance nanmean: 0.836065 2021-03-14 10:20:01.686: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:21:17.529: INFO @metrics_hook: train matching accuracy: 0.9895014784739616, N=110587, DB_SIZE=110592, N_IDS=11103 2021-03-14 10:21:17.690: INFO @decay_lr : LR updated to `0.0008348933` 2021-03-14 10:21:17.692: INFO @log_profile : T train: 123.904233 2021-03-14 10:21:17.692: INFO @log_profile : T read data: 2.155302 2021-03-14 10:21:17.692: INFO @log_profile : T hooks: 76.355938 2021-03-14 10:21:17.692: INFO @main_loop : Epoch 36 done 2021-03-14 10:21:17.692: INFO @main_loop : Training epoch 37 2021-03-14 10:23:23.995: INFO @log_variables: train loss nanmean: 0.063476 2021-03-14 10:23:23.996: INFO @log_variables: train positive_distance nanmean: 0.582758 2021-03-14 10:23:23.996: INFO @log_variables: train negative_distance nanmean: 1.419256 2021-03-14 10:23:23.996: INFO @log_variables: train distance nanmean: 0.836498 2021-03-14 10:23:23.996: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:24:39.989: INFO @metrics_hook: train matching accuracy: 0.9895465972184544, N=110586, DB_SIZE=110592, N_IDS=11071 2021-03-14 10:24:40.145: INFO @decay_lr : LR updated to `0.00083071884` 2021-03-14 10:24:40.146: INFO @log_profile : T train: 123.904334 2021-03-14 10:24:40.146: INFO @log_profile : T read data: 2.026104 2021-03-14 10:24:40.146: INFO @log_profile : T hooks: 76.471404 2021-03-14 10:24:40.146: INFO @main_loop : Epoch 37 done 2021-03-14 10:24:40.146: INFO @main_loop : Training epoch 38 2021-03-14 10:26:45.177: INFO @log_variables: train loss nanmean: 0.061814 2021-03-14 10:26:45.177: INFO @log_variables: train positive_distance nanmean: 0.580421 2021-03-14 10:26:45.177: INFO @log_variables: train negative_distance nanmean: 1.419236 2021-03-14 10:26:45.177: INFO @log_variables: train distance nanmean: 0.838815 2021-03-14 10:26:45.177: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:27:59.098: INFO @metrics_hook: train matching accuracy: 0.9905233840923066, N=110588, DB_SIZE=110592, N_IDS=11117 2021-03-14 10:27:59.251: INFO @decay_lr : LR updated to `0.00082656526` 2021-03-14 10:27:59.252: INFO @log_profile : T train: 123.900241 2021-03-14 10:27:59.252: INFO @log_profile : T read data: 0.767054 2021-03-14 10:27:59.252: INFO @log_profile : T hooks: 74.387062 2021-03-14 10:27:59.252: INFO @main_loop : Epoch 38 done 2021-03-14 10:27:59.252: INFO @main_loop : Training epoch 39 2021-03-14 10:30:05.256: INFO @log_variables: train loss nanmean: 0.061207 2021-03-14 10:30:05.256: INFO @log_variables: train positive_distance nanmean: 0.581734 2021-03-14 10:30:05.256: INFO @log_variables: train negative_distance nanmean: 1.419246 2021-03-14 10:30:05.256: INFO @log_variables: train distance nanmean: 0.837512 2021-03-14 10:30:05.257: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:31:20.105: INFO @metrics_hook: train matching accuracy: 0.9894739697783566, N=110583, DB_SIZE=110592, N_IDS=11080 2021-03-14 10:31:20.264: INFO @decay_lr : LR updated to `0.00082243246` 2021-03-14 10:31:20.266: INFO @log_profile : T train: 123.533598 2021-03-14 10:31:20.266: INFO @log_profile : T read data: 2.087656 2021-03-14 10:31:20.266: INFO @log_profile : T hooks: 75.340352 2021-03-14 10:31:20.266: INFO @main_loop : Epoch 39 done 2021-03-14 10:31:20.266: INFO @main_loop : Training epoch 40 2021-03-14 10:33:26.437: INFO @log_variables: train loss nanmean: 0.059587 2021-03-14 10:33:26.437: INFO @log_variables: train positive_distance nanmean: 0.580168 2021-03-14 10:33:26.437: INFO @log_variables: train negative_distance nanmean: 1.419250 2021-03-14 10:33:26.437: INFO @log_variables: train distance nanmean: 0.839082 2021-03-14 10:33:26.437: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:34:42.623: INFO @metrics_hook: train matching accuracy: 0.989808928716757, N=110587, DB_SIZE=110592, N_IDS=11066 2021-03-14 10:34:42.782: INFO @decay_lr : LR updated to `0.0008183203` 2021-03-14 10:34:42.783: INFO @log_profile : T train: 123.729693 2021-03-14 10:34:42.783: INFO @log_profile : T read data: 2.050194 2021-03-14 10:34:42.783: INFO @log_profile : T hooks: 76.686604 2021-03-14 10:34:42.783: INFO @main_loop : Epoch 40 done 2021-03-14 10:34:42.784: INFO @main_loop : Training epoch 41 2021-03-14 10:36:49.411: INFO @log_variables: train loss nanmean: 0.060112 2021-03-14 10:36:49.411: INFO @log_variables: train positive_distance nanmean: 0.580332 2021-03-14 10:36:49.411: INFO @log_variables: train negative_distance nanmean: 1.419261 2021-03-14 10:36:49.411: INFO @log_variables: train distance nanmean: 0.838928 2021-03-14 10:36:49.411: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:38:05.869: INFO @metrics_hook: train matching accuracy: 0.9900170000361703, N=110588, DB_SIZE=110592, N_IDS=11057 2021-03-14 10:38:06.026: INFO @decay_lr : LR updated to `0.0008142287` 2021-03-14 10:38:06.028: INFO @log_profile : T train: 123.939222 2021-03-14 10:38:06.028: INFO @log_profile : T read data: 2.288906 2021-03-14 10:38:06.028: INFO @log_profile : T hooks: 76.964086 2021-03-14 10:38:06.028: INFO @main_loop : Epoch 41 done 2021-03-14 10:38:06.028: INFO @main_loop : Training epoch 42 2021-03-14 10:40:12.717: INFO @log_variables: train loss nanmean: 0.059300 2021-03-14 10:40:12.717: INFO @log_variables: train positive_distance nanmean: 0.579874 2021-03-14 10:40:12.717: INFO @log_variables: train negative_distance nanmean: 1.419267 2021-03-14 10:40:12.717: INFO @log_variables: train distance nanmean: 0.839393 2021-03-14 10:40:12.717: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:41:28.068: INFO @metrics_hook: train matching accuracy: 0.9898541419877562, N=110587, DB_SIZE=110592, N_IDS=11083 2021-03-14 10:41:28.225: INFO @decay_lr : LR updated to `0.00081015757` 2021-03-14 10:41:28.226: INFO @log_profile : T train: 124.115871 2021-03-14 10:41:28.226: INFO @log_profile : T read data: 2.171482 2021-03-14 10:41:28.227: INFO @log_profile : T hooks: 75.859926 2021-03-14 10:41:28.227: INFO @main_loop : Epoch 42 done 2021-03-14 10:41:28.227: INFO @main_loop : Training epoch 43 2021-03-14 10:43:34.024: INFO @log_variables: train loss nanmean: 0.056795 2021-03-14 10:43:34.024: INFO @log_variables: train positive_distance nanmean: 0.578092 2021-03-14 10:43:34.024: INFO @log_variables: train negative_distance nanmean: 1.419265 2021-03-14 10:43:34.024: INFO @log_variables: train distance nanmean: 0.841173 2021-03-14 10:43:34.024: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:44:50.757: INFO @metrics_hook: train matching accuracy: 0.9900712554707564, N=110588, DB_SIZE=110592, N_IDS=11093 2021-03-14 10:44:50.917: INFO @decay_lr : LR updated to `0.0008061068` 2021-03-14 10:44:50.919: INFO @log_profile : T train: 123.210838 2021-03-14 10:44:50.919: INFO @log_profile : T read data: 2.199428 2021-03-14 10:44:50.919: INFO @log_profile : T hooks: 77.231095 2021-03-14 10:44:50.919: INFO @main_loop : Epoch 43 done 2021-03-14 10:44:50.919: INFO @main_loop : Training epoch 44 2021-03-14 10:46:57.412: INFO @log_variables: train loss nanmean: 0.057618 2021-03-14 10:46:57.413: INFO @log_variables: train positive_distance nanmean: 0.579345 2021-03-14 10:46:57.413: INFO @log_variables: train negative_distance nanmean: 1.419272 2021-03-14 10:46:57.413: INFO @log_variables: train distance nanmean: 0.839926 2021-03-14 10:46:57.413: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:48:11.673: INFO @metrics_hook: train matching accuracy: 0.9889950898387695, N=110587, DB_SIZE=110592, N_IDS=11065 2021-03-14 10:48:11.833: INFO @decay_lr : LR updated to `0.00080207625` 2021-03-14 10:48:11.834: INFO @log_profile : T train: 123.942358 2021-03-14 10:48:11.834: INFO @log_profile : T read data: 2.145643 2021-03-14 10:48:11.835: INFO @log_profile : T hooks: 74.775695 2021-03-14 10:48:11.835: INFO @main_loop : Epoch 44 done 2021-03-14 10:48:11.835: INFO @main_loop : Training epoch 45 2021-03-14 10:50:18.311: INFO @log_variables: train loss nanmean: 0.055678 2021-03-14 10:50:18.311: INFO @log_variables: train positive_distance nanmean: 0.578717 2021-03-14 10:50:18.311: INFO @log_variables: train negative_distance nanmean: 1.419301 2021-03-14 10:50:18.311: INFO @log_variables: train distance nanmean: 0.840584 2021-03-14 10:50:18.311: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:51:33.670: INFO @metrics_hook: train matching accuracy: 0.9901523714789529, N=110585, DB_SIZE=110592, N_IDS=11042 2021-03-14 10:51:33.821: INFO @decay_lr : LR updated to `0.00079806586` 2021-03-14 10:51:33.822: INFO @log_profile : T train: 123.922277 2021-03-14 10:51:33.822: INFO @log_profile : T read data: 2.154572 2021-03-14 10:51:33.823: INFO @log_profile : T hooks: 75.859360 2021-03-14 10:51:33.823: INFO @main_loop : Epoch 45 done 2021-03-14 10:51:33.823: INFO @main_loop : Training epoch 46 2021-03-14 10:53:40.305: INFO @log_variables: train loss nanmean: 0.054007 2021-03-14 10:53:40.305: INFO @log_variables: train positive_distance nanmean: 0.577033 2021-03-14 10:53:40.305: INFO @log_variables: train negative_distance nanmean: 1.419295 2021-03-14 10:53:40.305: INFO @log_variables: train distance nanmean: 0.842262 2021-03-14 10:53:40.305: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:54:56.869: INFO @metrics_hook: train matching accuracy: 0.9907494484030817, N=110588, DB_SIZE=110592, N_IDS=11068 2021-03-14 10:54:57.022: INFO @decay_lr : LR updated to `0.00079407555` 2021-03-14 10:54:57.023: INFO @log_profile : T train: 124.064472 2021-03-14 10:54:57.023: INFO @log_profile : T read data: 2.023761 2021-03-14 10:54:57.023: INFO @log_profile : T hooks: 77.061395 2021-03-14 10:54:57.023: INFO @main_loop : Epoch 46 done 2021-03-14 10:54:57.023: INFO @main_loop : Training epoch 47 2021-03-14 10:57:02.061: INFO @log_variables: train loss nanmean: 0.054069 2021-03-14 10:57:02.062: INFO @log_variables: train positive_distance nanmean: 0.576924 2021-03-14 10:57:02.062: INFO @log_variables: train negative_distance nanmean: 1.419292 2021-03-14 10:57:02.062: INFO @log_variables: train distance nanmean: 0.842367 2021-03-14 10:57:02.062: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 10:58:17.117: INFO @metrics_hook: train matching accuracy: 0.9888778370557917, N=110590, DB_SIZE=110592, N_IDS=11058 2021-03-14 10:58:17.273: INFO @decay_lr : LR updated to `0.00079010514` 2021-03-14 10:58:17.275: INFO @log_profile : T train: 123.850416 2021-03-14 10:58:17.275: INFO @log_profile : T read data: 0.781036 2021-03-14 10:58:17.275: INFO @log_profile : T hooks: 75.568567 2021-03-14 10:58:17.275: INFO @main_loop : Epoch 47 done 2021-03-14 10:58:17.275: INFO @main_loop : Training epoch 48 2021-03-14 11:00:23.604: INFO @log_variables: train loss nanmean: 0.052783 2021-03-14 11:00:23.604: INFO @log_variables: train positive_distance nanmean: 0.576924 2021-03-14 11:00:23.604: INFO @log_variables: train negative_distance nanmean: 1.419318 2021-03-14 11:00:23.604: INFO @log_variables: train distance nanmean: 0.842394 2021-03-14 11:00:23.604: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:01:40.753: INFO @metrics_hook: train matching accuracy: 0.9893568922487069, N=110588, DB_SIZE=110592, N_IDS=11093 2021-03-14 11:01:40.916: INFO @decay_lr : LR updated to `0.0007861546` 2021-03-14 11:01:40.917: INFO @log_profile : T train: 123.835214 2021-03-14 11:01:40.917: INFO @log_profile : T read data: 2.107068 2021-03-14 11:01:40.917: INFO @log_profile : T hooks: 77.648873 2021-03-14 11:01:40.917: INFO @main_loop : Epoch 48 done 2021-03-14 11:01:40.917: INFO @main_loop : Training epoch 49 2021-03-14 11:03:47.377: INFO @log_variables: train loss nanmean: 0.052730 2021-03-14 11:03:47.377: INFO @log_variables: train positive_distance nanmean: 0.577103 2021-03-14 11:03:47.377: INFO @log_variables: train negative_distance nanmean: 1.419275 2021-03-14 11:03:47.377: INFO @log_variables: train distance nanmean: 0.842172 2021-03-14 11:03:47.377: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:05:03.856: INFO @metrics_hook: train matching accuracy: 0.9899989148913083, N=110588, DB_SIZE=110592, N_IDS=11089 2021-03-14 11:05:04.016: INFO @decay_lr : LR updated to `0.0007822238` 2021-03-14 11:05:04.018: INFO @log_profile : T train: 123.877537 2021-03-14 11:05:04.018: INFO @log_profile : T read data: 2.188164 2021-03-14 11:05:04.018: INFO @log_profile : T hooks: 76.982693 2021-03-14 11:05:04.018: INFO @main_loop : Epoch 49 done 2021-03-14 11:05:04.018: INFO @main_loop : Training epoch 50 2021-03-14 11:07:10.614: INFO @log_variables: train loss nanmean: 0.051175 2021-03-14 11:07:10.614: INFO @log_variables: train positive_distance nanmean: 0.575294 2021-03-14 11:07:10.614: INFO @log_variables: train negative_distance nanmean: 1.419282 2021-03-14 11:07:10.614: INFO @log_variables: train distance nanmean: 0.843988 2021-03-14 11:07:10.615: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:08:25.287: INFO @metrics_hook: train matching accuracy: 0.9902516661693027, N=110583, DB_SIZE=110592, N_IDS=11093 2021-03-14 11:08:25.443: INFO @decay_lr : LR updated to `0.0007783127` 2021-03-14 11:08:25.443: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:304: The name tf.train.write_graph is deprecated. Please use tf.io.write_graph instead. 2021-03-14 11:08:25.743: INFO @save : Model saved to: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE/model_50.ckpt 2021-03-14 11:08:25.759: INFO @log_profile : T train: 123.999163 2021-03-14 11:08:25.759: INFO @log_profile : T read data: 2.198135 2021-03-14 11:08:25.759: INFO @log_profile : T hooks: 75.492727 2021-03-14 11:08:25.759: INFO @main_loop : Epoch 50 done 2021-03-14 11:08:25.759: INFO @main_loop : Training epoch 51 2021-03-14 11:10:31.424: INFO @log_variables: train loss nanmean: 0.051553 2021-03-14 11:10:31.424: INFO @log_variables: train positive_distance nanmean: 0.575825 2021-03-14 11:10:31.424: INFO @log_variables: train negative_distance nanmean: 1.419302 2021-03-14 11:10:31.424: INFO @log_variables: train distance nanmean: 0.843477 2021-03-14 11:10:31.425: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:11:46.261: INFO @metrics_hook: train matching accuracy: 0.9900531703258943, N=110588, DB_SIZE=110592, N_IDS=11081 2021-03-14 11:11:46.417: INFO @decay_lr : LR updated to `0.0007744211` 2021-03-14 11:11:46.418: INFO @log_profile : T train: 123.266098 2021-03-14 11:11:46.418: INFO @log_profile : T read data: 2.007133 2021-03-14 11:11:46.418: INFO @log_profile : T hooks: 75.335462 2021-03-14 11:11:46.418: INFO @main_loop : Epoch 51 done 2021-03-14 11:11:46.418: INFO @main_loop : Training epoch 52 2021-03-14 11:13:52.349: INFO @log_variables: train loss nanmean: 0.050814 2021-03-14 11:13:52.349: INFO @log_variables: train positive_distance nanmean: 0.574114 2021-03-14 11:13:52.349: INFO @log_variables: train negative_distance nanmean: 1.419295 2021-03-14 11:13:52.349: INFO @log_variables: train distance nanmean: 0.845180 2021-03-14 11:13:52.349: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:15:07.367: INFO @metrics_hook: train matching accuracy: 0.9900169097633538, N=110587, DB_SIZE=110592, N_IDS=11058 2021-03-14 11:15:07.532: INFO @decay_lr : LR updated to `0.000770549` 2021-03-14 11:15:07.533: INFO @log_profile : T train: 123.533965 2021-03-14 11:15:07.534: INFO @log_profile : T read data: 2.008962 2021-03-14 11:15:07.534: INFO @log_profile : T hooks: 75.520837 2021-03-14 11:15:07.534: INFO @main_loop : Epoch 52 done 2021-03-14 11:15:07.534: INFO @main_loop : Training epoch 53 2021-03-14 11:17:13.556: INFO @log_variables: train loss nanmean: 0.049941 2021-03-14 11:17:13.557: INFO @log_variables: train positive_distance nanmean: 0.574951 2021-03-14 11:17:13.557: INFO @log_variables: train negative_distance nanmean: 1.419293 2021-03-14 11:17:13.557: INFO @log_variables: train distance nanmean: 0.844342 2021-03-14 11:17:13.557: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:18:27.730: INFO @metrics_hook: train matching accuracy: 0.9902064512628523, N=110583, DB_SIZE=110592, N_IDS=11081 2021-03-14 11:18:27.881: INFO @decay_lr : LR updated to `0.00076669623` 2021-03-14 11:18:27.882: INFO @log_profile : T train: 123.456402 2021-03-14 11:18:27.883: INFO @log_profile : T read data: 2.179744 2021-03-14 11:18:27.883: INFO @log_profile : T hooks: 74.660841 2021-03-14 11:18:27.883: INFO @main_loop : Epoch 53 done 2021-03-14 11:18:27.883: INFO @main_loop : Training epoch 54 2021-03-14 11:20:34.006: INFO @log_variables: train loss nanmean: 0.048632 2021-03-14 11:20:34.007: INFO @log_variables: train positive_distance nanmean: 0.574950 2021-03-14 11:20:34.007: INFO @log_variables: train negative_distance nanmean: 1.419318 2021-03-14 11:20:34.007: INFO @log_variables: train distance nanmean: 0.844368 2021-03-14 11:20:34.007: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:21:49.884: INFO @metrics_hook: train matching accuracy: 0.9901978514847904, N=110588, DB_SIZE=110592, N_IDS=11082 2021-03-14 11:21:50.042: INFO @decay_lr : LR updated to `0.00076286274` 2021-03-14 11:21:50.043: INFO @log_profile : T train: 123.690157 2021-03-14 11:21:50.044: INFO @log_profile : T read data: 2.050393 2021-03-14 11:21:50.044: INFO @log_profile : T hooks: 76.369968 2021-03-14 11:21:50.044: INFO @main_loop : Epoch 54 done 2021-03-14 11:21:50.044: INFO @main_loop : Training epoch 55 2021-03-14 11:23:56.131: INFO @log_variables: train loss nanmean: 0.047912 2021-03-14 11:23:56.131: INFO @log_variables: train positive_distance nanmean: 0.574100 2021-03-14 11:23:56.131: INFO @log_variables: train negative_distance nanmean: 1.419298 2021-03-14 11:23:56.131: INFO @log_variables: train distance nanmean: 0.845198 2021-03-14 11:23:56.131: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:25:11.321: INFO @metrics_hook: train matching accuracy: 0.9904327853435335, N=110586, DB_SIZE=110592, N_IDS=11062 2021-03-14 11:25:11.481: INFO @decay_lr : LR updated to `0.00075904845` 2021-03-14 11:25:11.483: INFO @log_profile : T train: 123.590853 2021-03-14 11:25:11.483: INFO @log_profile : T read data: 2.100006 2021-03-14 11:25:11.483: INFO @log_profile : T hooks: 75.696416 2021-03-14 11:25:11.483: INFO @main_loop : Epoch 55 done 2021-03-14 11:25:11.483: INFO @main_loop : Training epoch 56 2021-03-14 11:27:16.613: INFO @log_variables: train loss nanmean: 0.047703 2021-03-14 11:27:16.613: INFO @log_variables: train positive_distance nanmean: 0.574141 2021-03-14 11:27:16.613: INFO @log_variables: train negative_distance nanmean: 1.419327 2021-03-14 11:27:16.613: INFO @log_variables: train distance nanmean: 0.845186 2021-03-14 11:27:16.613: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:28:32.556: INFO @metrics_hook: train matching accuracy: 0.9902969688200824, N=110584, DB_SIZE=110592, N_IDS=11088 2021-03-14 11:28:32.719: INFO @decay_lr : LR updated to `0.0007552532` 2021-03-14 11:28:32.720: INFO @log_profile : T train: 123.961439 2021-03-14 11:28:32.720: INFO @log_profile : T read data: 0.771581 2021-03-14 11:28:32.720: INFO @log_profile : T hooks: 76.453185 2021-03-14 11:28:32.720: INFO @main_loop : Epoch 56 done 2021-03-14 11:28:32.720: INFO @main_loop : Training epoch 57 2021-03-14 11:30:39.207: INFO @log_variables: train loss nanmean: 0.046882 2021-03-14 11:30:39.207: INFO @log_variables: train positive_distance nanmean: 0.574283 2021-03-14 11:30:39.207: INFO @log_variables: train negative_distance nanmean: 1.419328 2021-03-14 11:30:39.207: INFO @log_variables: train distance nanmean: 0.845046 2021-03-14 11:30:39.207: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:31:55.196: INFO @metrics_hook: train matching accuracy: 0.9906951929684957, N=110588, DB_SIZE=110592, N_IDS=11068 2021-03-14 11:31:55.357: INFO @decay_lr : LR updated to `0.0007514769` 2021-03-14 11:31:55.358: INFO @log_profile : T train: 123.843936 2021-03-14 11:31:55.358: INFO @log_profile : T read data: 2.250461 2021-03-14 11:31:55.358: INFO @log_profile : T hooks: 76.491388 2021-03-14 11:31:55.358: INFO @main_loop : Epoch 57 done 2021-03-14 11:31:55.358: INFO @main_loop : Training epoch 58 2021-03-14 11:34:02.075: INFO @log_variables: train loss nanmean: 0.045877 2021-03-14 11:34:02.075: INFO @log_variables: train positive_distance nanmean: 0.573848 2021-03-14 11:34:02.075: INFO @log_variables: train negative_distance nanmean: 1.419360 2021-03-14 11:34:02.075: INFO @log_variables: train distance nanmean: 0.845512 2021-03-14 11:34:02.075: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:35:17.501: INFO @metrics_hook: train matching accuracy: 0.9902068055015508, N=110587, DB_SIZE=110592, N_IDS=11068 2021-03-14 11:35:17.662: INFO @decay_lr : LR updated to `0.00074771955` 2021-03-14 11:35:17.663: INFO @log_profile : T train: 124.058495 2021-03-14 11:35:17.663: INFO @log_profile : T read data: 2.259067 2021-03-14 11:35:17.663: INFO @log_profile : T hooks: 75.935405 2021-03-14 11:35:17.663: INFO @main_loop : Epoch 58 done 2021-03-14 11:35:17.663: INFO @main_loop : Training epoch 59 2021-03-14 11:37:24.043: INFO @log_variables: train loss nanmean: 0.046764 2021-03-14 11:37:24.044: INFO @log_variables: train positive_distance nanmean: 0.573300 2021-03-14 11:37:24.044: INFO @log_variables: train negative_distance nanmean: 1.419322 2021-03-14 11:37:24.044: INFO @log_variables: train distance nanmean: 0.846022 2021-03-14 11:37:24.044: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:38:39.188: INFO @metrics_hook: train matching accuracy: 0.9897189670138888, N=110592, DB_SIZE=110592, N_IDS=11075 2021-03-14 11:38:39.349: INFO @decay_lr : LR updated to `0.0007439809` 2021-03-14 11:38:39.350: INFO @log_profile : T train: 123.729248 2021-03-14 11:38:39.350: INFO @log_profile : T read data: 2.247144 2021-03-14 11:38:39.350: INFO @log_profile : T hooks: 75.659493 2021-03-14 11:38:39.350: INFO @main_loop : Epoch 59 done 2021-03-14 11:38:39.350: INFO @main_loop : Training epoch 60 2021-03-14 11:40:45.451: INFO @log_variables: train loss nanmean: 0.045356 2021-03-14 11:40:45.451: INFO @log_variables: train positive_distance nanmean: 0.572487 2021-03-14 11:40:45.451: INFO @log_variables: train negative_distance nanmean: 1.419323 2021-03-14 11:40:45.451: INFO @log_variables: train distance nanmean: 0.846836 2021-03-14 11:40:45.452: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:42:00.364: INFO @metrics_hook: train matching accuracy: 0.9900438576660487, N=110585, DB_SIZE=110592, N_IDS=11099 2021-03-14 11:42:00.525: INFO @decay_lr : LR updated to `0.00074026105` 2021-03-14 11:42:00.526: INFO @log_profile : T train: 123.668470 2021-03-14 11:42:00.526: INFO @log_profile : T read data: 2.036855 2021-03-14 11:42:00.526: INFO @log_profile : T hooks: 75.419441 2021-03-14 11:42:00.526: INFO @main_loop : Epoch 60 done 2021-03-14 11:42:00.526: INFO @main_loop : Training epoch 61 2021-03-14 11:44:06.570: INFO @log_variables: train loss nanmean: 0.044452 2021-03-14 11:44:06.570: INFO @log_variables: train positive_distance nanmean: 0.572578 2021-03-14 11:44:06.570: INFO @log_variables: train negative_distance nanmean: 1.419329 2021-03-14 11:44:06.570: INFO @log_variables: train distance nanmean: 0.846751 2021-03-14 11:44:06.570: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:45:21.007: INFO @metrics_hook: train matching accuracy: 0.9900896094618911, N=110591, DB_SIZE=110592, N_IDS=11106 2021-03-14 11:45:21.167: INFO @decay_lr : LR updated to `0.00073655974` 2021-03-14 11:45:21.168: INFO @log_profile : T train: 123.564265 2021-03-14 11:45:21.168: INFO @log_profile : T read data: 2.085525 2021-03-14 11:45:21.168: INFO @log_profile : T hooks: 74.941777 2021-03-14 11:45:21.168: INFO @main_loop : Epoch 61 done 2021-03-14 11:45:21.168: INFO @main_loop : Training epoch 62 2021-03-14 11:47:27.304: INFO @log_variables: train loss nanmean: 0.044165 2021-03-14 11:47:27.304: INFO @log_variables: train positive_distance nanmean: 0.573244 2021-03-14 11:47:27.304: INFO @log_variables: train negative_distance nanmean: 1.419317 2021-03-14 11:47:27.304: INFO @log_variables: train distance nanmean: 0.846073 2021-03-14 11:47:27.304: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:48:42.204: INFO @metrics_hook: train matching accuracy: 0.9903330530009133, N=110583, DB_SIZE=110592, N_IDS=11084 2021-03-14 11:48:42.360: INFO @decay_lr : LR updated to `0.00073287694` 2021-03-14 11:48:42.362: INFO @log_profile : T train: 123.703365 2021-03-14 11:48:42.362: INFO @log_profile : T read data: 2.065316 2021-03-14 11:48:42.362: INFO @log_profile : T hooks: 75.373852 2021-03-14 11:48:42.362: INFO @main_loop : Epoch 62 done 2021-03-14 11:48:42.362: INFO @main_loop : Training epoch 63 2021-03-14 11:50:48.402: INFO @log_variables: train loss nanmean: 0.044658 2021-03-14 11:50:48.402: INFO @log_variables: train positive_distance nanmean: 0.572659 2021-03-14 11:50:48.402: INFO @log_variables: train negative_distance nanmean: 1.419333 2021-03-14 11:50:48.402: INFO @log_variables: train distance nanmean: 0.846674 2021-03-14 11:50:48.402: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:52:03.111: INFO @metrics_hook: train matching accuracy: 0.9905321698241172, N=110585, DB_SIZE=110592, N_IDS=11070 2021-03-14 11:52:03.284: INFO @decay_lr : LR updated to `0.00072921254` 2021-03-14 11:52:03.285: INFO @log_profile : T train: 123.572767 2021-03-14 11:52:03.285: INFO @log_profile : T read data: 2.074429 2021-03-14 11:52:03.285: INFO @log_profile : T hooks: 75.223829 2021-03-14 11:52:03.285: INFO @main_loop : Epoch 63 done 2021-03-14 11:52:03.285: INFO @main_loop : Training epoch 64 2021-03-14 11:54:09.110: INFO @log_variables: train loss nanmean: 0.043446 2021-03-14 11:54:09.110: INFO @log_variables: train positive_distance nanmean: 0.571611 2021-03-14 11:54:09.110: INFO @log_variables: train negative_distance nanmean: 1.419340 2021-03-14 11:54:09.110: INFO @log_variables: train distance nanmean: 0.847729 2021-03-14 11:54:09.110: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:55:25.093: INFO @metrics_hook: train matching accuracy: 0.9893663194444444, N=110592, DB_SIZE=110592, N_IDS=11068 2021-03-14 11:55:25.259: INFO @decay_lr : LR updated to `0.00072556647` 2021-03-14 11:55:25.260: INFO @log_profile : T train: 123.484596 2021-03-14 11:55:25.260: INFO @log_profile : T read data: 1.950388 2021-03-14 11:55:25.260: INFO @log_profile : T hooks: 76.489740 2021-03-14 11:55:25.260: INFO @main_loop : Epoch 64 done 2021-03-14 11:55:25.260: INFO @main_loop : Training epoch 65 2021-03-14 11:57:31.625: INFO @log_variables: train loss nanmean: 0.043261 2021-03-14 11:57:31.625: INFO @log_variables: train positive_distance nanmean: 0.571380 2021-03-14 11:57:31.625: INFO @log_variables: train negative_distance nanmean: 1.419309 2021-03-14 11:57:31.625: INFO @log_variables: train distance nanmean: 0.847929 2021-03-14 11:57:31.626: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 11:58:45.793: INFO @metrics_hook: train matching accuracy: 0.9902521069193764, N=110588, DB_SIZE=110592, N_IDS=11092 2021-03-14 11:58:45.957: INFO @decay_lr : LR updated to `0.0007219386` 2021-03-14 11:58:45.958: INFO @log_profile : T train: 123.925210 2021-03-14 11:58:45.959: INFO @log_profile : T read data: 2.045842 2021-03-14 11:58:45.959: INFO @log_profile : T hooks: 74.675857 2021-03-14 11:58:45.959: INFO @main_loop : Epoch 65 done 2021-03-14 11:58:45.959: INFO @main_loop : Training epoch 66 2021-03-14 12:00:51.066: INFO @log_variables: train loss nanmean: 0.042416 2021-03-14 12:00:51.066: INFO @log_variables: train positive_distance nanmean: 0.571072 2021-03-14 12:00:51.066: INFO @log_variables: train negative_distance nanmean: 1.419302 2021-03-14 12:00:51.066: INFO @log_variables: train distance nanmean: 0.848230 2021-03-14 12:00:51.066: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:02:07.071: INFO @metrics_hook: train matching accuracy: 0.9909211751833398, N=110587, DB_SIZE=110592, N_IDS=11079 2021-03-14 12:02:07.244: INFO @decay_lr : LR updated to `0.0007183289` 2021-03-14 12:02:07.245: INFO @log_profile : T train: 123.942038 2021-03-14 12:02:07.245: INFO @log_profile : T read data: 0.773156 2021-03-14 12:02:07.245: INFO @log_profile : T hooks: 76.519141 2021-03-14 12:02:07.245: INFO @main_loop : Epoch 66 done 2021-03-14 12:02:07.245: INFO @main_loop : Training epoch 67 2021-03-14 12:04:13.233: INFO @log_variables: train loss nanmean: 0.042847 2021-03-14 12:04:13.234: INFO @log_variables: train positive_distance nanmean: 0.571485 2021-03-14 12:04:13.234: INFO @log_variables: train negative_distance nanmean: 1.419341 2021-03-14 12:04:13.234: INFO @log_variables: train distance nanmean: 0.847857 2021-03-14 12:04:13.234: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:05:27.706: INFO @metrics_hook: train matching accuracy: 0.9907399034200864, N=110582, DB_SIZE=110592, N_IDS=11088 2021-03-14 12:05:27.864: INFO @decay_lr : LR updated to `0.0007147373` 2021-03-14 12:05:27.865: INFO @log_profile : T train: 123.584272 2021-03-14 12:05:27.865: INFO @log_profile : T read data: 2.016770 2021-03-14 12:05:27.865: INFO @log_profile : T hooks: 74.968091 2021-03-14 12:05:27.865: INFO @main_loop : Epoch 67 done 2021-03-14 12:05:27.865: INFO @main_loop : Training epoch 68 2021-03-14 12:07:34.270: INFO @log_variables: train loss nanmean: 0.041404 2021-03-14 12:07:34.271: INFO @log_variables: train positive_distance nanmean: 0.570999 2021-03-14 12:07:34.271: INFO @log_variables: train negative_distance nanmean: 1.419334 2021-03-14 12:07:34.271: INFO @log_variables: train distance nanmean: 0.848335 2021-03-14 12:07:34.271: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:08:49.545: INFO @metrics_hook: train matching accuracy: 0.9904965141828901, N=110591, DB_SIZE=110592, N_IDS=11091 2021-03-14 12:08:49.705: INFO @decay_lr : LR updated to `0.0007111636` 2021-03-14 12:08:49.706: INFO @log_profile : T train: 123.947031 2021-03-14 12:08:49.706: INFO @log_profile : T read data: 2.056517 2021-03-14 12:08:49.707: INFO @log_profile : T hooks: 75.786108 2021-03-14 12:08:49.707: INFO @main_loop : Epoch 68 done 2021-03-14 12:08:49.707: INFO @main_loop : Training epoch 69 2021-03-14 12:10:56.060: INFO @log_variables: train loss nanmean: 0.040875 2021-03-14 12:10:56.060: INFO @log_variables: train positive_distance nanmean: 0.571383 2021-03-14 12:10:56.060: INFO @log_variables: train negative_distance nanmean: 1.419349 2021-03-14 12:10:56.060: INFO @log_variables: train distance nanmean: 0.847966 2021-03-14 12:10:56.060: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:12:10.174: INFO @metrics_hook: train matching accuracy: 0.9903693120162049, N=110584, DB_SIZE=110592, N_IDS=11061 2021-03-14 12:12:10.342: INFO @decay_lr : LR updated to `0.00070760783` 2021-03-14 12:12:10.343: INFO @log_profile : T train: 123.863350 2021-03-14 12:12:10.343: INFO @log_profile : T read data: 2.080425 2021-03-14 12:12:10.343: INFO @log_profile : T hooks: 74.641336 2021-03-14 12:12:10.343: INFO @main_loop : Epoch 69 done 2021-03-14 12:12:10.343: INFO @main_loop : Training epoch 70 2021-03-14 12:14:16.715: INFO @log_variables: train loss nanmean: 0.040287 2021-03-14 12:14:16.716: INFO @log_variables: train positive_distance nanmean: 0.570833 2021-03-14 12:14:16.716: INFO @log_variables: train negative_distance nanmean: 1.419366 2021-03-14 12:14:16.716: INFO @log_variables: train distance nanmean: 0.848533 2021-03-14 12:14:16.716: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:15:31.173: INFO @metrics_hook: train matching accuracy: 0.9907313632582196, N=110588, DB_SIZE=110592, N_IDS=11081 2021-03-14 12:15:31.336: INFO @decay_lr : LR updated to `0.0007040698` 2021-03-14 12:15:31.337: INFO @log_profile : T train: 123.706582 2021-03-14 12:15:31.337: INFO @log_profile : T read data: 2.274079 2021-03-14 12:15:31.337: INFO @log_profile : T hooks: 74.961222 2021-03-14 12:15:31.337: INFO @main_loop : Epoch 70 done 2021-03-14 12:15:31.337: INFO @main_loop : Training epoch 71 2021-03-14 12:17:37.144: INFO @log_variables: train loss nanmean: 0.039807 2021-03-14 12:17:37.144: INFO @log_variables: train positive_distance nanmean: 0.570292 2021-03-14 12:17:37.144: INFO @log_variables: train negative_distance nanmean: 1.419361 2021-03-14 12:17:37.144: INFO @log_variables: train distance nanmean: 0.849069 2021-03-14 12:17:37.145: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:18:53.362: INFO @metrics_hook: train matching accuracy: 0.9907044036531332, N=110590, DB_SIZE=110592, N_IDS=11110 2021-03-14 12:18:53.526: INFO @decay_lr : LR updated to `0.00070054946` 2021-03-14 12:18:53.527: INFO @log_profile : T train: 123.411219 2021-03-14 12:18:53.527: INFO @log_profile : T read data: 2.005588 2021-03-14 12:18:53.527: INFO @log_profile : T hooks: 76.721733 2021-03-14 12:18:53.528: INFO @main_loop : Epoch 71 done 2021-03-14 12:18:53.528: INFO @main_loop : Training epoch 72 2021-03-14 12:20:59.643: INFO @log_variables: train loss nanmean: 0.041047 2021-03-14 12:20:59.644: INFO @log_variables: train positive_distance nanmean: 0.571182 2021-03-14 12:20:59.644: INFO @log_variables: train negative_distance nanmean: 1.419350 2021-03-14 12:20:59.644: INFO @log_variables: train distance nanmean: 0.848168 2021-03-14 12:20:59.644: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:22:15.751: INFO @metrics_hook: train matching accuracy: 0.9902069826112905, N=110589, DB_SIZE=110592, N_IDS=11096 2021-03-14 12:22:15.926: INFO @decay_lr : LR updated to `0.0006970467` 2021-03-14 12:22:15.927: INFO @log_profile : T train: 123.634260 2021-03-14 12:22:15.927: INFO @log_profile : T read data: 2.117995 2021-03-14 12:22:15.927: INFO @log_profile : T hooks: 76.596501 2021-03-14 12:22:15.928: INFO @main_loop : Epoch 72 done 2021-03-14 12:22:15.928: INFO @main_loop : Training epoch 73 2021-03-14 12:24:22.118: INFO @log_variables: train loss nanmean: 0.038752 2021-03-14 12:24:22.118: INFO @log_variables: train positive_distance nanmean: 0.568668 2021-03-14 12:24:22.118: INFO @log_variables: train negative_distance nanmean: 1.419357 2021-03-14 12:24:22.119: INFO @log_variables: train distance nanmean: 0.850689 2021-03-14 12:24:22.119: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:25:37.599: INFO @metrics_hook: train matching accuracy: 0.9910388107208739, N=110588, DB_SIZE=110592, N_IDS=11074 2021-03-14 12:25:37.758: INFO @decay_lr : LR updated to `0.00069356145` 2021-03-14 12:25:37.759: INFO @log_profile : T train: 123.723304 2021-03-14 12:25:37.759: INFO @log_profile : T read data: 2.101028 2021-03-14 12:25:37.759: INFO @log_profile : T hooks: 75.956060 2021-03-14 12:25:37.759: INFO @main_loop : Epoch 73 done 2021-03-14 12:25:37.759: INFO @main_loop : Training epoch 74 2021-03-14 12:27:44.342: INFO @log_variables: train loss nanmean: 0.039684 2021-03-14 12:27:44.342: INFO @log_variables: train positive_distance nanmean: 0.568930 2021-03-14 12:27:44.342: INFO @log_variables: train negative_distance nanmean: 1.419357 2021-03-14 12:27:44.342: INFO @log_variables: train distance nanmean: 0.850427 2021-03-14 12:27:44.342: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:29:00.355: INFO @metrics_hook: train matching accuracy: 0.990125421613752, N=110587, DB_SIZE=110592, N_IDS=11081 2021-03-14 12:29:00.521: INFO @decay_lr : LR updated to `0.0006900937` 2021-03-14 12:29:00.522: INFO @log_profile : T train: 123.988961 2021-03-14 12:29:00.522: INFO @log_profile : T read data: 2.207299 2021-03-14 12:29:00.522: INFO @log_profile : T hooks: 76.515595 2021-03-14 12:29:00.522: INFO @main_loop : Epoch 74 done 2021-03-14 12:29:00.522: INFO @main_loop : Training epoch 75 2021-03-14 12:31:04.963: INFO @log_variables: train loss nanmean: 0.038893 2021-03-14 12:31:04.963: INFO @log_variables: train positive_distance nanmean: 0.570222 2021-03-14 12:31:04.963: INFO @log_variables: train negative_distance nanmean: 1.419349 2021-03-14 12:31:04.963: INFO @log_variables: train distance nanmean: 0.849128 2021-03-14 12:31:04.963: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:32:19.412: INFO @metrics_hook: train matching accuracy: 0.9908307486413412, N=110587, DB_SIZE=110592, N_IDS=11088 2021-03-14 12:32:19.581: INFO @decay_lr : LR updated to `0.0006866432` 2021-03-14 12:32:19.582: INFO @log_profile : T train: 123.289927 2021-03-14 12:32:19.582: INFO @log_profile : T read data: 0.756794 2021-03-14 12:32:19.582: INFO @log_profile : T hooks: 74.962170 2021-03-14 12:32:19.582: INFO @main_loop : Epoch 75 done 2021-03-14 12:32:19.582: INFO @main_loop : Training epoch 76 2021-03-14 12:34:25.572: INFO @log_variables: train loss nanmean: 0.037703 2021-03-14 12:34:25.573: INFO @log_variables: train positive_distance nanmean: 0.568397 2021-03-14 12:34:25.573: INFO @log_variables: train negative_distance nanmean: 1.419382 2021-03-14 12:34:25.573: INFO @log_variables: train distance nanmean: 0.850985 2021-03-14 12:34:25.573: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:35:42.548: INFO @metrics_hook: train matching accuracy: 0.9909302178375397, N=110587, DB_SIZE=110592, N_IDS=11076 2021-03-14 12:35:42.711: INFO @decay_lr : LR updated to `0.00068321` 2021-03-14 12:35:42.713: INFO @log_profile : T train: 123.552989 2021-03-14 12:35:42.713: INFO @log_profile : T read data: 2.041097 2021-03-14 12:35:42.713: INFO @log_profile : T hooks: 77.485887 2021-03-14 12:35:42.713: INFO @main_loop : Epoch 76 done 2021-03-14 12:35:42.713: INFO @main_loop : Training epoch 77 2021-03-14 12:37:49.243: INFO @log_variables: train loss nanmean: 0.038164 2021-03-14 12:37:49.243: INFO @log_variables: train positive_distance nanmean: 0.570140 2021-03-14 12:37:49.243: INFO @log_variables: train negative_distance nanmean: 1.419379 2021-03-14 12:37:49.243: INFO @log_variables: train distance nanmean: 0.849239 2021-03-14 12:37:49.243: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:39:04.379: INFO @metrics_hook: train matching accuracy: 0.9901434177924873, N=110586, DB_SIZE=110592, N_IDS=11068 2021-03-14 12:39:04.544: INFO @decay_lr : LR updated to `0.00067979394` 2021-03-14 12:39:04.546: INFO @log_profile : T train: 123.971087 2021-03-14 12:39:04.546: INFO @log_profile : T read data: 2.161218 2021-03-14 12:39:04.546: INFO @log_profile : T hooks: 75.648831 2021-03-14 12:39:04.546: INFO @main_loop : Epoch 77 done 2021-03-14 12:39:04.546: INFO @main_loop : Training epoch 78 2021-03-14 12:41:10.755: INFO @log_variables: train loss nanmean: 0.037412 2021-03-14 12:41:10.755: INFO @log_variables: train positive_distance nanmean: 0.569239 2021-03-14 12:41:10.756: INFO @log_variables: train negative_distance nanmean: 1.419369 2021-03-14 12:41:10.756: INFO @log_variables: train distance nanmean: 0.850130 2021-03-14 12:41:10.756: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:42:27.857: INFO @metrics_hook: train matching accuracy: 0.990197762847351, N=110587, DB_SIZE=110592, N_IDS=11073 2021-03-14 12:42:28.025: INFO @decay_lr : LR updated to `0.00067639496` 2021-03-14 12:42:28.027: INFO @log_profile : T train: 123.726059 2021-03-14 12:42:28.027: INFO @log_profile : T read data: 2.097493 2021-03-14 12:42:28.027: INFO @log_profile : T hooks: 77.606252 2021-03-14 12:42:28.027: INFO @main_loop : Epoch 78 done 2021-03-14 12:42:28.027: INFO @main_loop : Training epoch 79 2021-03-14 12:44:34.029: INFO @log_variables: train loss nanmean: 0.037660 2021-03-14 12:44:34.029: INFO @log_variables: train positive_distance nanmean: 0.568553 2021-03-14 12:44:34.030: INFO @log_variables: train negative_distance nanmean: 1.419386 2021-03-14 12:44:34.030: INFO @log_variables: train distance nanmean: 0.850834 2021-03-14 12:44:34.030: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:45:49.050: INFO @metrics_hook: train matching accuracy: 0.9898907696759259, N=110592, DB_SIZE=110592, N_IDS=11074 2021-03-14 12:45:49.227: INFO @decay_lr : LR updated to `0.000673013` 2021-03-14 12:45:49.228: INFO @log_profile : T train: 123.381998 2021-03-14 12:45:49.229: INFO @log_profile : T read data: 2.213565 2021-03-14 12:45:49.229: INFO @log_profile : T hooks: 75.555035 2021-03-14 12:45:49.229: INFO @main_loop : Epoch 79 done 2021-03-14 12:45:49.229: INFO @main_loop : Training epoch 80 2021-03-14 12:47:55.601: INFO @log_variables: train loss nanmean: 0.036547 2021-03-14 12:47:55.601: INFO @log_variables: train positive_distance nanmean: 0.567646 2021-03-14 12:47:55.602: INFO @log_variables: train negative_distance nanmean: 1.419371 2021-03-14 12:47:55.602: INFO @log_variables: train distance nanmean: 0.851725 2021-03-14 12:47:55.602: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:49:10.666: INFO @metrics_hook: train matching accuracy: 0.9906406836370213, N=110585, DB_SIZE=110592, N_IDS=11092 2021-03-14 12:49:10.838: INFO @decay_lr : LR updated to `0.00066964794` 2021-03-14 12:49:10.840: INFO @log_profile : T train: 123.882452 2021-03-14 12:49:10.840: INFO @log_profile : T read data: 2.098433 2021-03-14 12:49:10.840: INFO @log_profile : T hooks: 75.579028 2021-03-14 12:49:10.840: INFO @main_loop : Epoch 80 done 2021-03-14 12:49:10.840: INFO @main_loop : Training epoch 81 2021-03-14 12:51:17.416: INFO @log_variables: train loss nanmean: 0.037353 2021-03-14 12:51:17.416: INFO @log_variables: train positive_distance nanmean: 0.568675 2021-03-14 12:51:17.416: INFO @log_variables: train negative_distance nanmean: 1.419370 2021-03-14 12:51:17.416: INFO @log_variables: train distance nanmean: 0.850695 2021-03-14 12:51:17.416: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:52:32.897: INFO @metrics_hook: train matching accuracy: 0.9907221528945798, N=110586, DB_SIZE=110592, N_IDS=11103 2021-03-14 12:52:33.063: INFO @decay_lr : LR updated to `0.0006662997` 2021-03-14 12:52:33.064: INFO @log_profile : T train: 124.095261 2021-03-14 12:52:33.064: INFO @log_profile : T read data: 2.075287 2021-03-14 12:52:33.065: INFO @log_profile : T hooks: 76.002089 2021-03-14 12:52:33.065: INFO @main_loop : Epoch 81 done 2021-03-14 12:52:33.065: INFO @main_loop : Training epoch 82 2021-03-14 12:54:39.033: INFO @log_variables: train loss nanmean: 0.036623 2021-03-14 12:54:39.033: INFO @log_variables: train positive_distance nanmean: 0.567578 2021-03-14 12:54:39.033: INFO @log_variables: train negative_distance nanmean: 1.419387 2021-03-14 12:54:39.033: INFO @log_variables: train distance nanmean: 0.851809 2021-03-14 12:54:39.033: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:55:55.717: INFO @metrics_hook: train matching accuracy: 0.9906049371552582, N=110590, DB_SIZE=110592, N_IDS=11077 2021-03-14 12:55:55.902: INFO @decay_lr : LR updated to `0.0006629682` 2021-03-14 12:55:55.903: INFO @log_profile : T train: 123.529007 2021-03-14 12:55:55.903: INFO @log_profile : T read data: 2.068345 2021-03-14 12:55:55.903: INFO @log_profile : T hooks: 77.190495 2021-03-14 12:55:55.903: INFO @main_loop : Epoch 82 done 2021-03-14 12:55:55.903: INFO @main_loop : Training epoch 83 2021-03-14 12:58:01.469: INFO @log_variables: train loss nanmean: 0.036311 2021-03-14 12:58:01.469: INFO @log_variables: train positive_distance nanmean: 0.567644 2021-03-14 12:58:01.469: INFO @log_variables: train negative_distance nanmean: 1.419379 2021-03-14 12:58:01.469: INFO @log_variables: train distance nanmean: 0.851736 2021-03-14 12:58:01.469: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 12:59:17.361: INFO @metrics_hook: train matching accuracy: 0.9903969617506103, N=110590, DB_SIZE=110592, N_IDS=11063 2021-03-14 12:59:17.537: INFO @decay_lr : LR updated to `0.00065965333` 2021-03-14 12:59:17.539: INFO @log_profile : T train: 123.318188 2021-03-14 12:59:17.539: INFO @log_profile : T read data: 1.868161 2021-03-14 12:59:17.539: INFO @log_profile : T hooks: 76.398200 2021-03-14 12:59:17.539: INFO @main_loop : Epoch 83 done 2021-03-14 12:59:17.539: INFO @main_loop : Training epoch 84 2021-03-14 13:01:22.681: INFO @log_variables: train loss nanmean: 0.035525 2021-03-14 13:01:22.681: INFO @log_variables: train positive_distance nanmean: 0.567095 2021-03-14 13:01:22.681: INFO @log_variables: train negative_distance nanmean: 1.419374 2021-03-14 13:01:22.681: INFO @log_variables: train distance nanmean: 0.852279 2021-03-14 13:01:22.681: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:02:39.401: INFO @metrics_hook: train matching accuracy: 0.9907946612652367, N=110588, DB_SIZE=110592, N_IDS=11083 2021-03-14 13:02:39.577: INFO @decay_lr : LR updated to `0.00065635506` 2021-03-14 13:02:39.579: INFO @log_profile : T train: 123.945290 2021-03-14 13:02:39.579: INFO @log_profile : T read data: 0.775347 2021-03-14 13:02:39.579: INFO @log_profile : T hooks: 77.267366 2021-03-14 13:02:39.579: INFO @main_loop : Epoch 84 done 2021-03-14 13:02:39.579: INFO @main_loop : Training epoch 85 2021-03-14 13:04:46.193: INFO @log_variables: train loss nanmean: 0.035128 2021-03-14 13:04:46.193: INFO @log_variables: train positive_distance nanmean: 0.566368 2021-03-14 13:04:46.194: INFO @log_variables: train negative_distance nanmean: 1.419378 2021-03-14 13:04:46.194: INFO @log_variables: train distance nanmean: 0.853010 2021-03-14 13:04:46.194: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:06:02.624: INFO @metrics_hook: train matching accuracy: 0.9899445685297549, N=110587, DB_SIZE=110592, N_IDS=11089 2021-03-14 13:06:02.795: INFO @decay_lr : LR updated to `0.0006530733` 2021-03-14 13:06:02.796: INFO @log_profile : T train: 124.081211 2021-03-14 13:06:02.796: INFO @log_profile : T read data: 2.132107 2021-03-14 13:06:02.796: INFO @log_profile : T hooks: 76.951115 2021-03-14 13:06:02.796: INFO @main_loop : Epoch 85 done 2021-03-14 13:06:02.796: INFO @main_loop : Training epoch 86 2021-03-14 13:08:09.260: INFO @log_variables: train loss nanmean: 0.035246 2021-03-14 13:08:09.260: INFO @log_variables: train positive_distance nanmean: 0.566868 2021-03-14 13:08:09.260: INFO @log_variables: train negative_distance nanmean: 1.419383 2021-03-14 13:08:09.261: INFO @log_variables: train distance nanmean: 0.852516 2021-03-14 13:08:09.261: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:09:25.310: INFO @metrics_hook: train matching accuracy: 0.9904146998715931, N=110586, DB_SIZE=110592, N_IDS=11083 2021-03-14 13:09:25.480: INFO @decay_lr : LR updated to `0.0006498079` 2021-03-14 13:09:25.481: INFO @log_profile : T train: 123.759809 2021-03-14 13:09:25.481: INFO @log_profile : T read data: 2.311960 2021-03-14 13:09:25.481: INFO @log_profile : T hooks: 76.561444 2021-03-14 13:09:25.481: INFO @main_loop : Epoch 86 done 2021-03-14 13:09:25.481: INFO @main_loop : Training epoch 87 2021-03-14 13:11:31.355: INFO @log_variables: train loss nanmean: 0.035652 2021-03-14 13:11:31.355: INFO @log_variables: train positive_distance nanmean: 0.566780 2021-03-14 13:11:31.355: INFO @log_variables: train negative_distance nanmean: 1.419378 2021-03-14 13:11:31.355: INFO @log_variables: train distance nanmean: 0.852598 2021-03-14 13:11:31.355: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:12:46.610: INFO @metrics_hook: train matching accuracy: 0.9904146998715931, N=110586, DB_SIZE=110592, N_IDS=11077 2021-03-14 13:12:46.780: INFO @decay_lr : LR updated to `0.0006465589` 2021-03-14 13:12:46.781: INFO @log_profile : T train: 123.363770 2021-03-14 13:12:46.781: INFO @log_profile : T read data: 2.143845 2021-03-14 13:12:46.781: INFO @log_profile : T hooks: 75.741748 2021-03-14 13:12:46.781: INFO @main_loop : Epoch 87 done 2021-03-14 13:12:46.781: INFO @main_loop : Training epoch 88 2021-03-14 13:14:53.239: INFO @log_variables: train loss nanmean: 0.034925 2021-03-14 13:14:53.239: INFO @log_variables: train positive_distance nanmean: 0.568042 2021-03-14 13:14:53.239: INFO @log_variables: train negative_distance nanmean: 1.419376 2021-03-14 13:14:53.240: INFO @log_variables: train distance nanmean: 0.851334 2021-03-14 13:14:53.240: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:16:09.400: INFO @metrics_hook: train matching accuracy: 0.9904419145107471, N=110587, DB_SIZE=110592, N_IDS=11066 2021-03-14 13:16:09.568: INFO @decay_lr : LR updated to `0.0006433261` 2021-03-14 13:16:09.569: INFO @log_profile : T train: 124.067521 2021-03-14 13:16:09.569: INFO @log_profile : T read data: 2.010570 2021-03-14 13:16:09.569: INFO @log_profile : T hooks: 76.658848 2021-03-14 13:16:09.569: INFO @main_loop : Epoch 88 done 2021-03-14 13:16:09.569: INFO @main_loop : Training epoch 89 2021-03-14 13:18:16.163: INFO @log_variables: train loss nanmean: 0.034682 2021-03-14 13:18:16.163: INFO @log_variables: train positive_distance nanmean: 0.566781 2021-03-14 13:18:16.163: INFO @log_variables: train negative_distance nanmean: 1.419388 2021-03-14 13:18:16.163: INFO @log_variables: train distance nanmean: 0.852607 2021-03-14 13:18:16.164: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:19:31.634: INFO @metrics_hook: train matching accuracy: 0.9903607049525721, N=110589, DB_SIZE=110592, N_IDS=11078 2021-03-14 13:19:31.803: INFO @decay_lr : LR updated to `0.0006401095` 2021-03-14 13:19:31.805: INFO @log_profile : T train: 124.116864 2021-03-14 13:19:31.805: INFO @log_profile : T read data: 2.082375 2021-03-14 13:19:31.805: INFO @log_profile : T hooks: 75.984608 2021-03-14 13:19:31.805: INFO @main_loop : Epoch 89 done 2021-03-14 13:19:31.805: INFO @main_loop : Training epoch 90 2021-03-14 13:21:38.131: INFO @log_variables: train loss nanmean: 0.034622 2021-03-14 13:21:38.131: INFO @log_variables: train positive_distance nanmean: 0.567048 2021-03-14 13:21:38.131: INFO @log_variables: train negative_distance nanmean: 1.419383 2021-03-14 13:21:38.131: INFO @log_variables: train distance nanmean: 0.852335 2021-03-14 13:21:38.131: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:22:53.415: INFO @metrics_hook: train matching accuracy: 0.9905956396321448, N=110587, DB_SIZE=110592, N_IDS=11065 2021-03-14 13:22:53.583: INFO @decay_lr : LR updated to `0.0006369089` 2021-03-14 13:22:53.585: INFO @log_profile : T train: 123.798219 2021-03-14 13:22:53.585: INFO @log_profile : T read data: 2.154527 2021-03-14 13:22:53.585: INFO @log_profile : T hooks: 75.775518 2021-03-14 13:22:53.585: INFO @main_loop : Epoch 90 done 2021-03-14 13:22:53.585: INFO @main_loop : Training epoch 91 2021-03-14 13:24:59.877: INFO @log_variables: train loss nanmean: 0.033317 2021-03-14 13:24:59.877: INFO @log_variables: train positive_distance nanmean: 0.566169 2021-03-14 13:24:59.877: INFO @log_variables: train negative_distance nanmean: 1.419405 2021-03-14 13:24:59.877: INFO @log_variables: train distance nanmean: 0.853236 2021-03-14 13:24:59.877: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:26:15.620: INFO @metrics_hook: train matching accuracy: 0.989854050241441, N=110586, DB_SIZE=110592, N_IDS=11075 2021-03-14 13:26:15.793: INFO @decay_lr : LR updated to `0.0006337244` 2021-03-14 13:26:15.794: INFO @log_profile : T train: 123.823171 2021-03-14 13:26:15.794: INFO @log_profile : T read data: 2.073163 2021-03-14 13:26:15.794: INFO @log_profile : T hooks: 76.261781 2021-03-14 13:26:15.794: INFO @main_loop : Epoch 91 done 2021-03-14 13:26:15.794: INFO @main_loop : Training epoch 92 2021-03-14 13:28:22.535: INFO @log_variables: train loss nanmean: 0.033585 2021-03-14 13:28:22.535: INFO @log_variables: train positive_distance nanmean: 0.565537 2021-03-14 13:28:22.535: INFO @log_variables: train negative_distance nanmean: 1.419377 2021-03-14 13:28:22.535: INFO @log_variables: train distance nanmean: 0.853840 2021-03-14 13:28:22.535: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:29:38.561: INFO @metrics_hook: train matching accuracy: 0.9904595684650305, N=110582, DB_SIZE=110592, N_IDS=11073 2021-03-14 13:29:38.733: INFO @decay_lr : LR updated to `0.00063055573` 2021-03-14 13:29:38.734: INFO @log_profile : T train: 124.189248 2021-03-14 13:29:38.734: INFO @log_profile : T read data: 2.152163 2021-03-14 13:29:38.734: INFO @log_profile : T hooks: 76.546929 2021-03-14 13:29:38.735: INFO @main_loop : Epoch 92 done 2021-03-14 13:29:38.735: INFO @main_loop : Training epoch 93 2021-03-14 13:31:45.112: INFO @log_variables: train loss nanmean: 0.032498 2021-03-14 13:31:45.112: INFO @log_variables: train positive_distance nanmean: 0.564821 2021-03-14 13:31:45.112: INFO @log_variables: train negative_distance nanmean: 1.419408 2021-03-14 13:31:45.112: INFO @log_variables: train distance nanmean: 0.854586 2021-03-14 13:31:45.112: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:33:00.041: INFO @metrics_hook: train matching accuracy: 0.9905321698241172, N=110585, DB_SIZE=110592, N_IDS=11080 2021-03-14 13:33:00.206: INFO @decay_lr : LR updated to `0.000627403` 2021-03-14 13:33:00.207: INFO @log_profile : T train: 123.822276 2021-03-14 13:33:00.207: INFO @log_profile : T read data: 2.187709 2021-03-14 13:33:00.207: INFO @log_profile : T hooks: 75.409802 2021-03-14 13:33:00.207: INFO @main_loop : Epoch 93 done 2021-03-14 13:33:00.207: INFO @main_loop : Training epoch 94 2021-03-14 13:35:05.640: INFO @log_variables: train loss nanmean: 0.032295 2021-03-14 13:35:05.640: INFO @log_variables: train positive_distance nanmean: 0.565649 2021-03-14 13:35:05.640: INFO @log_variables: train negative_distance nanmean: 1.419381 2021-03-14 13:35:05.640: INFO @log_variables: train distance nanmean: 0.853732 2021-03-14 13:35:05.640: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:36:21.249: INFO @metrics_hook: train matching accuracy: 0.9908398741273917, N=110588, DB_SIZE=110592, N_IDS=11113 2021-03-14 13:36:21.415: INFO @decay_lr : LR updated to `0.00062426593` 2021-03-14 13:36:21.417: INFO @log_profile : T train: 124.264288 2021-03-14 13:36:21.417: INFO @log_profile : T read data: 0.774355 2021-03-14 13:36:21.417: INFO @log_profile : T hooks: 76.120227 2021-03-14 13:36:21.417: INFO @main_loop : Epoch 94 done 2021-03-14 13:36:21.417: INFO @main_loop : Training epoch 95 2021-03-14 13:38:27.704: INFO @log_variables: train loss nanmean: 0.032907 2021-03-14 13:38:27.704: INFO @log_variables: train positive_distance nanmean: 0.565564 2021-03-14 13:38:27.704: INFO @log_variables: train negative_distance nanmean: 1.419404 2021-03-14 13:38:27.704: INFO @log_variables: train distance nanmean: 0.853839 2021-03-14 13:38:27.704: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:39:41.225: INFO @metrics_hook: train matching accuracy: 0.9911381988841365, N=110587, DB_SIZE=110592, N_IDS=11061 2021-03-14 13:39:41.392: INFO @decay_lr : LR updated to `0.0006211446` 2021-03-14 13:39:41.393: INFO @log_profile : T train: 123.744722 2021-03-14 13:39:41.393: INFO @log_profile : T read data: 2.163827 2021-03-14 13:39:41.394: INFO @log_profile : T hooks: 74.017352 2021-03-14 13:39:41.394: INFO @main_loop : Epoch 95 done 2021-03-14 13:39:41.394: INFO @main_loop : Training epoch 96 2021-03-14 13:41:47.647: INFO @log_variables: train loss nanmean: 0.032490 2021-03-14 13:41:47.647: INFO @log_variables: train positive_distance nanmean: 0.564533 2021-03-14 13:41:47.647: INFO @log_variables: train negative_distance nanmean: 1.419414 2021-03-14 13:41:47.647: INFO @log_variables: train distance nanmean: 0.854881 2021-03-14 13:41:47.647: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:43:03.421: INFO @metrics_hook: train matching accuracy: 0.9903695732771484, N=110587, DB_SIZE=110592, N_IDS=11096 2021-03-14 13:43:03.589: INFO @decay_lr : LR updated to `0.0006180389` 2021-03-14 13:43:03.590: INFO @log_profile : T train: 123.841880 2021-03-14 13:43:03.591: INFO @log_profile : T read data: 2.018105 2021-03-14 13:43:03.591: INFO @log_profile : T hooks: 76.286187 2021-03-14 13:43:03.591: INFO @main_loop : Epoch 96 done 2021-03-14 13:43:03.591: INFO @main_loop : Training epoch 97 2021-03-14 13:45:09.622: INFO @log_variables: train loss nanmean: 0.032715 2021-03-14 13:45:09.622: INFO @log_variables: train positive_distance nanmean: 0.566072 2021-03-14 13:45:09.622: INFO @log_variables: train negative_distance nanmean: 1.419392 2021-03-14 13:45:09.623: INFO @log_variables: train distance nanmean: 0.853320 2021-03-14 13:45:09.623: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:46:25.327: INFO @metrics_hook: train matching accuracy: 0.9904690424733468, N=110587, DB_SIZE=110592, N_IDS=11082 2021-03-14 13:46:25.494: INFO @decay_lr : LR updated to `0.0006149487` 2021-03-14 13:46:25.495: INFO @log_profile : T train: 123.531543 2021-03-14 13:46:25.496: INFO @log_profile : T read data: 2.146825 2021-03-14 13:46:25.496: INFO @log_profile : T hooks: 76.176100 2021-03-14 13:46:25.496: INFO @main_loop : Epoch 97 done 2021-03-14 13:46:25.496: INFO @main_loop : Training epoch 98 2021-03-14 13:48:31.643: INFO @log_variables: train loss nanmean: 0.032433 2021-03-14 13:48:31.643: INFO @log_variables: train positive_distance nanmean: 0.565626 2021-03-14 13:48:31.643: INFO @log_variables: train negative_distance nanmean: 1.419401 2021-03-14 13:48:31.643: INFO @log_variables: train distance nanmean: 0.853774 2021-03-14 13:48:31.643: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:49:46.755: INFO @metrics_hook: train matching accuracy: 0.9907491974499254, N=110585, DB_SIZE=110592, N_IDS=11110 2021-03-14 13:49:46.926: INFO @decay_lr : LR updated to `0.0006118739` 2021-03-14 13:49:46.927: INFO @log_profile : T train: 123.637356 2021-03-14 13:49:46.927: INFO @log_profile : T read data: 2.109638 2021-03-14 13:49:46.927: INFO @log_profile : T hooks: 75.633727 2021-03-14 13:49:46.927: INFO @main_loop : Epoch 98 done 2021-03-14 13:49:46.927: INFO @main_loop : Training epoch 99 2021-03-14 13:51:52.953: INFO @log_variables: train loss nanmean: 0.031784 2021-03-14 13:51:52.953: INFO @log_variables: train positive_distance nanmean: 0.564677 2021-03-14 13:51:52.953: INFO @log_variables: train negative_distance nanmean: 1.419413 2021-03-14 13:51:52.953: INFO @log_variables: train distance nanmean: 0.854736 2021-03-14 13:51:52.953: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:53:09.201: INFO @metrics_hook: train matching accuracy: 0.9907857020137627, N=110589, DB_SIZE=110592, N_IDS=11051 2021-03-14 13:53:09.370: INFO @decay_lr : LR updated to `0.00060881453` 2021-03-14 13:53:09.371: INFO @log_profile : T train: 123.490187 2021-03-14 13:53:09.371: INFO @log_profile : T read data: 2.140917 2021-03-14 13:53:09.371: INFO @log_profile : T hooks: 76.761991 2021-03-14 13:53:09.371: INFO @main_loop : Epoch 99 done 2021-03-14 13:53:09.371: INFO @main_loop : Training epoch 100 2021-03-14 13:55:15.737: INFO @log_variables: train loss nanmean: 0.031601 2021-03-14 13:55:15.737: INFO @log_variables: train positive_distance nanmean: 0.564958 2021-03-14 13:55:15.737: INFO @log_variables: train negative_distance nanmean: 1.419421 2021-03-14 13:55:15.737: INFO @log_variables: train distance nanmean: 0.854463 2021-03-14 13:55:15.737: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:56:31.737: INFO @metrics_hook: train matching accuracy: 0.9896823315579589, N=110587, DB_SIZE=110592, N_IDS=11066 2021-03-14 13:56:31.911: INFO @decay_lr : LR updated to `0.00060577044` 2021-03-14 13:56:32.134: INFO @save : Model saved to: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE/model_100.ckpt 2021-03-14 13:56:32.153: INFO @log_profile : T train: 123.835768 2021-03-14 13:56:32.153: INFO @log_profile : T read data: 2.138496 2021-03-14 13:56:32.153: INFO @log_profile : T hooks: 76.756577 2021-03-14 13:56:32.153: INFO @main_loop : Epoch 100 done 2021-03-14 13:56:32.153: INFO @main_loop : Training epoch 101 2021-03-14 13:58:38.445: INFO @log_variables: train loss nanmean: 0.031363 2021-03-14 13:58:38.445: INFO @log_variables: train positive_distance nanmean: 0.565082 2021-03-14 13:58:38.445: INFO @log_variables: train negative_distance nanmean: 1.419389 2021-03-14 13:58:38.445: INFO @log_variables: train distance nanmean: 0.854307 2021-03-14 13:58:38.445: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 13:59:55.054: INFO @metrics_hook: train matching accuracy: 0.9900890717547588, N=110585, DB_SIZE=110592, N_IDS=11087 2021-03-14 13:59:55.224: INFO @decay_lr : LR updated to `0.0006027416` 2021-03-14 13:59:55.225: INFO @log_profile : T train: 123.782242 2021-03-14 13:59:55.225: INFO @log_profile : T read data: 2.117813 2021-03-14 13:59:55.225: INFO @log_profile : T hooks: 77.121795 2021-03-14 13:59:55.225: INFO @main_loop : Epoch 101 done 2021-03-14 13:59:55.226: INFO @main_loop : Training epoch 102 2021-03-14 14:02:01.433: INFO @log_variables: train loss nanmean: 0.031374 2021-03-14 14:02:01.434: INFO @log_variables: train positive_distance nanmean: 0.564505 2021-03-14 14:02:01.434: INFO @log_variables: train negative_distance nanmean: 1.419403 2021-03-14 14:02:01.434: INFO @log_variables: train distance nanmean: 0.854898 2021-03-14 14:02:01.434: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:03:16.840: INFO @metrics_hook: train matching accuracy: 0.990396527557987, N=110585, DB_SIZE=110592, N_IDS=11075 2021-03-14 14:03:17.006: INFO @decay_lr : LR updated to `0.0005997279` 2021-03-14 14:03:17.007: INFO @log_profile : T train: 123.877971 2021-03-14 14:03:17.007: INFO @log_profile : T read data: 1.950941 2021-03-14 14:03:17.007: INFO @log_profile : T hooks: 75.902012 2021-03-14 14:03:17.008: INFO @main_loop : Epoch 102 done 2021-03-14 14:03:17.008: INFO @main_loop : Training epoch 103 2021-03-14 14:05:22.009: INFO @log_variables: train loss nanmean: 0.031352 2021-03-14 14:05:22.009: INFO @log_variables: train positive_distance nanmean: 0.565204 2021-03-14 14:05:22.010: INFO @log_variables: train negative_distance nanmean: 1.419415 2021-03-14 14:05:22.010: INFO @log_variables: train distance nanmean: 0.854210 2021-03-14 14:05:22.010: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:06:38.225: INFO @metrics_hook: train matching accuracy: 0.9902881015680104, N=110586, DB_SIZE=110592, N_IDS=11073 2021-03-14 14:06:38.403: INFO @decay_lr : LR updated to `0.0005967293` 2021-03-14 14:06:38.404: INFO @log_profile : T train: 123.829868 2021-03-14 14:06:38.404: INFO @log_profile : T read data: 0.774998 2021-03-14 14:06:38.404: INFO @log_profile : T hooks: 76.739896 2021-03-14 14:06:38.404: INFO @main_loop : Epoch 103 done 2021-03-14 14:06:38.404: INFO @main_loop : Training epoch 104 2021-03-14 14:08:44.934: INFO @log_variables: train loss nanmean: 0.031179 2021-03-14 14:08:44.934: INFO @log_variables: train positive_distance nanmean: 0.565161 2021-03-14 14:08:44.934: INFO @log_variables: train negative_distance nanmean: 1.419402 2021-03-14 14:08:44.934: INFO @log_variables: train distance nanmean: 0.854241 2021-03-14 14:08:44.934: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:09:58.627: INFO @metrics_hook: train matching accuracy: 0.9897000443106083, N=110583, DB_SIZE=110592, N_IDS=11074 2021-03-14 14:09:58.801: INFO @decay_lr : LR updated to `0.0005937456` 2021-03-14 14:09:58.802: INFO @log_profile : T train: 124.091140 2021-03-14 14:09:58.803: INFO @log_profile : T read data: 2.038019 2021-03-14 14:09:58.803: INFO @log_profile : T hooks: 74.218269 2021-03-14 14:09:58.803: INFO @main_loop : Epoch 104 done 2021-03-14 14:09:58.803: INFO @main_loop : Training epoch 105 2021-03-14 14:12:05.340: INFO @log_variables: train loss nanmean: 0.029808 2021-03-14 14:12:05.341: INFO @log_variables: train positive_distance nanmean: 0.563273 2021-03-14 14:12:05.341: INFO @log_variables: train negative_distance nanmean: 1.419434 2021-03-14 14:12:05.341: INFO @log_variables: train distance nanmean: 0.856161 2021-03-14 14:12:05.341: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:13:21.435: INFO @metrics_hook: train matching accuracy: 0.9909757575209556, N=110591, DB_SIZE=110592, N_IDS=11046 2021-03-14 14:13:21.612: INFO @decay_lr : LR updated to `0.0005907769` 2021-03-14 14:13:21.613: INFO @log_profile : T train: 123.988331 2021-03-14 14:13:21.613: INFO @log_profile : T read data: 2.157874 2021-03-14 14:13:21.613: INFO @log_profile : T hooks: 76.612656 2021-03-14 14:13:21.613: INFO @main_loop : Epoch 105 done 2021-03-14 14:13:21.613: INFO @main_loop : Training epoch 106 2021-03-14 14:15:28.328: INFO @log_variables: train loss nanmean: 0.030744 2021-03-14 14:15:28.328: INFO @log_variables: train positive_distance nanmean: 0.564730 2021-03-14 14:15:28.328: INFO @log_variables: train negative_distance nanmean: 1.419443 2021-03-14 14:15:28.328: INFO @log_variables: train distance nanmean: 0.854713 2021-03-14 14:15:28.328: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:16:44.458: INFO @metrics_hook: train matching accuracy: 0.9905053848031902, N=110589, DB_SIZE=110592, N_IDS=11037 2021-03-14 14:16:44.632: INFO @decay_lr : LR updated to `0.00058782304` 2021-03-14 14:16:44.633: INFO @log_profile : T train: 124.126180 2021-03-14 14:16:44.634: INFO @log_profile : T read data: 2.191831 2021-03-14 14:16:44.634: INFO @log_profile : T hooks: 76.651325 2021-03-14 14:16:44.634: INFO @main_loop : Epoch 106 done 2021-03-14 14:16:44.634: INFO @main_loop : Training epoch 107 2021-03-14 14:18:50.871: INFO @log_variables: train loss nanmean: 0.030327 2021-03-14 14:18:50.871: INFO @log_variables: train positive_distance nanmean: 0.563789 2021-03-14 14:18:50.872: INFO @log_variables: train negative_distance nanmean: 1.419414 2021-03-14 14:18:50.872: INFO @log_variables: train distance nanmean: 0.855625 2021-03-14 14:18:50.872: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:20:07.035: INFO @metrics_hook: train matching accuracy: 0.991219662169494, N=110588, DB_SIZE=110592, N_IDS=11130 2021-03-14 14:20:07.218: INFO @decay_lr : LR updated to `0.0005848839` 2021-03-14 14:20:07.219: INFO @log_profile : T train: 123.774263 2021-03-14 14:20:07.219: INFO @log_profile : T read data: 2.060148 2021-03-14 14:20:07.220: INFO @log_profile : T hooks: 76.700945 2021-03-14 14:20:07.220: INFO @main_loop : Epoch 107 done 2021-03-14 14:20:07.220: INFO @main_loop : Training epoch 108 2021-03-14 14:22:13.533: INFO @log_variables: train loss nanmean: 0.029480 2021-03-14 14:22:13.533: INFO @log_variables: train positive_distance nanmean: 0.563195 2021-03-14 14:22:13.533: INFO @log_variables: train negative_distance nanmean: 1.419396 2021-03-14 14:22:13.533: INFO @log_variables: train distance nanmean: 0.856201 2021-03-14 14:22:13.533: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:23:31.009: INFO @metrics_hook: train matching accuracy: 0.9906679808657437, N=110587, DB_SIZE=110592, N_IDS=11065 2021-03-14 14:23:31.187: INFO @decay_lr : LR updated to `0.0005819595` 2021-03-14 14:23:31.188: INFO @log_profile : T train: 123.713192 2021-03-14 14:23:31.188: INFO @log_profile : T read data: 2.201358 2021-03-14 14:23:31.188: INFO @log_profile : T hooks: 78.002948 2021-03-14 14:23:31.188: INFO @main_loop : Epoch 108 done 2021-03-14 14:23:31.189: INFO @main_loop : Training epoch 109 2021-03-14 14:25:37.862: INFO @log_variables: train loss nanmean: 0.029538 2021-03-14 14:25:37.862: INFO @log_variables: train positive_distance nanmean: 0.562689 2021-03-14 14:25:37.862: INFO @log_variables: train negative_distance nanmean: 1.419441 2021-03-14 14:25:37.862: INFO @log_variables: train distance nanmean: 0.856752 2021-03-14 14:25:37.862: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:26:53.757: INFO @metrics_hook: train matching accuracy: 0.98956487141462, N=110588, DB_SIZE=110592, N_IDS=11065 2021-03-14 14:26:53.936: INFO @decay_lr : LR updated to `0.0005790497` 2021-03-14 14:26:53.937: INFO @log_profile : T train: 124.107947 2021-03-14 14:26:53.937: INFO @log_profile : T read data: 2.181913 2021-03-14 14:26:53.937: INFO @log_profile : T hooks: 76.407643 2021-03-14 14:26:53.937: INFO @main_loop : Epoch 109 done 2021-03-14 14:26:53.937: INFO @main_loop : Training epoch 110 2021-03-14 14:29:00.530: INFO @log_variables: train loss nanmean: 0.028918 2021-03-14 14:29:00.530: INFO @log_variables: train positive_distance nanmean: 0.563166 2021-03-14 14:29:00.530: INFO @log_variables: train negative_distance nanmean: 1.419418 2021-03-14 14:29:00.530: INFO @log_variables: train distance nanmean: 0.856251 2021-03-14 14:29:00.530: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:30:16.354: INFO @metrics_hook: train matching accuracy: 0.9905778099285649, N=110590, DB_SIZE=110592, N_IDS=11047 2021-03-14 14:30:16.526: INFO @decay_lr : LR updated to `0.00057615445` 2021-03-14 14:30:16.527: INFO @log_profile : T train: 124.096382 2021-03-14 14:30:16.527: INFO @log_profile : T read data: 2.096909 2021-03-14 14:30:16.527: INFO @log_profile : T hooks: 76.345203 2021-03-14 14:30:16.527: INFO @main_loop : Epoch 110 done 2021-03-14 14:30:16.527: INFO @main_loop : Training epoch 111 2021-03-14 14:32:22.925: INFO @log_variables: train loss nanmean: 0.028730 2021-03-14 14:32:22.925: INFO @log_variables: train positive_distance nanmean: 0.562238 2021-03-14 14:32:22.925: INFO @log_variables: train negative_distance nanmean: 1.419410 2021-03-14 14:32:22.925: INFO @log_variables: train distance nanmean: 0.857172 2021-03-14 14:32:22.926: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:33:38.536: INFO @metrics_hook: train matching accuracy: 0.990297056562825, N=110585, DB_SIZE=110592, N_IDS=11067 2021-03-14 14:33:38.714: INFO @decay_lr : LR updated to `0.0005732737` 2021-03-14 14:33:38.715: INFO @log_profile : T train: 123.878213 2021-03-14 14:33:38.715: INFO @log_profile : T read data: 2.128540 2021-03-14 14:33:38.715: INFO @log_profile : T hooks: 76.131113 2021-03-14 14:33:38.715: INFO @main_loop : Epoch 111 done 2021-03-14 14:33:38.716: INFO @main_loop : Training epoch 112 2021-03-14 14:35:44.024: INFO @log_variables: train loss nanmean: 0.028080 2021-03-14 14:35:44.024: INFO @log_variables: train positive_distance nanmean: 0.561895 2021-03-14 14:35:44.025: INFO @log_variables: train negative_distance nanmean: 1.419461 2021-03-14 14:35:44.025: INFO @log_variables: train distance nanmean: 0.857566 2021-03-14 14:35:44.025: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:36:59.639: INFO @metrics_hook: train matching accuracy: 0.9907044036531332, N=110590, DB_SIZE=110592, N_IDS=11076 2021-03-14 14:36:59.815: INFO @decay_lr : LR updated to `0.0005704073` 2021-03-14 14:36:59.816: INFO @log_profile : T train: 124.151656 2021-03-14 14:36:59.816: INFO @log_profile : T read data: 0.775083 2021-03-14 14:36:59.816: INFO @log_profile : T hooks: 76.122688 2021-03-14 14:36:59.816: INFO @main_loop : Epoch 112 done 2021-03-14 14:36:59.816: INFO @main_loop : Training epoch 113 2021-03-14 14:39:04.881: INFO @log_variables: train loss nanmean: 0.028543 2021-03-14 14:39:04.882: INFO @log_variables: train positive_distance nanmean: 0.562402 2021-03-14 14:39:04.882: INFO @log_variables: train negative_distance nanmean: 1.419419 2021-03-14 14:39:04.882: INFO @log_variables: train distance nanmean: 0.857016 2021-03-14 14:39:04.882: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:40:20.522: INFO @metrics_hook: train matching accuracy: 0.9903608792758904, N=110591, DB_SIZE=110592, N_IDS=11095 2021-03-14 14:40:20.708: INFO @decay_lr : LR updated to `0.00056755525` 2021-03-14 14:40:20.709: INFO @log_profile : T train: 122.465922 2021-03-14 14:40:20.709: INFO @log_profile : T read data: 2.210055 2021-03-14 14:40:20.709: INFO @log_profile : T hooks: 76.165345 2021-03-14 14:40:20.709: INFO @main_loop : Epoch 113 done 2021-03-14 14:40:20.709: INFO @main_loop : Training epoch 114 2021-03-14 14:42:20.311: INFO @log_variables: train loss nanmean: 0.027957 2021-03-14 14:42:20.311: INFO @log_variables: train positive_distance nanmean: 0.562734 2021-03-14 14:42:20.311: INFO @log_variables: train negative_distance nanmean: 1.419443 2021-03-14 14:42:20.311: INFO @log_variables: train distance nanmean: 0.856709 2021-03-14 14:42:20.311: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:43:35.842: INFO @metrics_hook: train matching accuracy: 0.9906771078236337, N=110588, DB_SIZE=110592, N_IDS=11096 2021-03-14 14:43:36.012: INFO @decay_lr : LR updated to `0.00056471745` 2021-03-14 14:43:36.014: INFO @log_profile : T train: 117.185170 2021-03-14 14:43:36.014: INFO @log_profile : T read data: 2.033739 2021-03-14 14:43:36.014: INFO @log_profile : T hooks: 76.034607 2021-03-14 14:43:36.014: INFO @main_loop : Epoch 114 done 2021-03-14 14:43:36.014: INFO @main_loop : Training epoch 115 2021-03-14 14:45:41.927: INFO @log_variables: train loss nanmean: 0.028597 2021-03-14 14:45:41.927: INFO @log_variables: train positive_distance nanmean: 0.562534 2021-03-14 14:45:41.927: INFO @log_variables: train negative_distance nanmean: 1.419434 2021-03-14 14:45:41.927: INFO @log_variables: train distance nanmean: 0.856900 2021-03-14 14:45:41.927: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:46:57.881: INFO @metrics_hook: train matching accuracy: 0.9902064512628523, N=110583, DB_SIZE=110592, N_IDS=11111 2021-03-14 14:46:58.052: INFO @decay_lr : LR updated to `0.00056189386` 2021-03-14 14:46:58.053: INFO @log_profile : T train: 123.374900 2021-03-14 14:46:58.053: INFO @log_profile : T read data: 2.142667 2021-03-14 14:46:58.053: INFO @log_profile : T hooks: 76.471423 2021-03-14 14:46:58.053: INFO @main_loop : Epoch 115 done 2021-03-14 14:46:58.053: INFO @main_loop : Training epoch 116 2021-03-14 14:49:04.229: INFO @log_variables: train loss nanmean: 0.028656 2021-03-14 14:49:04.229: INFO @log_variables: train positive_distance nanmean: 0.563316 2021-03-14 14:49:04.229: INFO @log_variables: train negative_distance nanmean: 1.419414 2021-03-14 14:49:04.229: INFO @log_variables: train distance nanmean: 0.856098 2021-03-14 14:49:04.230: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:50:19.685: INFO @metrics_hook: train matching accuracy: 0.9906318102489443, N=110587, DB_SIZE=110592, N_IDS=11093 2021-03-14 14:50:19.863: INFO @decay_lr : LR updated to `0.0005590844` 2021-03-14 14:50:19.865: INFO @log_profile : T train: 123.696965 2021-03-14 14:50:19.865: INFO @log_profile : T read data: 2.086714 2021-03-14 14:50:19.865: INFO @log_profile : T hooks: 75.977312 2021-03-14 14:50:19.865: INFO @main_loop : Epoch 116 done 2021-03-14 14:50:19.865: INFO @main_loop : Training epoch 117 2021-03-14 14:52:26.046: INFO @log_variables: train loss nanmean: 0.027997 2021-03-14 14:52:26.046: INFO @log_variables: train positive_distance nanmean: 0.561422 2021-03-14 14:52:26.046: INFO @log_variables: train negative_distance nanmean: 1.419420 2021-03-14 14:52:26.046: INFO @log_variables: train distance nanmean: 0.857999 2021-03-14 14:52:26.046: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:53:42.035: INFO @metrics_hook: train matching accuracy: 0.9904867836828446, N=110583, DB_SIZE=110592, N_IDS=11106 2021-03-14 14:53:42.207: INFO @decay_lr : LR updated to `0.000556289` 2021-03-14 14:53:42.209: INFO @log_profile : T train: 123.704768 2021-03-14 14:53:42.209: INFO @log_profile : T read data: 2.096928 2021-03-14 14:53:42.209: INFO @log_profile : T hooks: 76.492377 2021-03-14 14:53:42.209: INFO @main_loop : Epoch 117 done 2021-03-14 14:53:42.209: INFO @main_loop : Training epoch 118 2021-03-14 14:55:48.758: INFO @log_variables: train loss nanmean: 0.028756 2021-03-14 14:55:48.759: INFO @log_variables: train positive_distance nanmean: 0.561371 2021-03-14 14:55:48.759: INFO @log_variables: train negative_distance nanmean: 1.419421 2021-03-14 14:55:48.759: INFO @log_variables: train distance nanmean: 0.858050 2021-03-14 14:55:48.759: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 14:57:03.917: INFO @metrics_hook: train matching accuracy: 0.9909032543923898, N=110589, DB_SIZE=110592, N_IDS=11079 2021-03-14 14:57:04.089: INFO @decay_lr : LR updated to `0.00055350753` 2021-03-14 14:57:04.090: INFO @log_profile : T train: 124.091903 2021-03-14 14:57:04.090: INFO @log_profile : T read data: 2.069554 2021-03-14 14:57:04.090: INFO @log_profile : T hooks: 75.668662 2021-03-14 14:57:04.090: INFO @main_loop : Epoch 118 done 2021-03-14 14:57:04.090: INFO @main_loop : Training epoch 119 2021-03-14 14:59:05.153: INFO @log_variables: train loss nanmean: 0.027423 2021-03-14 14:59:05.153: INFO @log_variables: train positive_distance nanmean: 0.561814 2021-03-14 14:59:05.153: INFO @log_variables: train negative_distance nanmean: 1.419444 2021-03-14 14:59:05.153: INFO @log_variables: train distance nanmean: 0.857630 2021-03-14 14:59:05.153: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:00:19.808: INFO @metrics_hook: train matching accuracy: 0.9907044036531332, N=110590, DB_SIZE=110592, N_IDS=11050 2021-03-14 15:00:19.981: INFO @decay_lr : LR updated to `0.00055074` 2021-03-14 15:00:19.982: INFO @log_profile : T train: 118.505710 2021-03-14 15:00:19.982: INFO @log_profile : T read data: 2.166314 2021-03-14 15:00:19.982: INFO @log_profile : T hooks: 75.169038 2021-03-14 15:00:19.982: INFO @main_loop : Epoch 119 done 2021-03-14 15:00:19.982: INFO @main_loop : Training epoch 120 2021-03-14 15:02:25.877: INFO @log_variables: train loss nanmean: 0.027860 2021-03-14 15:02:25.877: INFO @log_variables: train positive_distance nanmean: 0.563034 2021-03-14 15:02:25.877: INFO @log_variables: train negative_distance nanmean: 1.419436 2021-03-14 15:02:25.877: INFO @log_variables: train distance nanmean: 0.856402 2021-03-14 15:02:25.878: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:03:42.406: INFO @metrics_hook: train matching accuracy: 0.9906408529031441, N=110587, DB_SIZE=110592, N_IDS=11073 2021-03-14 15:03:42.577: INFO @decay_lr : LR updated to `0.0005479863` 2021-03-14 15:03:42.578: INFO @log_profile : T train: 123.362436 2021-03-14 15:03:42.578: INFO @log_profile : T read data: 2.144133 2021-03-14 15:03:42.578: INFO @log_profile : T hooks: 77.037297 2021-03-14 15:03:42.578: INFO @main_loop : Epoch 120 done 2021-03-14 15:03:42.579: INFO @main_loop : Training epoch 121 2021-03-14 15:05:47.754: INFO @log_variables: train loss nanmean: 0.026877 2021-03-14 15:05:47.754: INFO @log_variables: train positive_distance nanmean: 0.561233 2021-03-14 15:05:47.754: INFO @log_variables: train negative_distance nanmean: 1.419446 2021-03-14 15:05:47.755: INFO @log_variables: train distance nanmean: 0.858213 2021-03-14 15:05:47.755: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:07:02.615: INFO @metrics_hook: train matching accuracy: 0.9911835716029623, N=110589, DB_SIZE=110592, N_IDS=11088 2021-03-14 15:07:02.790: INFO @decay_lr : LR updated to `0.00054524635` 2021-03-14 15:07:02.791: INFO @log_profile : T train: 124.007897 2021-03-14 15:07:02.791: INFO @log_profile : T read data: 0.772755 2021-03-14 15:07:02.791: INFO @log_profile : T hooks: 75.381003 2021-03-14 15:07:02.791: INFO @main_loop : Epoch 121 done 2021-03-14 15:07:02.791: INFO @main_loop : Training epoch 122 2021-03-14 15:09:09.265: INFO @log_variables: train loss nanmean: 0.027446 2021-03-14 15:09:09.266: INFO @log_variables: train positive_distance nanmean: 0.560565 2021-03-14 15:09:09.266: INFO @log_variables: train negative_distance nanmean: 1.419462 2021-03-14 15:09:09.266: INFO @log_variables: train distance nanmean: 0.858897 2021-03-14 15:09:09.266: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:10:23.892: INFO @metrics_hook: train matching accuracy: 0.9904417416466971, N=110585, DB_SIZE=110592, N_IDS=11065 2021-03-14 15:10:24.068: INFO @decay_lr : LR updated to `0.00054252014` 2021-03-14 15:10:24.070: INFO @log_profile : T train: 124.037844 2021-03-14 15:10:24.070: INFO @log_profile : T read data: 2.072933 2021-03-14 15:10:24.070: INFO @log_profile : T hooks: 75.116337 2021-03-14 15:10:24.070: INFO @main_loop : Epoch 122 done 2021-03-14 15:10:24.070: INFO @main_loop : Training epoch 123 2021-03-14 15:12:22.377: INFO @log_variables: train loss nanmean: 0.026593 2021-03-14 15:12:22.378: INFO @log_variables: train positive_distance nanmean: 0.561055 2021-03-14 15:12:22.378: INFO @log_variables: train negative_distance nanmean: 1.419437 2021-03-14 15:12:22.378: INFO @log_variables: train distance nanmean: 0.858382 2021-03-14 15:12:22.378: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:13:38.012: INFO @metrics_hook: train matching accuracy: 0.9902063626991735, N=110582, DB_SIZE=110592, N_IDS=11092 2021-03-14 15:13:38.188: INFO @decay_lr : LR updated to `0.00053980754` 2021-03-14 15:13:38.189: INFO @log_profile : T train: 115.866690 2021-03-14 15:13:38.189: INFO @log_profile : T read data: 2.052357 2021-03-14 15:13:38.189: INFO @log_profile : T hooks: 76.149956 2021-03-14 15:13:38.189: INFO @main_loop : Epoch 123 done 2021-03-14 15:13:38.189: INFO @main_loop : Training epoch 124 2021-03-14 15:16:20.284: INFO @log_variables: train loss nanmean: 0.026216 2021-03-14 15:16:20.284: INFO @log_variables: train positive_distance nanmean: 0.561293 2021-03-14 15:16:20.284: INFO @log_variables: train negative_distance nanmean: 1.419446 2021-03-14 15:16:20.284: INFO @log_variables: train distance nanmean: 0.858153 2021-03-14 15:16:20.284: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:17:37.684: INFO @metrics_hook: train matching accuracy: 0.9909571822579916, N=110585, DB_SIZE=110592, N_IDS=11072 2021-03-14 15:17:37.870: INFO @decay_lr : LR updated to `0.0005371085` 2021-03-14 15:17:37.871: INFO @log_profile : T train: 159.647855 2021-03-14 15:17:37.871: INFO @log_profile : T read data: 2.082314 2021-03-14 15:17:37.871: INFO @log_profile : T hooks: 77.902101 2021-03-14 15:17:37.871: INFO @main_loop : Epoch 124 done 2021-03-14 15:17:37.871: INFO @main_loop : Training epoch 125 2021-03-14 15:20:00.658: INFO @log_variables: train loss nanmean: 0.027449 2021-03-14 15:20:00.659: INFO @log_variables: train positive_distance nanmean: 0.560808 2021-03-14 15:20:00.659: INFO @log_variables: train negative_distance nanmean: 1.419463 2021-03-14 15:20:00.659: INFO @log_variables: train distance nanmean: 0.858656 2021-03-14 15:20:00.659: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:21:17.148: INFO @metrics_hook: train matching accuracy: 0.9903693991047611, N=110585, DB_SIZE=110592, N_IDS=11081 2021-03-14 15:21:17.340: INFO @decay_lr : LR updated to `0.000534423` 2021-03-14 15:21:17.341: INFO @log_profile : T train: 140.381352 2021-03-14 15:21:17.341: INFO @log_profile : T read data: 2.068262 2021-03-14 15:21:17.341: INFO @log_profile : T hooks: 76.971428 2021-03-14 15:21:17.341: INFO @main_loop : Epoch 125 done 2021-03-14 15:21:17.342: INFO @main_loop : Training epoch 126 2021-03-14 15:23:25.271: INFO @log_variables: train loss nanmean: 0.027027 2021-03-14 15:23:25.271: INFO @log_variables: train positive_distance nanmean: 0.561372 2021-03-14 15:23:25.271: INFO @log_variables: train negative_distance nanmean: 1.419455 2021-03-14 15:23:25.271: INFO @log_variables: train distance nanmean: 0.858083 2021-03-14 15:23:25.271: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:24:40.536: INFO @metrics_hook: train matching accuracy: 0.9902612375552722, N=110589, DB_SIZE=110592, N_IDS=11077 2021-03-14 15:24:40.724: INFO @decay_lr : LR updated to `0.00053175085` 2021-03-14 15:24:40.725: INFO @log_profile : T train: 125.478269 2021-03-14 15:24:40.725: INFO @log_profile : T read data: 2.103087 2021-03-14 15:24:40.726: INFO @log_profile : T hooks: 75.753664 2021-03-14 15:24:40.726: INFO @main_loop : Epoch 126 done 2021-03-14 15:24:40.726: INFO @main_loop : Training epoch 127 2021-03-14 15:26:47.953: INFO @log_variables: train loss nanmean: 0.026632 2021-03-14 15:26:47.954: INFO @log_variables: train positive_distance nanmean: 0.561387 2021-03-14 15:26:47.954: INFO @log_variables: train negative_distance nanmean: 1.419438 2021-03-14 15:26:47.954: INFO @log_variables: train distance nanmean: 0.858051 2021-03-14 15:26:47.954: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:28:04.424: INFO @metrics_hook: train matching accuracy: 0.9904420009404276, N=110588, DB_SIZE=110592, N_IDS=11033 2021-03-14 15:28:04.605: INFO @decay_lr : LR updated to `0.0005290921` 2021-03-14 15:28:04.607: INFO @log_profile : T train: 124.792285 2021-03-14 15:28:04.607: INFO @log_profile : T read data: 2.037308 2021-03-14 15:28:04.607: INFO @log_profile : T hooks: 77.000847 2021-03-14 15:28:04.607: INFO @main_loop : Epoch 127 done 2021-03-14 15:28:04.607: INFO @main_loop : Training epoch 128 2021-03-14 15:30:03.125: INFO @log_variables: train loss nanmean: 0.027392 2021-03-14 15:30:03.125: INFO @log_variables: train positive_distance nanmean: 0.560724 2021-03-14 15:30:03.125: INFO @log_variables: train negative_distance nanmean: 1.419434 2021-03-14 15:30:03.125: INFO @log_variables: train distance nanmean: 0.858710 2021-03-14 15:30:03.126: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:31:19.017: INFO @metrics_hook: train matching accuracy: 0.9902968810757531, N=110583, DB_SIZE=110592, N_IDS=11055 2021-03-14 15:31:19.198: INFO @decay_lr : LR updated to `0.0005264466` 2021-03-14 15:31:19.199: INFO @log_profile : T train: 116.121748 2021-03-14 15:31:19.199: INFO @log_profile : T read data: 2.014400 2021-03-14 15:31:19.199: INFO @log_profile : T hooks: 76.405848 2021-03-14 15:31:19.199: INFO @main_loop : Epoch 128 done 2021-03-14 15:31:19.199: INFO @main_loop : Training epoch 129 2021-03-14 15:33:17.651: INFO @log_variables: train loss nanmean: 0.025820 2021-03-14 15:33:17.652: INFO @log_variables: train positive_distance nanmean: 0.561070 2021-03-14 15:33:17.652: INFO @log_variables: train negative_distance nanmean: 1.419463 2021-03-14 15:33:17.652: INFO @log_variables: train distance nanmean: 0.858393 2021-03-14 15:33:17.652: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:34:33.620: INFO @metrics_hook: train matching accuracy: 0.9903875716636825, N=110586, DB_SIZE=110592, N_IDS=11102 2021-03-14 15:34:33.804: INFO @decay_lr : LR updated to `0.0005238144` 2021-03-14 15:34:33.805: INFO @log_profile : T train: 115.989107 2021-03-14 15:34:33.805: INFO @log_profile : T read data: 2.076216 2021-03-14 15:34:33.805: INFO @log_profile : T hooks: 76.489453 2021-03-14 15:34:33.805: INFO @main_loop : Epoch 129 done 2021-03-14 15:34:33.805: INFO @main_loop : Training epoch 130 2021-03-14 15:36:32.139: INFO @log_variables: train loss nanmean: 0.026077 2021-03-14 15:36:32.139: INFO @log_variables: train positive_distance nanmean: 0.560591 2021-03-14 15:36:32.139: INFO @log_variables: train negative_distance nanmean: 1.419439 2021-03-14 15:36:32.139: INFO @log_variables: train distance nanmean: 0.858848 2021-03-14 15:36:32.139: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:37:47.935: INFO @metrics_hook: train matching accuracy: 0.9907398196796918, N=110581, DB_SIZE=110592, N_IDS=11065 2021-03-14 15:37:48.107: INFO @decay_lr : LR updated to `0.00052119535` 2021-03-14 15:37:48.108: INFO @log_profile : T train: 115.857477 2021-03-14 15:37:48.108: INFO @log_profile : T read data: 2.098608 2021-03-14 15:37:48.108: INFO @log_profile : T hooks: 76.296087 2021-03-14 15:37:48.108: INFO @main_loop : Epoch 130 done 2021-03-14 15:37:48.108: INFO @main_loop : Training epoch 131 2021-03-14 15:39:45.050: INFO @log_variables: train loss nanmean: 0.026277 2021-03-14 15:39:45.051: INFO @log_variables: train positive_distance nanmean: 0.561111 2021-03-14 15:39:45.051: INFO @log_variables: train negative_distance nanmean: 1.419463 2021-03-14 15:39:45.051: INFO @log_variables: train distance nanmean: 0.858352 2021-03-14 15:39:45.051: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:41:00.536: INFO @metrics_hook: train matching accuracy: 0.9909118859871229, N=110584, DB_SIZE=110592, N_IDS=11089 2021-03-14 15:41:00.719: INFO @decay_lr : LR updated to `0.0005185894` 2021-03-14 15:41:00.721: INFO @log_profile : T train: 115.816008 2021-03-14 15:41:00.721: INFO @log_profile : T read data: 0.762920 2021-03-14 15:41:00.721: INFO @log_profile : T hooks: 75.983099 2021-03-14 15:41:00.722: INFO @main_loop : Epoch 131 done 2021-03-14 15:41:00.722: INFO @main_loop : Training epoch 132 2021-03-14 15:43:00.131: INFO @log_variables: train loss nanmean: 0.025932 2021-03-14 15:43:00.131: INFO @log_variables: train positive_distance nanmean: 0.561086 2021-03-14 15:43:00.131: INFO @log_variables: train negative_distance nanmean: 1.419454 2021-03-14 15:43:00.131: INFO @log_variables: train distance nanmean: 0.858368 2021-03-14 15:43:00.131: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:44:16.251: INFO @metrics_hook: train matching accuracy: 0.9908580419390717, N=110589, DB_SIZE=110592, N_IDS=11073 2021-03-14 15:44:16.436: INFO @decay_lr : LR updated to `0.0005159965` 2021-03-14 15:44:16.438: INFO @log_profile : T train: 116.963996 2021-03-14 15:44:16.438: INFO @log_profile : T read data: 2.054867 2021-03-14 15:44:16.438: INFO @log_profile : T hooks: 76.647106 2021-03-14 15:44:16.438: INFO @main_loop : Epoch 132 done 2021-03-14 15:44:16.438: INFO @main_loop : Training epoch 133 2021-03-14 15:46:16.501: INFO @log_variables: train loss nanmean: 0.025821 2021-03-14 15:46:16.501: INFO @log_variables: train positive_distance nanmean: 0.560842 2021-03-14 15:46:16.501: INFO @log_variables: train negative_distance nanmean: 1.419469 2021-03-14 15:46:16.501: INFO @log_variables: train distance nanmean: 0.858626 2021-03-14 15:46:16.501: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:47:32.259: INFO @metrics_hook: train matching accuracy: 0.990514255744346, N=110587, DB_SIZE=110592, N_IDS=11084 2021-03-14 15:47:32.448: INFO @decay_lr : LR updated to `0.0005134165` 2021-03-14 15:47:32.449: INFO @log_profile : T train: 117.638633 2021-03-14 15:47:32.449: INFO @log_profile : T read data: 2.042607 2021-03-14 15:47:32.449: INFO @log_profile : T hooks: 76.280463 2021-03-14 15:47:32.449: INFO @main_loop : Epoch 133 done 2021-03-14 15:47:32.449: INFO @main_loop : Training epoch 134 2021-03-14 15:49:32.739: INFO @log_variables: train loss nanmean: 0.025525 2021-03-14 15:49:32.739: INFO @log_variables: train positive_distance nanmean: 0.559963 2021-03-14 15:49:32.739: INFO @log_variables: train negative_distance nanmean: 1.419464 2021-03-14 15:49:32.739: INFO @log_variables: train distance nanmean: 0.859501 2021-03-14 15:49:32.739: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:50:49.909: INFO @metrics_hook: train matching accuracy: 0.9902612375552722, N=110589, DB_SIZE=110592, N_IDS=11080 2021-03-14 15:50:50.095: INFO @decay_lr : LR updated to `0.0005108494` 2021-03-14 15:50:50.096: INFO @log_profile : T train: 117.839834 2021-03-14 15:50:50.097: INFO @log_profile : T read data: 2.057086 2021-03-14 15:50:50.097: INFO @log_profile : T hooks: 77.700141 2021-03-14 15:50:50.097: INFO @main_loop : Epoch 134 done 2021-03-14 15:50:50.097: INFO @main_loop : Training epoch 135 2021-03-14 15:52:55.395: INFO @log_variables: train loss nanmean: 0.026069 2021-03-14 15:52:55.395: INFO @log_variables: train positive_distance nanmean: 0.560441 2021-03-14 15:52:55.395: INFO @log_variables: train negative_distance nanmean: 1.419446 2021-03-14 15:52:55.395: INFO @log_variables: train distance nanmean: 0.859006 2021-03-14 15:52:55.395: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:54:11.927: INFO @metrics_hook: train matching accuracy: 0.989854050241441, N=110586, DB_SIZE=110592, N_IDS=11070 2021-03-14 15:54:12.116: INFO @decay_lr : LR updated to `0.00050829514` 2021-03-14 15:54:12.117: INFO @log_profile : T train: 122.775039 2021-03-14 15:54:12.117: INFO @log_profile : T read data: 2.127273 2021-03-14 15:54:12.117: INFO @log_profile : T hooks: 77.067968 2021-03-14 15:54:12.117: INFO @main_loop : Epoch 135 done 2021-03-14 15:54:12.117: INFO @main_loop : Training epoch 136 2021-03-14 15:56:10.837: INFO @log_variables: train loss nanmean: 0.025022 2021-03-14 15:56:10.837: INFO @log_variables: train positive_distance nanmean: 0.559117 2021-03-14 15:56:10.837: INFO @log_variables: train negative_distance nanmean: 1.419466 2021-03-14 15:56:10.837: INFO @log_variables: train distance nanmean: 0.860349 2021-03-14 15:56:10.838: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 15:57:25.536: INFO @metrics_hook: train matching accuracy: 0.9906767705997251, N=110584, DB_SIZE=110592, N_IDS=11045 2021-03-14 15:57:25.719: INFO @decay_lr : LR updated to `0.0005057537` 2021-03-14 15:57:25.720: INFO @log_profile : T train: 116.313957 2021-03-14 15:57:25.720: INFO @log_profile : T read data: 2.018613 2021-03-14 15:57:25.720: INFO @log_profile : T hooks: 75.219365 2021-03-14 15:57:25.720: INFO @main_loop : Epoch 136 done 2021-03-14 15:57:25.720: INFO @main_loop : Training epoch 137 2021-03-14 15:59:25.343: INFO @log_variables: train loss nanmean: 0.024578 2021-03-14 15:59:25.343: INFO @log_variables: train positive_distance nanmean: 0.560896 2021-03-14 15:59:25.343: INFO @log_variables: train negative_distance nanmean: 1.419474 2021-03-14 15:59:25.343: INFO @log_variables: train distance nanmean: 0.858578 2021-03-14 15:59:25.343: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:00:42.138: INFO @metrics_hook: train matching accuracy: 0.9903785289277124, N=110586, DB_SIZE=110592, N_IDS=11084 2021-03-14 16:00:42.318: INFO @decay_lr : LR updated to `0.0005032249` 2021-03-14 16:00:42.320: INFO @log_profile : T train: 116.976063 2021-03-14 16:00:42.320: INFO @log_profile : T read data: 2.254958 2021-03-14 16:00:42.320: INFO @log_profile : T hooks: 77.317623 2021-03-14 16:00:42.320: INFO @main_loop : Epoch 137 done 2021-03-14 16:00:42.320: INFO @main_loop : Training epoch 138 2021-03-14 16:02:40.966: INFO @log_variables: train loss nanmean: 0.026207 2021-03-14 16:02:40.966: INFO @log_variables: train positive_distance nanmean: 0.561104 2021-03-14 16:02:40.966: INFO @log_variables: train negative_distance nanmean: 1.419451 2021-03-14 16:02:40.966: INFO @log_variables: train distance nanmean: 0.858347 2021-03-14 16:02:40.966: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:03:59.029: INFO @metrics_hook: train matching accuracy: 0.9906230219730536, N=110590, DB_SIZE=110592, N_IDS=11039 2021-03-14 16:03:59.210: INFO @decay_lr : LR updated to `0.00050070876` 2021-03-14 16:03:59.211: INFO @log_profile : T train: 116.152637 2021-03-14 16:03:59.211: INFO @log_profile : T read data: 2.124231 2021-03-14 16:03:59.211: INFO @log_profile : T hooks: 78.564199 2021-03-14 16:03:59.211: INFO @main_loop : Epoch 138 done 2021-03-14 16:03:59.211: INFO @main_loop : Training epoch 139 2021-03-14 16:05:57.792: INFO @log_variables: train loss nanmean: 0.025380 2021-03-14 16:05:57.792: INFO @log_variables: train positive_distance nanmean: 0.560286 2021-03-14 16:05:57.792: INFO @log_variables: train negative_distance nanmean: 1.419466 2021-03-14 16:05:57.792: INFO @log_variables: train distance nanmean: 0.859180 2021-03-14 16:05:57.793: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:07:13.103: INFO @metrics_hook: train matching accuracy: 0.9908490821954968, N=110590, DB_SIZE=110592, N_IDS=11071 2021-03-14 16:07:13.291: INFO @decay_lr : LR updated to `0.0004982052` 2021-03-14 16:07:13.292: INFO @log_profile : T train: 116.088863 2021-03-14 16:07:13.292: INFO @log_profile : T read data: 2.096067 2021-03-14 16:07:13.292: INFO @log_profile : T hooks: 75.845584 2021-03-14 16:07:13.293: INFO @main_loop : Epoch 139 done 2021-03-14 16:07:13.293: INFO @main_loop : Training epoch 140 2021-03-14 16:09:10.711: INFO @log_variables: train loss nanmean: 0.024093 2021-03-14 16:09:10.711: INFO @log_variables: train positive_distance nanmean: 0.559509 2021-03-14 16:09:10.711: INFO @log_variables: train negative_distance nanmean: 1.419465 2021-03-14 16:09:10.711: INFO @log_variables: train distance nanmean: 0.859956 2021-03-14 16:09:10.711: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:10:25.878: INFO @metrics_hook: train matching accuracy: 0.9909120503499539, N=110586, DB_SIZE=110592, N_IDS=11052 2021-03-14 16:10:26.057: INFO @decay_lr : LR updated to `0.00049571414` 2021-03-14 16:10:26.059: INFO @log_profile : T train: 116.272362 2021-03-14 16:10:26.059: INFO @log_profile : T read data: 0.764076 2021-03-14 16:10:26.059: INFO @log_profile : T hooks: 75.678437 2021-03-14 16:10:26.059: INFO @main_loop : Epoch 140 done 2021-03-14 16:10:26.059: INFO @main_loop : Training epoch 141 2021-03-14 16:12:24.773: INFO @log_variables: train loss nanmean: 0.024962 2021-03-14 16:12:24.773: INFO @log_variables: train positive_distance nanmean: 0.559917 2021-03-14 16:12:24.773: INFO @log_variables: train negative_distance nanmean: 1.419456 2021-03-14 16:12:24.773: INFO @log_variables: train distance nanmean: 0.859540 2021-03-14 16:12:24.773: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:13:39.049: INFO @metrics_hook: train matching accuracy: 0.9901796775389512, N=110587, DB_SIZE=110592, N_IDS=11104 2021-03-14 16:13:39.229: INFO @decay_lr : LR updated to `0.00049323554` 2021-03-14 16:13:39.230: INFO @log_profile : T train: 116.215248 2021-03-14 16:13:39.230: INFO @log_profile : T read data: 2.108712 2021-03-14 16:13:39.230: INFO @log_profile : T hooks: 74.796785 2021-03-14 16:13:39.230: INFO @main_loop : Epoch 141 done 2021-03-14 16:13:39.230: INFO @main_loop : Training epoch 142 2021-03-14 16:15:38.059: INFO @log_variables: train loss nanmean: 0.024944 2021-03-14 16:15:38.059: INFO @log_variables: train positive_distance nanmean: 0.559502 2021-03-14 16:15:38.060: INFO @log_variables: train negative_distance nanmean: 1.419452 2021-03-14 16:15:38.060: INFO @log_variables: train distance nanmean: 0.859950 2021-03-14 16:15:38.060: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:16:53.320: INFO @metrics_hook: train matching accuracy: 0.9910568149963377, N=110587, DB_SIZE=110592, N_IDS=11067 2021-03-14 16:16:53.504: INFO @decay_lr : LR updated to `0.00049076934` 2021-03-14 16:16:53.505: INFO @log_profile : T train: 116.366618 2021-03-14 16:16:53.506: INFO @log_profile : T read data: 2.094216 2021-03-14 16:16:53.506: INFO @log_profile : T hooks: 75.763248 2021-03-14 16:16:53.506: INFO @main_loop : Epoch 142 done 2021-03-14 16:16:53.506: INFO @main_loop : Training epoch 143 2021-03-14 16:18:51.959: INFO @log_variables: train loss nanmean: 0.024178 2021-03-14 16:18:51.959: INFO @log_variables: train positive_distance nanmean: 0.559835 2021-03-14 16:18:51.959: INFO @log_variables: train negative_distance nanmean: 1.419466 2021-03-14 16:18:51.959: INFO @log_variables: train distance nanmean: 0.859631 2021-03-14 16:18:51.959: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:20:06.988: INFO @metrics_hook: train matching accuracy: 0.9914546145081656, N=110586, DB_SIZE=110592, N_IDS=11096 2021-03-14 16:20:07.171: INFO @decay_lr : LR updated to `0.0004883155` 2021-03-14 16:20:07.172: INFO @log_profile : T train: 115.913116 2021-03-14 16:20:07.173: INFO @log_profile : T read data: 2.175989 2021-03-14 16:20:07.173: INFO @log_profile : T hooks: 75.527111 2021-03-14 16:20:07.173: INFO @main_loop : Epoch 143 done 2021-03-14 16:20:07.173: INFO @main_loop : Training epoch 144 2021-03-14 16:22:09.947: INFO @log_variables: train loss nanmean: 0.024677 2021-03-14 16:22:09.947: INFO @log_variables: train positive_distance nanmean: 0.560123 2021-03-14 16:22:09.947: INFO @log_variables: train negative_distance nanmean: 1.419472 2021-03-14 16:22:09.948: INFO @log_variables: train distance nanmean: 0.859349 2021-03-14 16:22:09.948: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:23:25.835: INFO @metrics_hook: train matching accuracy: 0.9909658983007931, N=110581, DB_SIZE=110592, N_IDS=11070 2021-03-14 16:23:26.024: INFO @decay_lr : LR updated to `0.0004858739` 2021-03-14 16:23:26.026: INFO @log_profile : T train: 120.264480 2021-03-14 16:23:26.026: INFO @log_profile : T read data: 2.141893 2021-03-14 16:23:26.026: INFO @log_profile : T hooks: 76.395523 2021-03-14 16:23:26.026: INFO @main_loop : Epoch 144 done 2021-03-14 16:23:26.026: INFO @main_loop : Training epoch 145 2021-03-14 16:25:31.819: INFO @log_variables: train loss nanmean: 0.024051 2021-03-14 16:25:31.819: INFO @log_variables: train positive_distance nanmean: 0.559975 2021-03-14 16:25:31.819: INFO @log_variables: train negative_distance nanmean: 1.419466 2021-03-14 16:25:31.819: INFO @log_variables: train distance nanmean: 0.859491 2021-03-14 16:25:31.819: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:26:47.867: INFO @metrics_hook: train matching accuracy: 0.9910838623333876, N=110586, DB_SIZE=110592, N_IDS=11041 2021-03-14 16:26:48.050: INFO @decay_lr : LR updated to `0.00048344454` 2021-03-14 16:26:48.052: INFO @log_profile : T train: 123.291970 2021-03-14 16:26:48.052: INFO @log_profile : T read data: 2.109635 2021-03-14 16:26:48.052: INFO @log_profile : T hooks: 76.573176 2021-03-14 16:26:48.052: INFO @main_loop : Epoch 145 done 2021-03-14 16:26:48.052: INFO @main_loop : Training epoch 146 2021-03-14 16:28:46.919: INFO @log_variables: train loss nanmean: 0.023438 2021-03-14 16:28:46.919: INFO @log_variables: train positive_distance nanmean: 0.559466 2021-03-14 16:28:46.919: INFO @log_variables: train negative_distance nanmean: 1.419466 2021-03-14 16:28:46.919: INFO @log_variables: train distance nanmean: 0.860000 2021-03-14 16:28:46.920: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:30:02.538: INFO @metrics_hook: train matching accuracy: 0.9905140841886332, N=110585, DB_SIZE=110592, N_IDS=11062 2021-03-14 16:30:02.726: INFO @decay_lr : LR updated to `0.00048102732` 2021-03-14 16:30:02.727: INFO @log_profile : T train: 116.232706 2021-03-14 16:30:02.727: INFO @log_profile : T read data: 2.263179 2021-03-14 16:30:02.727: INFO @log_profile : T hooks: 76.128792 2021-03-14 16:30:02.727: INFO @main_loop : Epoch 146 done 2021-03-14 16:30:02.727: INFO @main_loop : Training epoch 147 2021-03-14 16:32:01.274: INFO @log_variables: train loss nanmean: 0.025296 2021-03-14 16:32:01.275: INFO @log_variables: train positive_distance nanmean: 0.559334 2021-03-14 16:32:01.275: INFO @log_variables: train negative_distance nanmean: 1.419460 2021-03-14 16:32:01.275: INFO @log_variables: train distance nanmean: 0.860127 2021-03-14 16:32:01.275: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:33:17.451: INFO @metrics_hook: train matching accuracy: 0.9907857853332128, N=110590, DB_SIZE=110592, N_IDS=11078 2021-03-14 16:33:17.641: INFO @decay_lr : LR updated to `0.00047862218` 2021-03-14 16:33:17.642: INFO @log_profile : T train: 116.131312 2021-03-14 16:33:17.642: INFO @log_profile : T read data: 2.027380 2021-03-14 16:33:17.642: INFO @log_profile : T hooks: 76.705743 2021-03-14 16:33:17.642: INFO @main_loop : Epoch 147 done 2021-03-14 16:33:17.642: INFO @main_loop : Training epoch 148 2021-03-14 16:35:16.582: INFO @log_variables: train loss nanmean: 0.023807 2021-03-14 16:35:16.582: INFO @log_variables: train positive_distance nanmean: 0.559060 2021-03-14 16:35:16.582: INFO @log_variables: train negative_distance nanmean: 1.419476 2021-03-14 16:35:16.582: INFO @log_variables: train distance nanmean: 0.860416 2021-03-14 16:35:16.582: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:36:34.700: INFO @metrics_hook: train matching accuracy: 0.990442173795099, N=110590, DB_SIZE=110592, N_IDS=11089 2021-03-14 16:36:34.883: INFO @decay_lr : LR updated to `0.00047622906` 2021-03-14 16:36:34.884: INFO @log_profile : T train: 116.442031 2021-03-14 16:36:34.884: INFO @log_profile : T read data: 2.106801 2021-03-14 16:36:34.884: INFO @log_profile : T hooks: 78.641805 2021-03-14 16:36:34.884: INFO @main_loop : Epoch 148 done 2021-03-14 16:36:34.884: INFO @main_loop : Training epoch 149 2021-03-14 16:38:32.178: INFO @log_variables: train loss nanmean: 0.022851 2021-03-14 16:38:32.179: INFO @log_variables: train positive_distance nanmean: 0.559304 2021-03-14 16:38:32.179: INFO @log_variables: train negative_distance nanmean: 1.419473 2021-03-14 16:38:32.179: INFO @log_variables: train distance nanmean: 0.860169 2021-03-14 16:38:32.179: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:39:49.237: INFO @metrics_hook: train matching accuracy: 0.990288013745083, N=110585, DB_SIZE=110592, N_IDS=11084 2021-03-14 16:39:49.414: INFO @decay_lr : LR updated to `0.0004738479` 2021-03-14 16:39:49.415: INFO @log_profile : T train: 116.156316 2021-03-14 16:39:49.415: INFO @log_profile : T read data: 0.750869 2021-03-14 16:39:49.415: INFO @log_profile : T hooks: 77.573072 2021-03-14 16:39:49.415: INFO @main_loop : Epoch 149 done 2021-03-14 16:39:49.415: INFO @main_loop : Training epoch 150 2021-03-14 16:41:48.154: INFO @log_variables: train loss nanmean: 0.023785 2021-03-14 16:41:48.155: INFO @log_variables: train positive_distance nanmean: 0.559392 2021-03-14 16:41:48.155: INFO @log_variables: train negative_distance nanmean: 1.419468 2021-03-14 16:41:48.155: INFO @log_variables: train distance nanmean: 0.860075 2021-03-14 16:41:48.155: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:43:05.043: INFO @metrics_hook: train matching accuracy: 0.9908487511981625, N=110586, DB_SIZE=110592, N_IDS=11053 2021-03-14 16:43:05.229: INFO @decay_lr : LR updated to `0.00047147868` 2021-03-14 16:43:05.544: INFO @save : Model saved to: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE/model_150.ckpt 2021-03-14 16:43:05.564: INFO @log_profile : T train: 116.356865 2021-03-14 16:43:05.564: INFO @log_profile : T read data: 1.991607 2021-03-14 16:43:05.564: INFO @log_profile : T hooks: 77.749397 2021-03-14 16:43:05.564: INFO @main_loop : Epoch 150 done 2021-03-14 16:43:05.564: INFO @main_loop : Training epoch 151 2021-03-14 16:45:11.852: INFO @log_variables: train loss nanmean: 0.023956 2021-03-14 16:45:11.852: INFO @log_variables: train positive_distance nanmean: 0.558523 2021-03-14 16:45:11.852: INFO @log_variables: train negative_distance nanmean: 1.419468 2021-03-14 16:45:11.852: INFO @log_variables: train distance nanmean: 0.860944 2021-03-14 16:45:11.852: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:46:26.666: INFO @metrics_hook: train matching accuracy: 0.991292002748942, N=110588, DB_SIZE=110592, N_IDS=11075 2021-03-14 16:46:26.856: INFO @decay_lr : LR updated to `0.0004691213` 2021-03-14 16:46:26.857: INFO @log_profile : T train: 123.874337 2021-03-14 16:46:26.857: INFO @log_profile : T read data: 2.018115 2021-03-14 16:46:26.857: INFO @log_profile : T hooks: 75.349596 2021-03-14 16:46:26.857: INFO @main_loop : Epoch 151 done 2021-03-14 16:46:26.857: INFO @main_loop : Training epoch 152 2021-03-14 16:48:32.368: INFO @log_variables: train loss nanmean: 0.023200 2021-03-14 16:48:32.369: INFO @log_variables: train positive_distance nanmean: 0.557700 2021-03-14 16:48:32.369: INFO @log_variables: train negative_distance nanmean: 1.419476 2021-03-14 16:48:32.369: INFO @log_variables: train distance nanmean: 0.861776 2021-03-14 16:48:32.369: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:49:48.049: INFO @metrics_hook: train matching accuracy: 0.9906861503960647, N=110588, DB_SIZE=110592, N_IDS=11067 2021-03-14 16:49:48.236: INFO @decay_lr : LR updated to `0.0004667757` 2021-03-14 16:49:48.237: INFO @log_profile : T train: 122.962062 2021-03-14 16:49:48.237: INFO @log_profile : T read data: 2.159007 2021-03-14 16:49:48.237: INFO @log_profile : T hooks: 76.208160 2021-03-14 16:49:48.237: INFO @main_loop : Epoch 152 done 2021-03-14 16:49:48.238: INFO @main_loop : Training epoch 153 2021-03-14 16:51:47.101: INFO @log_variables: train loss nanmean: 0.023027 2021-03-14 16:51:47.101: INFO @log_variables: train positive_distance nanmean: 0.557753 2021-03-14 16:51:47.101: INFO @log_variables: train negative_distance nanmean: 1.419477 2021-03-14 16:51:47.101: INFO @log_variables: train distance nanmean: 0.861724 2021-03-14 16:51:47.101: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:53:02.841: INFO @metrics_hook: train matching accuracy: 0.9906946039554716, N=110581, DB_SIZE=110592, N_IDS=11072 2021-03-14 16:53:03.020: INFO @decay_lr : LR updated to `0.00046444184` 2021-03-14 16:53:03.022: INFO @log_profile : T train: 116.333917 2021-03-14 16:53:03.022: INFO @log_profile : T read data: 2.123961 2021-03-14 16:53:03.022: INFO @log_profile : T hooks: 76.275381 2021-03-14 16:53:03.022: INFO @main_loop : Epoch 153 done 2021-03-14 16:53:03.022: INFO @main_loop : Training epoch 154 2021-03-14 16:55:01.683: INFO @log_variables: train loss nanmean: 0.023170 2021-03-14 16:55:01.683: INFO @log_variables: train positive_distance nanmean: 0.558491 2021-03-14 16:55:01.683: INFO @log_variables: train negative_distance nanmean: 1.419518 2021-03-14 16:55:01.683: INFO @log_variables: train distance nanmean: 0.861027 2021-03-14 16:55:01.683: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:56:17.731: INFO @metrics_hook: train matching accuracy: 0.9908759619123405, N=110587, DB_SIZE=110592, N_IDS=11049 2021-03-14 16:56:17.912: INFO @decay_lr : LR updated to `0.00046211964` 2021-03-14 16:56:17.914: INFO @log_profile : T train: 116.218266 2021-03-14 16:56:17.914: INFO @log_profile : T read data: 2.064289 2021-03-14 16:56:17.914: INFO @log_profile : T hooks: 76.559125 2021-03-14 16:56:17.915: INFO @main_loop : Epoch 154 done 2021-03-14 16:56:17.915: INFO @main_loop : Training epoch 155 2021-03-14 16:58:16.708: INFO @log_variables: train loss nanmean: 0.022283 2021-03-14 16:58:16.709: INFO @log_variables: train positive_distance nanmean: 0.558306 2021-03-14 16:58:16.709: INFO @log_variables: train negative_distance nanmean: 1.419482 2021-03-14 16:58:16.709: INFO @log_variables: train distance nanmean: 0.861175 2021-03-14 16:58:16.709: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 16:59:32.021: INFO @metrics_hook: train matching accuracy: 0.9912197415656169, N=110589, DB_SIZE=110592, N_IDS=11050 2021-03-14 16:59:32.207: INFO @decay_lr : LR updated to `0.00045980903` 2021-03-14 16:59:32.208: INFO @log_profile : T train: 116.227866 2021-03-14 16:59:32.208: INFO @log_profile : T read data: 2.180661 2021-03-14 16:59:32.208: INFO @log_profile : T hooks: 75.833430 2021-03-14 16:59:32.208: INFO @main_loop : Epoch 155 done 2021-03-14 16:59:32.208: INFO @main_loop : Training epoch 156 2021-03-14 17:01:31.628: INFO @log_variables: train loss nanmean: 0.021823 2021-03-14 17:01:31.629: INFO @log_variables: train positive_distance nanmean: 0.556951 2021-03-14 17:01:31.629: INFO @log_variables: train negative_distance nanmean: 1.419489 2021-03-14 17:01:31.629: INFO @log_variables: train distance nanmean: 0.862538 2021-03-14 17:01:31.629: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:02:48.121: INFO @metrics_hook: train matching accuracy: 0.9909033366488833, N=110590, DB_SIZE=110592, N_IDS=11099 2021-03-14 17:02:48.311: INFO @decay_lr : LR updated to `0.00045750997` 2021-03-14 17:02:48.312: INFO @log_profile : T train: 116.832195 2021-03-14 17:02:48.312: INFO @log_profile : T read data: 2.174798 2021-03-14 17:02:48.312: INFO @log_profile : T hooks: 77.045640 2021-03-14 17:02:48.312: INFO @main_loop : Epoch 156 done 2021-03-14 17:02:48.312: INFO @main_loop : Training epoch 157 2021-03-14 17:04:54.902: INFO @log_variables: train loss nanmean: 0.023302 2021-03-14 17:04:54.903: INFO @log_variables: train positive_distance nanmean: 0.558474 2021-03-14 17:04:54.903: INFO @log_variables: train negative_distance nanmean: 1.419495 2021-03-14 17:04:54.903: INFO @log_variables: train distance nanmean: 0.861021 2021-03-14 17:04:54.903: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:06:10.151: INFO @metrics_hook: train matching accuracy: 0.990532512275181, N=110589, DB_SIZE=110592, N_IDS=11082 2021-03-14 17:06:10.331: INFO @decay_lr : LR updated to `0.0004552224` 2021-03-14 17:06:10.333: INFO @log_profile : T train: 124.138611 2021-03-14 17:06:10.333: INFO @log_profile : T read data: 2.072546 2021-03-14 17:06:10.333: INFO @log_profile : T hooks: 75.757974 2021-03-14 17:06:10.333: INFO @main_loop : Epoch 157 done 2021-03-14 17:06:10.333: INFO @main_loop : Training epoch 158 2021-03-14 17:08:08.970: INFO @log_variables: train loss nanmean: 0.023429 2021-03-14 17:08:08.970: INFO @log_variables: train positive_distance nanmean: 0.558123 2021-03-14 17:08:08.970: INFO @log_variables: train negative_distance nanmean: 1.419500 2021-03-14 17:08:08.971: INFO @log_variables: train distance nanmean: 0.861377 2021-03-14 17:08:08.971: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:09:24.296: INFO @metrics_hook: train matching accuracy: 0.9908123310092871, N=110583, DB_SIZE=110592, N_IDS=11091 2021-03-14 17:09:24.481: INFO @decay_lr : LR updated to `0.0004529463` 2021-03-14 17:09:24.482: INFO @log_profile : T train: 116.287337 2021-03-14 17:09:24.483: INFO @log_profile : T read data: 1.959394 2021-03-14 17:09:24.483: INFO @log_profile : T hooks: 75.852461 2021-03-14 17:09:24.483: INFO @main_loop : Epoch 158 done 2021-03-14 17:09:24.483: INFO @main_loop : Training epoch 159 2021-03-14 17:11:29.940: INFO @log_variables: train loss nanmean: 0.022848 2021-03-14 17:11:29.940: INFO @log_variables: train positive_distance nanmean: 0.558366 2021-03-14 17:11:29.940: INFO @log_variables: train negative_distance nanmean: 1.419487 2021-03-14 17:11:29.940: INFO @log_variables: train distance nanmean: 0.861121 2021-03-14 17:11:29.940: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:12:45.718: INFO @metrics_hook: train matching accuracy: 0.9910568149963377, N=110587, DB_SIZE=110592, N_IDS=11053 2021-03-14 17:12:45.902: INFO @decay_lr : LR updated to `0.00045068155` 2021-03-14 17:12:45.903: INFO @log_profile : T train: 124.272076 2021-03-14 17:12:45.903: INFO @log_profile : T read data: 0.784744 2021-03-14 17:12:45.903: INFO @log_profile : T hooks: 76.312274 2021-03-14 17:12:45.904: INFO @main_loop : Epoch 159 done 2021-03-14 17:12:45.904: INFO @main_loop : Training epoch 160 2021-03-14 17:14:52.548: INFO @log_variables: train loss nanmean: 0.022372 2021-03-14 17:14:52.548: INFO @log_variables: train positive_distance nanmean: 0.557486 2021-03-14 17:14:52.548: INFO @log_variables: train negative_distance nanmean: 1.419505 2021-03-14 17:14:52.548: INFO @log_variables: train distance nanmean: 0.862019 2021-03-14 17:14:52.548: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:16:09.106: INFO @metrics_hook: train matching accuracy: 0.9902881015680104, N=110586, DB_SIZE=110592, N_IDS=11080 2021-03-14 17:16:09.296: INFO @decay_lr : LR updated to `0.00044842815` 2021-03-14 17:16:09.297: INFO @log_profile : T train: 124.136865 2021-03-14 17:16:09.297: INFO @log_profile : T read data: 2.102845 2021-03-14 17:16:09.298: INFO @log_profile : T hooks: 77.103479 2021-03-14 17:16:09.298: INFO @main_loop : Epoch 160 done 2021-03-14 17:16:09.298: INFO @main_loop : Training epoch 161 2021-03-14 17:18:15.786: INFO @log_variables: train loss nanmean: 0.021926 2021-03-14 17:18:15.786: INFO @log_variables: train positive_distance nanmean: 0.558228 2021-03-14 17:18:15.786: INFO @log_variables: train negative_distance nanmean: 1.419494 2021-03-14 17:18:15.786: INFO @log_variables: train distance nanmean: 0.861265 2021-03-14 17:18:15.786: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:19:30.632: INFO @metrics_hook: train matching accuracy: 0.9905049555089344, N=110584, DB_SIZE=110592, N_IDS=11072 2021-03-14 17:19:30.814: INFO @decay_lr : LR updated to `0.00044618602` 2021-03-14 17:19:30.816: INFO @log_profile : T train: 123.926539 2021-03-14 17:19:30.816: INFO @log_profile : T read data: 2.149657 2021-03-14 17:19:30.816: INFO @log_profile : T hooks: 75.391289 2021-03-14 17:19:30.816: INFO @main_loop : Epoch 161 done 2021-03-14 17:19:30.816: INFO @main_loop : Training epoch 162 2021-03-14 17:21:37.689: INFO @log_variables: train loss nanmean: 0.022506 2021-03-14 17:21:37.689: INFO @log_variables: train positive_distance nanmean: 0.557158 2021-03-14 17:21:37.690: INFO @log_variables: train negative_distance nanmean: 1.419496 2021-03-14 17:21:37.690: INFO @log_variables: train distance nanmean: 0.862338 2021-03-14 17:21:37.690: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:22:52.519: INFO @metrics_hook: train matching accuracy: 0.9907404058306507, N=110588, DB_SIZE=110592, N_IDS=11068 2021-03-14 17:22:52.720: INFO @decay_lr : LR updated to `0.0004439551` 2021-03-14 17:22:52.721: INFO @log_profile : T train: 124.344107 2021-03-14 17:22:52.721: INFO @log_profile : T read data: 2.146960 2021-03-14 17:22:52.721: INFO @log_profile : T hooks: 75.362964 2021-03-14 17:22:52.721: INFO @main_loop : Epoch 162 done 2021-03-14 17:22:52.721: INFO @main_loop : Training epoch 163 2021-03-14 17:24:59.132: INFO @log_variables: train loss nanmean: 0.021706 2021-03-14 17:24:59.133: INFO @log_variables: train positive_distance nanmean: 0.557014 2021-03-14 17:24:59.133: INFO @log_variables: train negative_distance nanmean: 1.419490 2021-03-14 17:24:59.133: INFO @log_variables: train distance nanmean: 0.862476 2021-03-14 17:24:59.133: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:26:12.753: INFO @metrics_hook: train matching accuracy: 0.9904600860852896, N=110588, DB_SIZE=110592, N_IDS=11067 2021-03-14 17:26:12.942: INFO @decay_lr : LR updated to `0.00044173532` 2021-03-14 17:26:12.944: INFO @log_profile : T train: 124.018140 2021-03-14 17:26:12.944: INFO @log_profile : T read data: 1.994479 2021-03-14 17:26:12.944: INFO @log_profile : T hooks: 74.159194 2021-03-14 17:26:12.944: INFO @main_loop : Epoch 163 done 2021-03-14 17:26:12.944: INFO @main_loop : Training epoch 164 2021-03-14 17:28:19.359: INFO @log_variables: train loss nanmean: 0.022769 2021-03-14 17:28:19.359: INFO @log_variables: train positive_distance nanmean: 0.557062 2021-03-14 17:28:19.359: INFO @log_variables: train negative_distance nanmean: 1.419523 2021-03-14 17:28:19.359: INFO @log_variables: train distance nanmean: 0.862462 2021-03-14 17:28:19.360: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:29:34.552: INFO @metrics_hook: train matching accuracy: 0.9903514879687486, N=110587, DB_SIZE=110592, N_IDS=11076 2021-03-14 17:29:34.741: INFO @decay_lr : LR updated to `0.00043952663` 2021-03-14 17:29:34.742: INFO @log_profile : T train: 123.963344 2021-03-14 17:29:34.742: INFO @log_profile : T read data: 2.079606 2021-03-14 17:29:34.742: INFO @log_profile : T hooks: 75.704182 2021-03-14 17:29:34.742: INFO @main_loop : Epoch 164 done 2021-03-14 17:29:34.742: INFO @main_loop : Training epoch 165 2021-03-14 17:31:41.358: INFO @log_variables: train loss nanmean: 0.022093 2021-03-14 17:31:41.358: INFO @log_variables: train positive_distance nanmean: 0.557262 2021-03-14 17:31:41.358: INFO @log_variables: train negative_distance nanmean: 1.419470 2021-03-14 17:31:41.359: INFO @log_variables: train distance nanmean: 0.862208 2021-03-14 17:31:41.359: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:32:57.058: INFO @metrics_hook: train matching accuracy: 0.9903060117195978, N=110584, DB_SIZE=110592, N_IDS=11110 2021-03-14 17:32:57.248: INFO @decay_lr : LR updated to `0.000437329` 2021-03-14 17:32:57.249: INFO @log_profile : T train: 124.133435 2021-03-14 17:32:57.249: INFO @log_profile : T read data: 2.088973 2021-03-14 17:32:57.249: INFO @log_profile : T hooks: 76.233563 2021-03-14 17:32:57.249: INFO @main_loop : Epoch 165 done 2021-03-14 17:32:57.249: INFO @main_loop : Training epoch 166 2021-03-14 17:35:03.816: INFO @log_variables: train loss nanmean: 0.021214 2021-03-14 17:35:03.816: INFO @log_variables: train positive_distance nanmean: 0.556758 2021-03-14 17:35:03.816: INFO @log_variables: train negative_distance nanmean: 1.419515 2021-03-14 17:35:03.816: INFO @log_variables: train distance nanmean: 0.862757 2021-03-14 17:35:03.816: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:36:19.417: INFO @metrics_hook: train matching accuracy: 0.9907041514825432, N=110587, DB_SIZE=110592, N_IDS=11077 2021-03-14 17:36:19.606: INFO @decay_lr : LR updated to `0.00043514234` 2021-03-14 17:36:19.607: INFO @log_profile : T train: 124.199881 2021-03-14 17:36:19.608: INFO @log_profile : T read data: 1.971523 2021-03-14 17:36:19.608: INFO @log_profile : T hooks: 76.135997 2021-03-14 17:36:19.608: INFO @main_loop : Epoch 166 done 2021-03-14 17:36:19.608: INFO @main_loop : Training epoch 167 2021-03-14 17:38:26.319: INFO @log_variables: train loss nanmean: 0.021933 2021-03-14 17:38:26.319: INFO @log_variables: train positive_distance nanmean: 0.556585 2021-03-14 17:38:26.319: INFO @log_variables: train negative_distance nanmean: 1.419467 2021-03-14 17:38:26.319: INFO @log_variables: train distance nanmean: 0.862883 2021-03-14 17:38:26.320: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:39:41.107: INFO @metrics_hook: train matching accuracy: 0.990306099380567, N=110585, DB_SIZE=110592, N_IDS=11071 2021-03-14 17:39:41.289: INFO @decay_lr : LR updated to `0.00043296663` 2021-03-14 17:39:41.291: INFO @log_profile : T train: 124.244818 2021-03-14 17:39:41.291: INFO @log_profile : T read data: 2.071402 2021-03-14 17:39:41.291: INFO @log_profile : T hooks: 75.316196 2021-03-14 17:39:41.291: INFO @main_loop : Epoch 167 done 2021-03-14 17:39:41.291: INFO @main_loop : Training epoch 168 2021-03-14 17:41:46.909: INFO @log_variables: train loss nanmean: 0.020753 2021-03-14 17:41:46.909: INFO @log_variables: train positive_distance nanmean: 0.557426 2021-03-14 17:41:46.909: INFO @log_variables: train negative_distance nanmean: 1.419533 2021-03-14 17:41:46.909: INFO @log_variables: train distance nanmean: 0.862106 2021-03-14 17:41:46.909: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:43:00.495: INFO @metrics_hook: train matching accuracy: 0.9909756759200651, N=110590, DB_SIZE=110592, N_IDS=11086 2021-03-14 17:43:00.683: INFO @decay_lr : LR updated to `0.0004308018` 2021-03-14 17:43:00.684: INFO @log_profile : T train: 124.434788 2021-03-14 17:43:00.684: INFO @log_profile : T read data: 0.787062 2021-03-14 17:43:00.684: INFO @log_profile : T hooks: 74.119397 2021-03-14 17:43:00.684: INFO @main_loop : Epoch 168 done 2021-03-14 17:43:00.684: INFO @main_loop : Training epoch 169 2021-03-14 17:45:07.125: INFO @log_variables: train loss nanmean: 0.021652 2021-03-14 17:45:07.125: INFO @log_variables: train positive_distance nanmean: 0.557454 2021-03-14 17:45:07.125: INFO @log_variables: train negative_distance nanmean: 1.419508 2021-03-14 17:45:07.125: INFO @log_variables: train distance nanmean: 0.862054 2021-03-14 17:45:07.125: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:46:22.704: INFO @metrics_hook: train matching accuracy: 0.9903153173519491, N=110587, DB_SIZE=110592, N_IDS=11079 2021-03-14 17:46:22.877: INFO @decay_lr : LR updated to `0.0004286478` 2021-03-14 17:46:22.878: INFO @log_profile : T train: 123.989419 2021-03-14 17:46:22.878: INFO @log_profile : T read data: 2.064724 2021-03-14 17:46:22.878: INFO @log_profile : T hooks: 76.089622 2021-03-14 17:46:22.878: INFO @main_loop : Epoch 169 done 2021-03-14 17:46:22.878: INFO @main_loop : Training epoch 170 2021-03-14 17:48:29.594: INFO @log_variables: train loss nanmean: 0.021523 2021-03-14 17:48:29.594: INFO @log_variables: train positive_distance nanmean: 0.556738 2021-03-14 17:48:29.594: INFO @log_variables: train negative_distance nanmean: 1.419507 2021-03-14 17:48:29.594: INFO @log_variables: train distance nanmean: 0.862769 2021-03-14 17:48:29.594: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:49:45.245: INFO @metrics_hook: train matching accuracy: 0.9907310279968169, N=110584, DB_SIZE=110592, N_IDS=11079 2021-03-14 17:49:45.424: INFO @decay_lr : LR updated to `0.00042650456` 2021-03-14 17:49:45.425: INFO @log_profile : T train: 124.241454 2021-03-14 17:49:45.426: INFO @log_profile : T read data: 2.078465 2021-03-14 17:49:45.426: INFO @log_profile : T hooks: 76.176509 2021-03-14 17:49:45.426: INFO @main_loop : Epoch 170 done 2021-03-14 17:49:45.426: INFO @main_loop : Training epoch 171 2021-03-14 17:51:52.049: INFO @log_variables: train loss nanmean: 0.020836 2021-03-14 17:51:52.050: INFO @log_variables: train positive_distance nanmean: 0.555743 2021-03-14 17:51:52.050: INFO @log_variables: train negative_distance nanmean: 1.419517 2021-03-14 17:51:52.050: INFO @log_variables: train distance nanmean: 0.863774 2021-03-14 17:51:52.050: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:53:06.007: INFO @metrics_hook: train matching accuracy: 0.9897364018628204, N=110585, DB_SIZE=110592, N_IDS=11062 2021-03-14 17:53:06.191: INFO @decay_lr : LR updated to `0.00042437203` 2021-03-14 17:53:06.192: INFO @log_profile : T train: 124.182841 2021-03-14 17:53:06.192: INFO @log_profile : T read data: 2.039234 2021-03-14 17:53:06.193: INFO @log_profile : T hooks: 74.493240 2021-03-14 17:53:06.193: INFO @main_loop : Epoch 171 done 2021-03-14 17:53:06.193: INFO @main_loop : Training epoch 172 2021-03-14 17:55:12.765: INFO @log_variables: train loss nanmean: 0.021771 2021-03-14 17:55:12.765: INFO @log_variables: train positive_distance nanmean: 0.557251 2021-03-14 17:55:12.765: INFO @log_variables: train negative_distance nanmean: 1.419478 2021-03-14 17:55:12.765: INFO @log_variables: train distance nanmean: 0.862227 2021-03-14 17:55:12.765: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:56:28.250: INFO @metrics_hook: train matching accuracy: 0.9905957246717546, N=110588, DB_SIZE=110592, N_IDS=11073 2021-03-14 17:56:28.447: INFO @decay_lr : LR updated to `0.00042225016` 2021-03-14 17:56:28.448: INFO @log_profile : T train: 124.113123 2021-03-14 17:56:28.448: INFO @log_profile : T read data: 2.058673 2021-03-14 17:56:28.448: INFO @log_profile : T hooks: 76.033168 2021-03-14 17:56:28.448: INFO @main_loop : Epoch 172 done 2021-03-14 17:56:28.449: INFO @main_loop : Training epoch 173 2021-03-14 17:58:34.497: INFO @log_variables: train loss nanmean: 0.022154 2021-03-14 17:58:34.497: INFO @log_variables: train positive_distance nanmean: 0.557270 2021-03-14 17:58:34.497: INFO @log_variables: train negative_distance nanmean: 1.419510 2021-03-14 17:58:34.497: INFO @log_variables: train distance nanmean: 0.862239 2021-03-14 17:58:34.497: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 17:59:50.200: INFO @metrics_hook: train matching accuracy: 0.9907134460620309, N=110590, DB_SIZE=110592, N_IDS=11051 2021-03-14 17:59:50.384: INFO @decay_lr : LR updated to `0.0004201389` 2021-03-14 17:59:50.385: INFO @log_profile : T train: 123.691178 2021-03-14 17:59:50.385: INFO @log_profile : T read data: 1.996078 2021-03-14 17:59:50.385: INFO @log_profile : T hooks: 76.199157 2021-03-14 17:59:50.385: INFO @main_loop : Epoch 173 done 2021-03-14 17:59:50.385: INFO @main_loop : Training epoch 174 2021-03-14 18:01:56.392: INFO @log_variables: train loss nanmean: 0.021283 2021-03-14 18:01:56.392: INFO @log_variables: train positive_distance nanmean: 0.556644 2021-03-14 18:01:56.392: INFO @log_variables: train negative_distance nanmean: 1.419485 2021-03-14 18:01:56.392: INFO @log_variables: train distance nanmean: 0.862840 2021-03-14 18:01:56.392: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:03:12.442: INFO @metrics_hook: train matching accuracy: 0.99026106142675, N=110587, DB_SIZE=110592, N_IDS=11114 2021-03-14 18:03:12.630: INFO @decay_lr : LR updated to `0.00041803822` 2021-03-14 18:03:12.631: INFO @log_profile : T train: 123.600042 2021-03-14 18:03:12.631: INFO @log_profile : T read data: 2.009559 2021-03-14 18:03:12.631: INFO @log_profile : T hooks: 76.586385 2021-03-14 18:03:12.631: INFO @main_loop : Epoch 174 done 2021-03-14 18:03:12.631: INFO @main_loop : Training epoch 175 2021-03-14 18:05:19.381: INFO @log_variables: train loss nanmean: 0.020294 2021-03-14 18:05:19.381: INFO @log_variables: train positive_distance nanmean: 0.556094 2021-03-14 18:05:19.381: INFO @log_variables: train negative_distance nanmean: 1.419507 2021-03-14 18:05:19.381: INFO @log_variables: train distance nanmean: 0.863413 2021-03-14 18:05:19.381: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:06:34.702: INFO @metrics_hook: train matching accuracy: 0.9914094008283146, N=110586, DB_SIZE=110592, N_IDS=11084 2021-03-14 18:06:34.886: INFO @decay_lr : LR updated to `0.00041594804` 2021-03-14 18:06:34.887: INFO @log_profile : T train: 124.253943 2021-03-14 18:06:34.887: INFO @log_profile : T read data: 2.085556 2021-03-14 18:06:34.887: INFO @log_profile : T hooks: 75.864842 2021-03-14 18:06:34.887: INFO @main_loop : Epoch 175 done 2021-03-14 18:06:34.888: INFO @main_loop : Training epoch 176 2021-03-14 18:08:41.281: INFO @log_variables: train loss nanmean: 0.019771 2021-03-14 18:08:41.281: INFO @log_variables: train positive_distance nanmean: 0.555480 2021-03-14 18:08:41.281: INFO @log_variables: train negative_distance nanmean: 1.419521 2021-03-14 18:08:41.281: INFO @log_variables: train distance nanmean: 0.864041 2021-03-14 18:08:41.281: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:09:55.979: INFO @metrics_hook: train matching accuracy: 0.9901073363053523, N=110587, DB_SIZE=110592, N_IDS=11079 2021-03-14 18:09:56.179: INFO @decay_lr : LR updated to `0.0004138683` 2021-03-14 18:09:56.180: INFO @log_profile : T train: 123.881949 2021-03-14 18:09:56.180: INFO @log_profile : T read data: 2.116456 2021-03-14 18:09:56.180: INFO @log_profile : T hooks: 75.244376 2021-03-14 18:09:56.181: INFO @main_loop : Epoch 176 done 2021-03-14 18:09:56.181: INFO @main_loop : Training epoch 177 2021-03-14 18:12:01.212: INFO @log_variables: train loss nanmean: 0.020669 2021-03-14 18:12:01.212: INFO @log_variables: train positive_distance nanmean: 0.555428 2021-03-14 18:12:01.212: INFO @log_variables: train negative_distance nanmean: 1.419487 2021-03-14 18:12:01.212: INFO @log_variables: train distance nanmean: 0.864059 2021-03-14 18:12:01.212: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:13:17.314: INFO @metrics_hook: train matching accuracy: 0.9907765761203747, N=110588, DB_SIZE=110592, N_IDS=11064 2021-03-14 18:13:17.503: INFO @decay_lr : LR updated to `0.00041179897` 2021-03-14 18:13:17.505: INFO @log_profile : T train: 123.876106 2021-03-14 18:13:17.505: INFO @log_profile : T read data: 0.763960 2021-03-14 18:13:17.505: INFO @log_profile : T hooks: 76.633681 2021-03-14 18:13:17.505: INFO @main_loop : Epoch 177 done 2021-03-14 18:13:17.505: INFO @main_loop : Training epoch 178 2021-03-14 18:15:24.324: INFO @log_variables: train loss nanmean: 0.021445 2021-03-14 18:15:24.325: INFO @log_variables: train positive_distance nanmean: 0.556067 2021-03-14 18:15:24.325: INFO @log_variables: train negative_distance nanmean: 1.419500 2021-03-14 18:15:24.325: INFO @log_variables: train distance nanmean: 0.863433 2021-03-14 18:15:24.325: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:16:40.215: INFO @metrics_hook: train matching accuracy: 0.9900888028793112, N=110582, DB_SIZE=110592, N_IDS=11112 2021-03-14 18:16:40.418: INFO @decay_lr : LR updated to `0.00040973997` 2021-03-14 18:16:40.419: INFO @log_profile : T train: 124.374779 2021-03-14 18:16:40.419: INFO @log_profile : T read data: 2.049508 2021-03-14 18:16:40.419: INFO @log_profile : T hooks: 76.439249 2021-03-14 18:16:40.419: INFO @main_loop : Epoch 178 done 2021-03-14 18:16:40.419: INFO @main_loop : Training epoch 179 2021-03-14 18:18:46.994: INFO @log_variables: train loss nanmean: 0.021010 2021-03-14 18:18:46.994: INFO @log_variables: train positive_distance nanmean: 0.556439 2021-03-14 18:18:46.994: INFO @log_variables: train negative_distance nanmean: 1.419523 2021-03-14 18:18:46.994: INFO @log_variables: train distance nanmean: 0.863084 2021-03-14 18:18:46.994: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:20:03.099: INFO @metrics_hook: train matching accuracy: 0.9911021891869897, N=110589, DB_SIZE=110592, N_IDS=11048 2021-03-14 18:20:03.286: INFO @decay_lr : LR updated to `0.00040769126` 2021-03-14 18:20:03.287: INFO @log_profile : T train: 124.000311 2021-03-14 18:20:03.287: INFO @log_profile : T read data: 2.174575 2021-03-14 18:20:03.287: INFO @log_profile : T hooks: 76.642182 2021-03-14 18:20:03.287: INFO @main_loop : Epoch 179 done 2021-03-14 18:20:03.287: INFO @main_loop : Training epoch 180 2021-03-14 18:22:10.273: INFO @log_variables: train loss nanmean: 0.020252 2021-03-14 18:22:10.273: INFO @log_variables: train positive_distance nanmean: 0.555066 2021-03-14 18:22:10.273: INFO @log_variables: train negative_distance nanmean: 1.419536 2021-03-14 18:22:10.273: INFO @log_variables: train distance nanmean: 0.864470 2021-03-14 18:22:10.273: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:23:25.973: INFO @metrics_hook: train matching accuracy: 0.9907855353703419, N=110587, DB_SIZE=110592, N_IDS=11080 2021-03-14 18:23:26.166: INFO @decay_lr : LR updated to `0.0004056528` 2021-03-14 18:23:26.168: INFO @log_profile : T train: 124.427802 2021-03-14 18:23:26.168: INFO @log_profile : T read data: 2.180245 2021-03-14 18:23:26.168: INFO @log_profile : T hooks: 76.221045 2021-03-14 18:23:26.168: INFO @main_loop : Epoch 180 done 2021-03-14 18:23:26.168: INFO @main_loop : Training epoch 181 2021-03-14 18:25:32.832: INFO @log_variables: train loss nanmean: 0.020519 2021-03-14 18:25:32.832: INFO @log_variables: train positive_distance nanmean: 0.555294 2021-03-14 18:25:32.832: INFO @log_variables: train negative_distance nanmean: 1.419500 2021-03-14 18:25:32.832: INFO @log_variables: train distance nanmean: 0.864205 2021-03-14 18:25:32.832: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:26:48.665: INFO @metrics_hook: train matching accuracy: 0.9900263131719579, N=110591, DB_SIZE=110592, N_IDS=11070 2021-03-14 18:26:48.850: INFO @decay_lr : LR updated to `0.00040362452` 2021-03-14 18:26:48.851: INFO @log_profile : T train: 124.024551 2021-03-14 18:26:48.851: INFO @log_profile : T read data: 2.239200 2021-03-14 18:26:48.851: INFO @log_profile : T hooks: 76.368584 2021-03-14 18:26:48.851: INFO @main_loop : Epoch 181 done 2021-03-14 18:26:48.851: INFO @main_loop : Training epoch 182 2021-03-14 18:28:55.208: INFO @log_variables: train loss nanmean: 0.019833 2021-03-14 18:28:55.208: INFO @log_variables: train positive_distance nanmean: 0.555469 2021-03-14 18:28:55.208: INFO @log_variables: train negative_distance nanmean: 1.419521 2021-03-14 18:28:55.208: INFO @log_variables: train distance nanmean: 0.864051 2021-03-14 18:28:55.208: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:30:10.022: INFO @metrics_hook: train matching accuracy: 0.9911562841925362, N=110587, DB_SIZE=110592, N_IDS=11093 2021-03-14 18:30:10.216: INFO @decay_lr : LR updated to `0.0004016064` 2021-03-14 18:30:10.218: INFO @log_profile : T train: 123.987350 2021-03-14 18:30:10.218: INFO @log_profile : T read data: 1.974560 2021-03-14 18:30:10.218: INFO @log_profile : T hooks: 75.354156 2021-03-14 18:30:10.218: INFO @main_loop : Epoch 182 done 2021-03-14 18:30:10.218: INFO @main_loop : Training epoch 183 2021-03-14 18:32:16.493: INFO @log_variables: train loss nanmean: 0.020579 2021-03-14 18:32:16.493: INFO @log_variables: train positive_distance nanmean: 0.556471 2021-03-14 18:32:16.494: INFO @log_variables: train negative_distance nanmean: 1.419516 2021-03-14 18:32:16.494: INFO @log_variables: train distance nanmean: 0.863045 2021-03-14 18:32:16.494: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:33:31.982: INFO @metrics_hook: train matching accuracy: 0.990071435030292, N=110590, DB_SIZE=110592, N_IDS=11082 2021-03-14 18:33:32.177: INFO @decay_lr : LR updated to `0.0003995984` 2021-03-14 18:33:32.178: INFO @log_profile : T train: 123.706283 2021-03-14 18:33:32.178: INFO @log_profile : T read data: 2.176777 2021-03-14 18:33:32.178: INFO @log_profile : T hooks: 76.026691 2021-03-14 18:33:32.178: INFO @main_loop : Epoch 183 done 2021-03-14 18:33:32.178: INFO @main_loop : Training epoch 184 2021-03-14 18:35:39.051: INFO @log_variables: train loss nanmean: 0.019861 2021-03-14 18:35:39.051: INFO @log_variables: train positive_distance nanmean: 0.555371 2021-03-14 18:35:39.051: INFO @log_variables: train negative_distance nanmean: 1.419525 2021-03-14 18:35:39.052: INFO @log_variables: train distance nanmean: 0.864154 2021-03-14 18:35:39.052: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:36:54.669: INFO @metrics_hook: train matching accuracy: 0.9903064500085904, N=110589, DB_SIZE=110592, N_IDS=11091 2021-03-14 18:36:54.866: INFO @decay_lr : LR updated to `0.0003976004` 2021-03-14 18:36:54.868: INFO @log_profile : T train: 124.369653 2021-03-14 18:36:54.868: INFO @log_profile : T read data: 2.101908 2021-03-14 18:36:54.868: INFO @log_profile : T hooks: 76.167105 2021-03-14 18:36:54.868: INFO @main_loop : Epoch 184 done 2021-03-14 18:36:54.868: INFO @main_loop : Training epoch 185 2021-03-14 18:39:00.915: INFO @log_variables: train loss nanmean: 0.019848 2021-03-14 18:39:00.915: INFO @log_variables: train positive_distance nanmean: 0.555433 2021-03-14 18:39:00.915: INFO @log_variables: train negative_distance nanmean: 1.419522 2021-03-14 18:39:00.915: INFO @log_variables: train distance nanmean: 0.864088 2021-03-14 18:39:00.915: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:40:16.852: INFO @metrics_hook: train matching accuracy: 0.991300966659734, N=110587, DB_SIZE=110592, N_IDS=11092 2021-03-14 18:40:17.042: INFO @decay_lr : LR updated to `0.0003956124` 2021-03-14 18:40:17.043: INFO @log_profile : T train: 123.723398 2021-03-14 18:40:17.044: INFO @log_profile : T read data: 1.962270 2021-03-14 18:40:17.044: INFO @log_profile : T hooks: 76.439583 2021-03-14 18:40:17.044: INFO @main_loop : Epoch 185 done 2021-03-14 18:40:17.044: INFO @main_loop : Training epoch 186 2021-03-14 18:42:23.255: INFO @log_variables: train loss nanmean: 0.019813 2021-03-14 18:42:23.255: INFO @log_variables: train positive_distance nanmean: 0.555579 2021-03-14 18:42:23.255: INFO @log_variables: train negative_distance nanmean: 1.419504 2021-03-14 18:42:23.255: INFO @log_variables: train distance nanmean: 0.863925 2021-03-14 18:42:23.255: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:43:38.807: INFO @metrics_hook: train matching accuracy: 0.99172612104278, N=110589, DB_SIZE=110592, N_IDS=11074 2021-03-14 18:43:38.997: INFO @decay_lr : LR updated to `0.00039363434` 2021-03-14 18:43:38.999: INFO @log_profile : T train: 123.879173 2021-03-14 18:43:38.999: INFO @log_profile : T read data: 1.946423 2021-03-14 18:43:38.999: INFO @log_profile : T hooks: 76.079080 2021-03-14 18:43:38.999: INFO @main_loop : Epoch 186 done 2021-03-14 18:43:38.999: INFO @main_loop : Training epoch 187 2021-03-14 18:45:44.164: INFO @log_variables: train loss nanmean: 0.020319 2021-03-14 18:45:44.164: INFO @log_variables: train positive_distance nanmean: 0.555316 2021-03-14 18:45:44.164: INFO @log_variables: train negative_distance nanmean: 1.419506 2021-03-14 18:45:44.164: INFO @log_variables: train distance nanmean: 0.864190 2021-03-14 18:45:44.164: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:47:00.731: INFO @metrics_hook: train matching accuracy: 0.9912100632115824, N=110581, DB_SIZE=110592, N_IDS=11071 2021-03-14 18:47:00.929: INFO @decay_lr : LR updated to `0.00039166617` 2021-03-14 18:47:00.931: INFO @log_profile : T train: 124.010787 2021-03-14 18:47:00.931: INFO @log_profile : T read data: 0.753659 2021-03-14 18:47:00.931: INFO @log_profile : T hooks: 77.116971 2021-03-14 18:47:00.931: INFO @main_loop : Epoch 187 done 2021-03-14 18:47:00.931: INFO @main_loop : Training epoch 188 2021-03-14 18:49:07.715: INFO @log_variables: train loss nanmean: 0.019919 2021-03-14 18:49:07.716: INFO @log_variables: train positive_distance nanmean: 0.556483 2021-03-14 18:49:07.716: INFO @log_variables: train negative_distance nanmean: 1.419515 2021-03-14 18:49:07.716: INFO @log_variables: train distance nanmean: 0.863031 2021-03-14 18:49:07.716: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:50:23.095: INFO @metrics_hook: train matching accuracy: 0.9907041514825432, N=110587, DB_SIZE=110592, N_IDS=11094 2021-03-14 18:50:23.300: INFO @decay_lr : LR updated to `0.00038970783` 2021-03-14 18:50:23.301: INFO @log_profile : T train: 124.265571 2021-03-14 18:50:23.301: INFO @log_profile : T read data: 2.119126 2021-03-14 18:50:23.302: INFO @log_profile : T hooks: 75.935143 2021-03-14 18:50:23.302: INFO @main_loop : Epoch 188 done 2021-03-14 18:50:23.302: INFO @main_loop : Training epoch 189 2021-03-14 18:52:29.791: INFO @log_variables: train loss nanmean: 0.019484 2021-03-14 18:52:29.791: INFO @log_variables: train positive_distance nanmean: 0.555322 2021-03-14 18:52:29.791: INFO @log_variables: train negative_distance nanmean: 1.419492 2021-03-14 18:52:29.791: INFO @log_variables: train distance nanmean: 0.864170 2021-03-14 18:52:29.792: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:53:45.464: INFO @metrics_hook: train matching accuracy: 0.9908398741273917, N=110588, DB_SIZE=110592, N_IDS=11071 2021-03-14 18:53:45.670: INFO @decay_lr : LR updated to `0.0003877593` 2021-03-14 18:53:45.671: INFO @log_profile : T train: 123.966180 2021-03-14 18:53:45.671: INFO @log_profile : T read data: 2.128037 2021-03-14 18:53:45.671: INFO @log_profile : T hooks: 76.225286 2021-03-14 18:53:45.671: INFO @main_loop : Epoch 189 done 2021-03-14 18:53:45.672: INFO @main_loop : Training epoch 190 2021-03-14 18:55:52.369: INFO @log_variables: train loss nanmean: 0.019689 2021-03-14 18:55:52.370: INFO @log_variables: train positive_distance nanmean: 0.556142 2021-03-14 18:55:52.370: INFO @log_variables: train negative_distance nanmean: 1.419544 2021-03-14 18:55:52.370: INFO @log_variables: train distance nanmean: 0.863402 2021-03-14 18:55:52.370: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 18:57:07.467: INFO @metrics_hook: train matching accuracy: 0.990342270651535, N=110585, DB_SIZE=110592, N_IDS=11075 2021-03-14 18:57:07.658: INFO @decay_lr : LR updated to `0.0003858205` 2021-03-14 18:57:07.659: INFO @log_profile : T train: 124.240246 2021-03-14 18:57:07.659: INFO @log_profile : T read data: 2.058288 2021-03-14 18:57:07.659: INFO @log_profile : T hooks: 75.638780 2021-03-14 18:57:07.659: INFO @main_loop : Epoch 190 done 2021-03-14 18:57:07.660: INFO @main_loop : Training epoch 191 2021-03-14 18:59:13.523: INFO @log_variables: train loss nanmean: 0.019367 2021-03-14 18:59:13.523: INFO @log_variables: train positive_distance nanmean: 0.554952 2021-03-14 18:59:13.524: INFO @log_variables: train negative_distance nanmean: 1.419535 2021-03-14 18:59:13.524: INFO @log_variables: train distance nanmean: 0.864583 2021-03-14 18:59:13.524: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:00:30.853: INFO @metrics_hook: train matching accuracy: 0.9908307486413412, N=110587, DB_SIZE=110592, N_IDS=11118 2021-03-14 19:00:31.050: INFO @decay_lr : LR updated to `0.0003838914` 2021-03-14 19:00:31.051: INFO @log_profile : T train: 123.470328 2021-03-14 19:00:31.051: INFO @log_profile : T read data: 2.011806 2021-03-14 19:00:31.051: INFO @log_profile : T hooks: 77.859315 2021-03-14 19:00:31.051: INFO @main_loop : Epoch 191 done 2021-03-14 19:00:31.051: INFO @main_loop : Training epoch 192 2021-03-14 19:02:38.026: INFO @log_variables: train loss nanmean: 0.019242 2021-03-14 19:02:38.026: INFO @log_variables: train positive_distance nanmean: 0.555144 2021-03-14 19:02:38.026: INFO @log_variables: train negative_distance nanmean: 1.419534 2021-03-14 19:02:38.026: INFO @log_variables: train distance nanmean: 0.864390 2021-03-14 19:02:38.026: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:03:54.604: INFO @metrics_hook: train matching accuracy: 0.9909029253515396, N=110585, DB_SIZE=110592, N_IDS=11079 2021-03-14 19:03:54.800: INFO @decay_lr : LR updated to `0.00038197194` 2021-03-14 19:03:54.801: INFO @log_profile : T train: 124.369611 2021-03-14 19:03:54.802: INFO @log_profile : T read data: 2.212003 2021-03-14 19:03:54.802: INFO @log_profile : T hooks: 77.117489 2021-03-14 19:03:54.802: INFO @main_loop : Epoch 192 done 2021-03-14 19:03:54.802: INFO @main_loop : Training epoch 193 2021-03-14 19:06:01.143: INFO @log_variables: train loss nanmean: 0.019977 2021-03-14 19:06:01.143: INFO @log_variables: train positive_distance nanmean: 0.554926 2021-03-14 19:06:01.143: INFO @log_variables: train negative_distance nanmean: 1.419523 2021-03-14 19:06:01.143: INFO @log_variables: train distance nanmean: 0.864596 2021-03-14 19:06:01.143: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:07:16.468: INFO @metrics_hook: train matching accuracy: 0.9915176835499128, N=110583, DB_SIZE=110592, N_IDS=11039 2021-03-14 19:07:16.666: INFO @decay_lr : LR updated to `0.0003800621` 2021-03-14 19:07:16.667: INFO @log_profile : T train: 124.004290 2021-03-14 19:07:16.667: INFO @log_profile : T read data: 1.972133 2021-03-14 19:07:16.667: INFO @log_profile : T hooks: 75.838574 2021-03-14 19:07:16.667: INFO @main_loop : Epoch 193 done 2021-03-14 19:07:16.667: INFO @main_loop : Training epoch 194 2021-03-14 19:09:23.491: INFO @log_variables: train loss nanmean: 0.019544 2021-03-14 19:09:23.492: INFO @log_variables: train positive_distance nanmean: 0.556112 2021-03-14 19:09:23.492: INFO @log_variables: train negative_distance nanmean: 1.419526 2021-03-14 19:09:23.492: INFO @log_variables: train distance nanmean: 0.863413 2021-03-14 19:09:23.492: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:10:39.800: INFO @metrics_hook: train matching accuracy: 0.9906948563987557, N=110584, DB_SIZE=110592, N_IDS=11096 2021-03-14 19:10:39.993: INFO @decay_lr : LR updated to `0.00037816179` 2021-03-14 19:10:39.995: INFO @log_profile : T train: 124.321801 2021-03-14 19:10:39.995: INFO @log_profile : T read data: 2.100463 2021-03-14 19:10:39.995: INFO @log_profile : T hooks: 76.854059 2021-03-14 19:10:39.995: INFO @main_loop : Epoch 194 done 2021-03-14 19:10:39.995: INFO @main_loop : Training epoch 195 2021-03-14 19:12:46.675: INFO @log_variables: train loss nanmean: 0.019905 2021-03-14 19:12:46.675: INFO @log_variables: train positive_distance nanmean: 0.555120 2021-03-14 19:12:46.675: INFO @log_variables: train negative_distance nanmean: 1.419510 2021-03-14 19:12:46.675: INFO @log_variables: train distance nanmean: 0.864390 2021-03-14 19:12:46.675: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:14:02.061: INFO @metrics_hook: train matching accuracy: 0.9909212572792708, N=110588, DB_SIZE=110592, N_IDS=11058 2021-03-14 19:14:02.257: INFO @decay_lr : LR updated to `0.00037627097` 2021-03-14 19:14:02.258: INFO @log_profile : T train: 124.066426 2021-03-14 19:14:02.258: INFO @log_profile : T read data: 2.218150 2021-03-14 19:14:02.258: INFO @log_profile : T hooks: 75.928134 2021-03-14 19:14:02.258: INFO @main_loop : Epoch 195 done 2021-03-14 19:14:02.258: INFO @main_loop : Training epoch 196 2021-03-14 19:16:07.609: INFO @log_variables: train loss nanmean: 0.019958 2021-03-14 19:16:07.609: INFO @log_variables: train positive_distance nanmean: 0.556155 2021-03-14 19:16:07.610: INFO @log_variables: train negative_distance nanmean: 1.419513 2021-03-14 19:16:07.610: INFO @log_variables: train distance nanmean: 0.863358 2021-03-14 19:16:07.610: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:17:24.964: INFO @metrics_hook: train matching accuracy: 0.9901884540258988, N=110584, DB_SIZE=110592, N_IDS=11101 2021-03-14 19:17:25.163: INFO @decay_lr : LR updated to `0.0003743896` 2021-03-14 19:17:25.164: INFO @log_profile : T train: 124.182292 2021-03-14 19:17:25.164: INFO @log_profile : T read data: 0.769582 2021-03-14 19:17:25.164: INFO @log_profile : T hooks: 77.903404 2021-03-14 19:17:25.164: INFO @main_loop : Epoch 196 done 2021-03-14 19:17:25.164: INFO @main_loop : Training epoch 197 2021-03-14 19:19:31.451: INFO @log_variables: train loss nanmean: 0.019915 2021-03-14 19:19:31.451: INFO @log_variables: train positive_distance nanmean: 0.554580 2021-03-14 19:19:31.451: INFO @log_variables: train negative_distance nanmean: 1.419510 2021-03-14 19:19:31.451: INFO @log_variables: train distance nanmean: 0.864930 2021-03-14 19:19:31.451: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:20:48.491: INFO @metrics_hook: train matching accuracy: 0.9905685116695453, N=110587, DB_SIZE=110592, N_IDS=11074 2021-03-14 19:20:48.681: INFO @decay_lr : LR updated to `0.00037251765` 2021-03-14 19:20:48.683: INFO @log_profile : T train: 123.848342 2021-03-14 19:20:48.683: INFO @log_profile : T read data: 2.039844 2021-03-14 19:20:48.683: INFO @log_profile : T hooks: 77.579987 2021-03-14 19:20:48.683: INFO @main_loop : Epoch 197 done 2021-03-14 19:20:48.683: INFO @main_loop : Training epoch 198 2021-03-14 19:22:55.529: INFO @log_variables: train loss nanmean: 0.018141 2021-03-14 19:22:55.529: INFO @log_variables: train positive_distance nanmean: 0.553561 2021-03-14 19:22:55.529: INFO @log_variables: train negative_distance nanmean: 1.419538 2021-03-14 19:22:55.529: INFO @log_variables: train distance nanmean: 0.865977 2021-03-14 19:22:55.529: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:24:10.018: INFO @metrics_hook: train matching accuracy: 0.9912918452607021, N=110586, DB_SIZE=110592, N_IDS=11095 2021-03-14 19:24:10.216: INFO @decay_lr : LR updated to `0.00037065506` 2021-03-14 19:24:10.218: INFO @log_profile : T train: 124.301342 2021-03-14 19:24:10.218: INFO @log_profile : T read data: 2.147308 2021-03-14 19:24:10.218: INFO @log_profile : T hooks: 75.035402 2021-03-14 19:24:10.218: INFO @main_loop : Epoch 198 done 2021-03-14 19:24:10.218: INFO @main_loop : Training epoch 199 2021-03-14 19:26:16.445: INFO @log_variables: train loss nanmean: 0.019697 2021-03-14 19:26:16.445: INFO @log_variables: train positive_distance nanmean: 0.554508 2021-03-14 19:26:16.445: INFO @log_variables: train negative_distance nanmean: 1.419527 2021-03-14 19:26:16.446: INFO @log_variables: train distance nanmean: 0.865019 2021-03-14 19:26:16.446: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:27:33.082: INFO @metrics_hook: train matching accuracy: 0.9899078503540392, N=110581, DB_SIZE=110592, N_IDS=11063 2021-03-14 19:27:33.281: INFO @decay_lr : LR updated to `0.0003688018` 2021-03-14 19:27:33.282: INFO @log_profile : T train: 123.719077 2021-03-14 19:27:33.282: INFO @log_profile : T read data: 2.110014 2021-03-14 19:27:33.282: INFO @log_profile : T hooks: 77.184667 2021-03-14 19:27:33.283: INFO @main_loop : Epoch 199 done 2021-03-14 19:27:33.283: INFO @main_loop : Training epoch 200 2021-03-14 19:29:39.936: INFO @log_variables: train loss nanmean: 0.019481 2021-03-14 19:29:39.936: INFO @log_variables: train positive_distance nanmean: 0.555244 2021-03-14 19:29:39.936: INFO @log_variables: train negative_distance nanmean: 1.419544 2021-03-14 19:29:39.936: INFO @log_variables: train distance nanmean: 0.864300 2021-03-14 19:29:39.936: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE 2021-03-14 19:30:55.403: INFO @metrics_hook: train matching accuracy: 0.9902070711637581, N=110590, DB_SIZE=110592, N_IDS=11080 2021-03-14 19:30:55.598: INFO @decay_lr : LR updated to `0.00036695777` 2021-03-14 19:30:55.847: INFO @save : Model saved to: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE/model_200.ckpt 2021-03-14 19:30:55.867: INFO @stop_after : EpochStopperHook triggered 2021-03-14 19:30:55.868: INFO @log_profile : T train: 124.180694 2021-03-14 19:30:55.868: INFO @log_profile : T read data: 2.071416 2021-03-14 19:30:55.868: INFO @log_profile : T hooks: 76.282134 2021-03-14 19:30:55.868: INFO @main_loop : Training terminated: Training terminated after epoch 200 2021-03-14 19:30:55.868: INFO @log_dir : Output dir: log-nightly/embedder/2021-03-14-nightly-embedder-04dafbcc-FdqE