2021-04-14 11:49:09.628: INFO @api : Creating an output dir with specified name: 2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:49:09.638: INFO @api : Created output directory with name log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:49:09.638: INFO @api : Creating dataset 2021-04-14 11:49:10.891: WARNING @lazy_loader : The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. 2021-04-14 11:49:10.892: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/utils/profiler.py:12: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2021-04-14 11:49:10.917: INFO @api : EmbeddingDatasetWrapper created 2021-04-14 11:49:10.917: INFO @api : Creating a model 2021-04-14 11:49:10.928: INFO @model : Creating TF model on 1 GPU devices 2021-04-14 11:49:10.929: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:427: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. 2021-04-14 11:49:10.951318: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3494015000 Hz 2021-04-14 11:49:10.952181: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556b53d649a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-04-14 11:49:10.952232: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-04-14 11:49:10.954130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2021-04-14 11:49:12.018359: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.019094: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556b55388120 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-04-14 11:49:12.019133: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1 2021-04-14 11:49:12.019360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.020343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62 pciBusID: 0000:41:00.0 2021-04-14 11:49:12.020758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2 2021-04-14 11:49:12.022592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-04-14 11:49:12.024332: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2021-04-14 11:49:12.024675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2021-04-14 11:49:12.026072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2021-04-14 11:49:12.026818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2021-04-14 11:49:12.029759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-04-14 11:49:12.029948: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.030776: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.031507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0 2021-04-14 11:49:12.031595: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.2 2021-04-14 11:49:12.032461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-04-14 11:49:12.032480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 2021-04-14 11:49:12.032487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N 2021-04-14 11:49:12.032594: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.033455: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-04-14 11:49:12.034174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10489 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:41:00.0, compute capability: 6.1) 2021-04-14 11:49:12.035: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2021-04-14 11:49:12.035: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead. 2021-04-14 11:49:12.036: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:142: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. 2021-04-14 11:49:12.036: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:143: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-04-14 11:49:12.037: WARNING @module_wrapper: From /home/kopalja/faces/models/embedding_body.py:59: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-04-14 11:49:12.042: WARNING @deprecation : From /usr/lib/python3.8/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. 2021-04-14 11:49:12.362: WARNING @deprecation : From /home/kopalja/faces/models/embedding_body.py:69: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.flatten instead. 2021-04-14 11:49:12.366: WARNING @deprecation : From /home/kopalja/faces/models/embedding_body.py:72: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.Dense instead. 2021-04-14 11:49:12.407: WARNING @deprecation : From /usr/lib/python3.8/site-packages/tensorflow_core/contrib/losses/python/metric_learning/metric_loss_ops.py:219: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where 2021-04-14 11:49:12.418: WARNING @module_wrapper: From /home/kopalja/faces/models/embedding_body.py:18: The name tf.diag_part is deprecated. Please use tf.linalg.tensor_diag_part instead. 2021-04-14 11:49:12.441: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:149: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead. 2021-04-14 11:49:12.441: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/graph_tower.py:80: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-04-14 11:49:13.254: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:182: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead. 2021-04-14 11:49:13.303: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:516: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead. 2021-04-14 11:49:13.942: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:517: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2021-04-14 11:49:14.017: INFO @model : Number of parameters: 1847248 2021-04-14 11:49:14.017: INFO @api : BodyEmbeddingModel created 2021-04-14 11:49:14.017: INFO @api : Creating hooks 2021-04-14 11:49:14.022: INFO @api : ComputeStats created 2021-04-14 11:49:14.023: INFO @api : LogVariables created 2021-04-14 11:49:14.024: INFO @api : LogProfile created 2021-04-14 11:49:14.024: INFO @api : LogDir created 2021-04-14 11:49:14.086: INFO @loader : Loading faiss with AVX2 support. 2021-04-14 11:49:14.086: INFO @loader : Loading faiss. 2021-04-14 11:49:14.199: INFO @api : ComputeMatchingMetrics created 2021-04-14 11:49:14.199: WARNING @abstract_hook: Argument `streams` was not recognized by `WriteTensorBoard`. Recognized arguments are `['model', 'output_dir', 'image_variables', 'flush_secs', 'visualize_graph', 'on_unknown_type', 'on_missing_variable', 'kwargs']`. 2021-04-14 11:49:14.199: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/write_tensorboard.py:68: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead. 2021-04-14 11:49:14.200: INFO @api : WriteTensorBoard created 2021-04-14 11:49:14.200: INFO @api : DecayLR created 2021-04-14 11:49:14.201: INFO @api : SaveEvery created 2021-04-14 11:49:14.201: INFO @api : StopAfter created 2021-04-14 11:49:14.202: INFO @api : ShowProgress created 2021-04-14 11:49:14.202: WARNING @api : TrainingTrace hook added between hooks. Add it to your config.yaml to suppress this warning. 2021-04-14 11:49:14.202: INFO @api : Creating main loop 2021-04-14 11:49:14.203: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:49:14.203: INFO @main_loop : Training epoch 1 2021-04-14 11:49:16.778: WARNING @main_loop : Some sources provided by the stream do not match model placeholders. Set `main_loop.on_unused_sources` to `ignore` in order to suppress this warning. Extra sources: ['filenames'] 2021-04-14 11:49:17.579321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2021-04-14 11:49:17.730029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2021-04-14 11:51:24.103: INFO @log_variables: train loss nanmean: 0.475797 2021-04-14 11:51:24.103: INFO @log_variables: train positive_distance nanmean: 0.613178 2021-04-14 11:51:24.103: INFO @log_variables: train negative_distance nanmean: 1.413681 2021-04-14 11:51:24.103: INFO @log_variables: train distance_margin nanmean: 0.800502 2021-04-14 11:51:24.103: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:52:44.499: INFO @metrics_hook: train matching accuracy: 0.8783231453683944, N=110588, DB_SIZE=110592, N_IDS=11545 2021-04-14 11:52:44.517: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/write_tensorboard.py:102: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead. 2021-04-14 11:52:44.565: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/hooks/decay_lr.py:89: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2021-04-14 11:52:44.641: INFO @decay_lr : LR updated to `0.000995` 2021-04-14 11:52:44.642: INFO @log_profile : T train: 126.006908 2021-04-14 11:52:44.642: INFO @log_profile : T read data: 3.555279 2021-04-14 11:52:44.642: INFO @log_profile : T hooks: 80.827957 2021-04-14 11:52:44.642: INFO @main_loop : Epoch 1 done 2021-04-14 11:52:44.642: INFO @main_loop : Training epoch 2 2021-04-14 11:54:51.921: INFO @log_variables: train loss nanmean: 0.301091 2021-04-14 11:54:51.921: INFO @log_variables: train positive_distance nanmean: 0.638571 2021-04-14 11:54:51.921: INFO @log_variables: train negative_distance nanmean: 1.417664 2021-04-14 11:54:51.921: INFO @log_variables: train distance_margin nanmean: 0.779093 2021-04-14 11:54:51.921: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:56:12.361: INFO @metrics_hook: train matching accuracy: 0.9511967952833515, N=110587, DB_SIZE=110592, N_IDS=11540 2021-04-14 11:56:12.482: INFO @decay_lr : LR updated to `0.000990025` 2021-04-14 11:56:12.484: INFO @log_profile : T train: 124.670061 2021-04-14 11:56:12.484: INFO @log_profile : T read data: 2.244555 2021-04-14 11:56:12.484: INFO @log_profile : T hooks: 80.876942 2021-04-14 11:56:12.485: INFO @main_loop : Epoch 2 done 2021-04-14 11:56:12.485: INFO @main_loop : Training epoch 3 2021-04-14 11:58:20.656: INFO @log_variables: train loss nanmean: 0.248061 2021-04-14 11:58:20.656: INFO @log_variables: train positive_distance nanmean: 0.634604 2021-04-14 11:58:20.656: INFO @log_variables: train negative_distance nanmean: 1.418140 2021-04-14 11:58:20.657: INFO @log_variables: train distance_margin nanmean: 0.783535 2021-04-14 11:58:20.657: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 11:59:51.147: INFO @metrics_hook: train matching accuracy: 0.96757157584417, N=110582, DB_SIZE=110592, N_IDS=11560 2021-04-14 11:59:51.293: INFO @decay_lr : LR updated to `0.0009850749` 2021-04-14 11:59:51.295: INFO @log_profile : T train: 125.438968 2021-04-14 11:59:51.295: INFO @log_profile : T read data: 2.386919 2021-04-14 11:59:51.295: INFO @log_profile : T hooks: 90.929746 2021-04-14 11:59:51.295: INFO @main_loop : Epoch 3 done 2021-04-14 11:59:51.295: INFO @main_loop : Training epoch 4 2021-04-14 12:02:03.117: INFO @log_variables: train loss nanmean: 0.220294 2021-04-14 12:02:03.117: INFO @log_variables: train positive_distance nanmean: 0.629867 2021-04-14 12:02:03.117: INFO @log_variables: train negative_distance nanmean: 1.418393 2021-04-14 12:02:03.118: INFO @log_variables: train distance_margin nanmean: 0.788526 2021-04-14 12:02:03.118: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:03:34.489: INFO @metrics_hook: train matching accuracy: 0.9740392440546162, N=110590, DB_SIZE=110592, N_IDS=11555 2021-04-14 12:03:34.652: INFO @decay_lr : LR updated to `0.0009801495` 2021-04-14 12:03:34.653: INFO @log_profile : T train: 129.151147 2021-04-14 12:03:34.653: INFO @log_profile : T read data: 2.306407 2021-04-14 12:03:34.654: INFO @log_profile : T hooks: 91.846215 2021-04-14 12:03:34.654: INFO @main_loop : Epoch 4 done 2021-04-14 12:03:34.654: INFO @main_loop : Training epoch 5 2021-04-14 12:05:45.977: INFO @log_variables: train loss nanmean: 0.198286 2021-04-14 12:05:45.977: INFO @log_variables: train positive_distance nanmean: 0.624809 2021-04-14 12:05:45.978: INFO @log_variables: train negative_distance nanmean: 1.418551 2021-04-14 12:05:45.978: INFO @log_variables: train distance_margin nanmean: 0.793742 2021-04-14 12:05:45.978: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:07:14.873: INFO @metrics_hook: train matching accuracy: 0.9773660318666015, N=110586, DB_SIZE=110592, N_IDS=11556 2021-04-14 12:07:15.042: INFO @decay_lr : LR updated to `0.00097524875` 2021-04-14 12:07:15.044: INFO @log_profile : T train: 128.500532 2021-04-14 12:07:15.044: INFO @log_profile : T read data: 2.492375 2021-04-14 12:07:15.044: INFO @log_profile : T hooks: 89.342834 2021-04-14 12:07:15.044: INFO @main_loop : Epoch 5 done 2021-04-14 12:07:15.044: INFO @main_loop : Training epoch 6 2021-04-14 12:09:27.733: INFO @log_variables: train loss nanmean: 0.180747 2021-04-14 12:09:27.733: INFO @log_variables: train positive_distance nanmean: 0.618702 2021-04-14 12:09:27.733: INFO @log_variables: train negative_distance nanmean: 1.418681 2021-04-14 12:09:27.733: INFO @log_variables: train distance_margin nanmean: 0.799979 2021-04-14 12:09:27.733: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:10:56.848: INFO @metrics_hook: train matching accuracy: 0.9800057875603625, N=110582, DB_SIZE=110592, N_IDS=11543 2021-04-14 12:10:56.981: INFO @decay_lr : LR updated to `0.0009703725` 2021-04-14 12:10:56.983: INFO @log_profile : T train: 129.944176 2021-04-14 12:10:56.983: INFO @log_profile : T read data: 2.403971 2021-04-14 12:10:56.983: INFO @log_profile : T hooks: 89.536738 2021-04-14 12:10:56.983: INFO @main_loop : Epoch 6 done 2021-04-14 12:10:56.983: INFO @main_loop : Training epoch 7 2021-04-14 12:13:11.116: INFO @log_variables: train loss nanmean: 0.168465 2021-04-14 12:13:11.116: INFO @log_variables: train positive_distance nanmean: 0.614347 2021-04-14 12:13:11.116: INFO @log_variables: train negative_distance nanmean: 1.418757 2021-04-14 12:13:11.117: INFO @log_variables: train distance_margin nanmean: 0.804410 2021-04-14 12:13:11.117: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:14:39.848: INFO @metrics_hook: train matching accuracy: 0.9825744902111498, N=110585, DB_SIZE=110592, N_IDS=11514 2021-04-14 12:14:39.991: INFO @decay_lr : LR updated to `0.0009655207` 2021-04-14 12:14:39.993: INFO @log_profile : T train: 131.362710 2021-04-14 12:14:39.993: INFO @log_profile : T read data: 2.403507 2021-04-14 12:14:39.993: INFO @log_profile : T hooks: 89.191318 2021-04-14 12:14:39.993: INFO @main_loop : Epoch 7 done 2021-04-14 12:14:39.993: INFO @main_loop : Training epoch 8 2021-04-14 12:16:52.870: INFO @log_variables: train loss nanmean: 0.154236 2021-04-14 12:16:52.870: INFO @log_variables: train positive_distance nanmean: 0.610962 2021-04-14 12:16:52.870: INFO @log_variables: train negative_distance nanmean: 1.418826 2021-04-14 12:16:52.870: INFO @log_variables: train distance_margin nanmean: 0.807865 2021-04-14 12:16:52.870: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:18:22.680: INFO @metrics_hook: train matching accuracy: 0.9835422205342449, N=110586, DB_SIZE=110592, N_IDS=11555 2021-04-14 12:18:22.835: INFO @decay_lr : LR updated to `0.00096069305` 2021-04-14 12:18:22.837: INFO @log_profile : T train: 131.629796 2021-04-14 12:18:22.837: INFO @log_profile : T read data: 0.911064 2021-04-14 12:18:22.837: INFO @log_profile : T hooks: 90.248338 2021-04-14 12:18:22.837: INFO @main_loop : Epoch 8 done 2021-04-14 12:18:22.837: INFO @main_loop : Training epoch 9 2021-04-14 12:20:34.275: INFO @log_variables: train loss nanmean: 0.150370 2021-04-14 12:20:34.275: INFO @log_variables: train positive_distance nanmean: 0.611500 2021-04-14 12:20:34.275: INFO @log_variables: train negative_distance nanmean: 1.418857 2021-04-14 12:20:34.275: INFO @log_variables: train distance_margin nanmean: 0.807357 2021-04-14 12:20:34.275: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:22:04.122: INFO @metrics_hook: train matching accuracy: 0.9835601573450287, N=110585, DB_SIZE=110592, N_IDS=11551 2021-04-14 12:22:04.279: INFO @decay_lr : LR updated to `0.0009558896` 2021-04-14 12:22:04.281: INFO @log_profile : T train: 128.798359 2021-04-14 12:22:04.281: INFO @log_profile : T read data: 2.306118 2021-04-14 12:22:04.281: INFO @log_profile : T hooks: 90.286524 2021-04-14 12:22:04.281: INFO @main_loop : Epoch 9 done 2021-04-14 12:22:04.281: INFO @main_loop : Training epoch 10 2021-04-14 12:24:17.093: INFO @log_variables: train loss nanmean: 0.142772 2021-04-14 12:24:17.093: INFO @log_variables: train positive_distance nanmean: 0.607944 2021-04-14 12:24:17.093: INFO @log_variables: train negative_distance nanmean: 1.418889 2021-04-14 12:24:17.093: INFO @log_variables: train distance_margin nanmean: 0.810944 2021-04-14 12:24:17.093: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:25:46.276: INFO @metrics_hook: train matching accuracy: 0.984482805393039, N=110587, DB_SIZE=110592, N_IDS=11542 2021-04-14 12:25:46.438: INFO @decay_lr : LR updated to `0.00095111015` 2021-04-14 12:25:46.439: INFO @log_profile : T train: 129.927016 2021-04-14 12:25:46.439: INFO @log_profile : T read data: 2.548915 2021-04-14 12:25:46.439: INFO @log_profile : T hooks: 89.627758 2021-04-14 12:25:46.440: INFO @main_loop : Epoch 10 done 2021-04-14 12:25:46.440: INFO @main_loop : Training epoch 11 2021-04-14 12:27:59.984: INFO @log_variables: train loss nanmean: 0.137392 2021-04-14 12:27:59.985: INFO @log_variables: train positive_distance nanmean: 0.606809 2021-04-14 12:27:59.985: INFO @log_variables: train negative_distance nanmean: 1.418943 2021-04-14 12:27:59.985: INFO @log_variables: train distance_margin nanmean: 0.812134 2021-04-14 12:27:59.985: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:29:27.818: INFO @metrics_hook: train matching accuracy: 0.984753947154251, N=110586, DB_SIZE=110592, N_IDS=11547 2021-04-14 12:29:27.971: INFO @decay_lr : LR updated to `0.0009463546` 2021-04-14 12:29:27.973: INFO @log_profile : T train: 130.698380 2021-04-14 12:29:27.973: INFO @log_profile : T read data: 2.496371 2021-04-14 12:29:27.973: INFO @log_profile : T hooks: 88.281770 2021-04-14 12:29:27.973: INFO @main_loop : Epoch 11 done 2021-04-14 12:29:27.973: INFO @main_loop : Training epoch 12 2021-04-14 12:31:40.572: INFO @log_variables: train loss nanmean: 0.128467 2021-04-14 12:31:40.572: INFO @log_variables: train positive_distance nanmean: 0.602991 2021-04-14 12:31:40.573: INFO @log_variables: train negative_distance nanmean: 1.418951 2021-04-14 12:31:40.573: INFO @log_variables: train distance_margin nanmean: 0.815959 2021-04-14 12:31:40.573: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:33:09.146: INFO @metrics_hook: train matching accuracy: 0.9863185879118892, N=110588, DB_SIZE=110592, N_IDS=11548 2021-04-14 12:33:09.284: INFO @decay_lr : LR updated to `0.0009416228` 2021-04-14 12:33:09.286: INFO @log_profile : T train: 129.904135 2021-04-14 12:33:09.286: INFO @log_profile : T read data: 2.335666 2021-04-14 12:33:09.286: INFO @log_profile : T hooks: 89.017880 2021-04-14 12:33:09.286: INFO @main_loop : Epoch 12 done 2021-04-14 12:33:09.286: INFO @main_loop : Training epoch 13 2021-04-14 12:35:23.129: INFO @log_variables: train loss nanmean: 0.124877 2021-04-14 12:35:23.129: INFO @log_variables: train positive_distance nanmean: 0.602866 2021-04-14 12:35:23.129: INFO @log_variables: train negative_distance nanmean: 1.418990 2021-04-14 12:35:23.129: INFO @log_variables: train distance_margin nanmean: 0.816123 2021-04-14 12:35:23.130: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:36:52.436: INFO @metrics_hook: train matching accuracy: 0.9857578196352194, N=110587, DB_SIZE=110592, N_IDS=11543 2021-04-14 12:36:52.591: INFO @decay_lr : LR updated to `0.0009369147` 2021-04-14 12:36:52.592: INFO @log_profile : T train: 131.157124 2021-04-14 12:36:52.592: INFO @log_profile : T read data: 2.339202 2021-04-14 12:36:52.592: INFO @log_profile : T hooks: 89.754406 2021-04-14 12:36:52.592: INFO @main_loop : Epoch 13 done 2021-04-14 12:36:52.592: INFO @main_loop : Training epoch 14 2021-04-14 12:39:05.606: INFO @log_variables: train loss nanmean: 0.119773 2021-04-14 12:39:05.606: INFO @log_variables: train positive_distance nanmean: 0.600746 2021-04-14 12:39:05.606: INFO @log_variables: train negative_distance nanmean: 1.419020 2021-04-14 12:39:05.606: INFO @log_variables: train distance_margin nanmean: 0.818274 2021-04-14 12:39:05.606: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:40:35.587: INFO @metrics_hook: train matching accuracy: 0.9862100770427171, N=110588, DB_SIZE=110592, N_IDS=11531 2021-04-14 12:40:35.749: INFO @decay_lr : LR updated to `0.00093223015` 2021-04-14 12:40:35.750: INFO @log_profile : T train: 130.314328 2021-04-14 12:40:35.750: INFO @log_profile : T read data: 2.342624 2021-04-14 12:40:35.750: INFO @log_profile : T hooks: 90.447039 2021-04-14 12:40:35.750: INFO @main_loop : Epoch 14 done 2021-04-14 12:40:35.750: INFO @main_loop : Training epoch 15 2021-04-14 12:42:51.122: INFO @log_variables: train loss nanmean: 0.113959 2021-04-14 12:42:51.122: INFO @log_variables: train positive_distance nanmean: 0.599175 2021-04-14 12:42:51.122: INFO @log_variables: train negative_distance nanmean: 1.419038 2021-04-14 12:42:51.122: INFO @log_variables: train distance_margin nanmean: 0.819863 2021-04-14 12:42:51.122: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:44:19.129: INFO @metrics_hook: train matching accuracy: 0.9871950226980883, N=110582, DB_SIZE=110592, N_IDS=11553 2021-04-14 12:44:19.302: INFO @decay_lr : LR updated to `0.000927569` 2021-04-14 12:44:19.304: INFO @log_profile : T train: 134.123049 2021-04-14 12:44:19.304: INFO @log_profile : T read data: 0.901580 2021-04-14 12:44:19.304: INFO @log_profile : T hooks: 88.472868 2021-04-14 12:44:19.304: INFO @main_loop : Epoch 15 done 2021-04-14 12:44:19.304: INFO @main_loop : Training epoch 16 2021-04-14 12:46:32.151: INFO @log_variables: train loss nanmean: 0.109997 2021-04-14 12:46:32.152: INFO @log_variables: train positive_distance nanmean: 0.596859 2021-04-14 12:46:32.152: INFO @log_variables: train negative_distance nanmean: 1.419066 2021-04-14 12:46:32.152: INFO @log_variables: train distance_margin nanmean: 0.822207 2021-04-14 12:46:32.152: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:48:01.225: INFO @metrics_hook: train matching accuracy: 0.9875657882838075, N=110582, DB_SIZE=110592, N_IDS=11564 2021-04-14 12:48:01.375: INFO @decay_lr : LR updated to `0.0009229311` 2021-04-14 12:48:01.376: INFO @log_profile : T train: 130.072429 2021-04-14 12:48:01.376: INFO @log_profile : T read data: 2.435897 2021-04-14 12:48:01.376: INFO @log_profile : T hooks: 89.507722 2021-04-14 12:48:01.376: INFO @main_loop : Epoch 16 done 2021-04-14 12:48:01.376: INFO @main_loop : Training epoch 17 2021-04-14 12:50:19.919: INFO @log_variables: train loss nanmean: 0.107551 2021-04-14 12:50:19.919: INFO @log_variables: train positive_distance nanmean: 0.597288 2021-04-14 12:50:19.919: INFO @log_variables: train negative_distance nanmean: 1.419072 2021-04-14 12:50:19.919: INFO @log_variables: train distance_margin nanmean: 0.821784 2021-04-14 12:50:19.920: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:51:48.602: INFO @metrics_hook: train matching accuracy: 0.9868697099037835, N=110584, DB_SIZE=110592, N_IDS=11571 2021-04-14 12:51:48.764: INFO @decay_lr : LR updated to `0.0009183165` 2021-04-14 12:51:48.766: INFO @log_profile : T train: 135.783994 2021-04-14 12:51:48.766: INFO @log_profile : T read data: 2.405189 2021-04-14 12:51:48.766: INFO @log_profile : T hooks: 89.146442 2021-04-14 12:51:48.766: INFO @main_loop : Epoch 17 done 2021-04-14 12:51:48.766: INFO @main_loop : Training epoch 18 2021-04-14 12:54:03.441: INFO @log_variables: train loss nanmean: 0.102573 2021-04-14 12:54:03.441: INFO @log_variables: train positive_distance nanmean: 0.595030 2021-04-14 12:54:03.441: INFO @log_variables: train negative_distance nanmean: 1.419105 2021-04-14 12:54:03.442: INFO @log_variables: train distance_margin nanmean: 0.824074 2021-04-14 12:54:03.442: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:55:32.480: INFO @metrics_hook: train matching accuracy: 0.9874579520381959, N=110588, DB_SIZE=110592, N_IDS=11557 2021-04-14 12:55:32.620: INFO @decay_lr : LR updated to `0.0009137249` 2021-04-14 12:55:32.621: INFO @log_profile : T train: 131.797401 2021-04-14 12:55:32.621: INFO @log_profile : T read data: 2.552387 2021-04-14 12:55:32.621: INFO @log_profile : T hooks: 89.450770 2021-04-14 12:55:32.621: INFO @main_loop : Epoch 18 done 2021-04-14 12:55:32.621: INFO @main_loop : Training epoch 19 2021-04-14 12:57:47.713: INFO @log_variables: train loss nanmean: 0.099183 2021-04-14 12:57:47.714: INFO @log_variables: train positive_distance nanmean: 0.593102 2021-04-14 12:57:47.714: INFO @log_variables: train negative_distance nanmean: 1.419105 2021-04-14 12:57:47.714: INFO @log_variables: train distance_margin nanmean: 0.826002 2021-04-14 12:57:47.714: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 12:59:16.335: INFO @metrics_hook: train matching accuracy: 0.9876385799287433, N=110586, DB_SIZE=110592, N_IDS=11541 2021-04-14 12:59:16.479: INFO @decay_lr : LR updated to `0.0009091563` 2021-04-14 12:59:16.480: INFO @log_profile : T train: 132.323781 2021-04-14 12:59:16.480: INFO @log_profile : T read data: 2.402296 2021-04-14 12:59:16.480: INFO @log_profile : T hooks: 89.076729 2021-04-14 12:59:16.480: INFO @main_loop : Epoch 19 done 2021-04-14 12:59:16.480: INFO @main_loop : Training epoch 20 2021-04-14 13:01:30.450: INFO @log_variables: train loss nanmean: 0.095320 2021-04-14 13:01:30.450: INFO @log_variables: train positive_distance nanmean: 0.591818 2021-04-14 13:01:30.450: INFO @log_variables: train negative_distance nanmean: 1.419141 2021-04-14 13:01:30.450: INFO @log_variables: train distance_margin nanmean: 0.827323 2021-04-14 13:01:30.450: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:03:00.614: INFO @metrics_hook: train matching accuracy: 0.9877111440662639, N=110588, DB_SIZE=110592, N_IDS=11541 2021-04-14 13:03:00.784: INFO @decay_lr : LR updated to `0.0009046105` 2021-04-14 13:03:00.786: INFO @log_profile : T train: 131.129733 2021-04-14 13:03:00.786: INFO @log_profile : T read data: 2.471346 2021-04-14 13:03:00.786: INFO @log_profile : T hooks: 90.647368 2021-04-14 13:03:00.786: INFO @main_loop : Epoch 20 done 2021-04-14 13:03:00.786: INFO @main_loop : Training epoch 21 2021-04-14 13:05:16.176: INFO @log_variables: train loss nanmean: 0.094584 2021-04-14 13:05:16.176: INFO @log_variables: train positive_distance nanmean: 0.591373 2021-04-14 13:05:16.176: INFO @log_variables: train negative_distance nanmean: 1.419124 2021-04-14 13:05:16.176: INFO @log_variables: train distance_margin nanmean: 0.827751 2021-04-14 13:05:16.177: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:06:46.085: INFO @metrics_hook: train matching accuracy: 0.9872403533997088, N=110583, DB_SIZE=110592, N_IDS=11547 2021-04-14 13:06:46.234: INFO @decay_lr : LR updated to `0.00090008747` 2021-04-14 13:06:46.235: INFO @log_profile : T train: 132.678197 2021-04-14 13:06:46.235: INFO @log_profile : T read data: 2.378499 2021-04-14 13:06:46.235: INFO @log_profile : T hooks: 90.337528 2021-04-14 13:06:46.235: INFO @main_loop : Epoch 21 done 2021-04-14 13:06:46.236: INFO @main_loop : Training epoch 22 2021-04-14 13:09:00.289: INFO @log_variables: train loss nanmean: 0.093674 2021-04-14 13:09:00.289: INFO @log_variables: train positive_distance nanmean: 0.592770 2021-04-14 13:09:00.290: INFO @log_variables: train negative_distance nanmean: 1.419139 2021-04-14 13:09:00.290: INFO @log_variables: train distance_margin nanmean: 0.826369 2021-04-14 13:09:00.290: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:10:28.595: INFO @metrics_hook: train matching accuracy: 0.9887962527579846, N=110588, DB_SIZE=110592, N_IDS=11586 2021-04-14 13:10:28.749: INFO @decay_lr : LR updated to `0.000895587` 2021-04-14 13:10:28.751: INFO @log_profile : T train: 131.448717 2021-04-14 13:10:28.751: INFO @log_profile : T read data: 2.248081 2021-04-14 13:10:28.751: INFO @log_profile : T hooks: 88.765363 2021-04-14 13:10:28.751: INFO @main_loop : Epoch 22 done 2021-04-14 13:10:28.751: INFO @main_loop : Training epoch 23 2021-04-14 13:12:40.926: INFO @log_variables: train loss nanmean: 0.089638 2021-04-14 13:12:40.926: INFO @log_variables: train positive_distance nanmean: 0.590809 2021-04-14 13:12:40.926: INFO @log_variables: train negative_distance nanmean: 1.419145 2021-04-14 13:12:40.926: INFO @log_variables: train distance_margin nanmean: 0.828336 2021-04-14 13:12:40.926: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:14:09.582: INFO @metrics_hook: train matching accuracy: 0.987855715409587, N=110587, DB_SIZE=110592, N_IDS=11525 2021-04-14 13:14:09.735: INFO @decay_lr : LR updated to `0.0008911091` 2021-04-14 13:14:09.737: INFO @log_profile : T train: 130.956598 2021-04-14 13:14:09.737: INFO @log_profile : T read data: 0.882493 2021-04-14 13:14:09.737: INFO @log_profile : T hooks: 89.092612 2021-04-14 13:14:09.737: INFO @main_loop : Epoch 23 done 2021-04-14 13:14:09.737: INFO @main_loop : Training epoch 24 2021-04-14 13:16:25.091: INFO @log_variables: train loss nanmean: 0.086659 2021-04-14 13:16:25.091: INFO @log_variables: train positive_distance nanmean: 0.589644 2021-04-14 13:16:25.091: INFO @log_variables: train negative_distance nanmean: 1.419181 2021-04-14 13:16:25.092: INFO @log_variables: train distance_margin nanmean: 0.829536 2021-04-14 13:16:25.092: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:17:54.051: INFO @metrics_hook: train matching accuracy: 0.9884608428287213, N=110580, DB_SIZE=110592, N_IDS=11540 2021-04-14 13:17:54.225: INFO @decay_lr : LR updated to `0.00088665355` 2021-04-14 13:17:54.226: INFO @log_profile : T train: 132.612520 2021-04-14 13:17:54.226: INFO @log_profile : T read data: 2.404431 2021-04-14 13:17:54.226: INFO @log_profile : T hooks: 89.416106 2021-04-14 13:17:54.226: INFO @main_loop : Epoch 24 done 2021-04-14 13:17:54.226: INFO @main_loop : Training epoch 25 2021-04-14 13:20:09.967: INFO @log_variables: train loss nanmean: 0.085978 2021-04-14 13:20:09.968: INFO @log_variables: train positive_distance nanmean: 0.589327 2021-04-14 13:20:09.968: INFO @log_variables: train negative_distance nanmean: 1.419202 2021-04-14 13:20:09.968: INFO @log_variables: train distance_margin nanmean: 0.829875 2021-04-14 13:20:09.968: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:21:39.335: INFO @metrics_hook: train matching accuracy: 0.9885518962617329, N=110586, DB_SIZE=110592, N_IDS=11527 2021-04-14 13:21:39.478: INFO @decay_lr : LR updated to `0.0008822203` 2021-04-14 13:21:39.480: INFO @log_profile : T train: 133.148325 2021-04-14 13:21:39.480: INFO @log_profile : T read data: 2.239634 2021-04-14 13:21:39.480: INFO @log_profile : T hooks: 89.809121 2021-04-14 13:21:39.480: INFO @main_loop : Epoch 25 done 2021-04-14 13:21:39.480: INFO @main_loop : Training epoch 26 2021-04-14 13:23:53.689: INFO @log_variables: train loss nanmean: 0.081830 2021-04-14 13:23:53.689: INFO @log_variables: train positive_distance nanmean: 0.587264 2021-04-14 13:23:53.689: INFO @log_variables: train negative_distance nanmean: 1.419188 2021-04-14 13:23:53.689: INFO @log_variables: train distance_margin nanmean: 0.831924 2021-04-14 13:23:53.689: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:25:23.020: INFO @metrics_hook: train matching accuracy: 0.9878464529547407, N=110585, DB_SIZE=110592, N_IDS=11568 2021-04-14 13:25:23.169: INFO @decay_lr : LR updated to `0.0008778092` 2021-04-14 13:25:23.170: INFO @log_profile : T train: 131.398989 2021-04-14 13:25:23.170: INFO @log_profile : T read data: 2.476398 2021-04-14 13:25:23.170: INFO @log_profile : T hooks: 89.759977 2021-04-14 13:25:23.170: INFO @main_loop : Epoch 26 done 2021-04-14 13:25:23.170: INFO @main_loop : Training epoch 27 2021-04-14 13:27:36.772: INFO @log_variables: train loss nanmean: 0.080328 2021-04-14 13:27:36.772: INFO @log_variables: train positive_distance nanmean: 0.586923 2021-04-14 13:27:36.772: INFO @log_variables: train negative_distance nanmean: 1.419207 2021-04-14 13:27:36.772: INFO @log_variables: train distance_margin nanmean: 0.832284 2021-04-14 13:27:36.772: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:29:05.462: INFO @metrics_hook: train matching accuracy: 0.9888050928688984, N=110586, DB_SIZE=110592, N_IDS=11557 2021-04-14 13:29:05.626: INFO @decay_lr : LR updated to `0.00087342015` 2021-04-14 13:29:05.628: INFO @log_profile : T train: 130.845925 2021-04-14 13:29:05.628: INFO @log_profile : T read data: 2.400284 2021-04-14 13:29:05.628: INFO @log_profile : T hooks: 89.156489 2021-04-14 13:29:05.628: INFO @main_loop : Epoch 27 done 2021-04-14 13:29:05.628: INFO @main_loop : Training epoch 28 2021-04-14 13:31:19.108: INFO @log_variables: train loss nanmean: 0.078217 2021-04-14 13:31:19.108: INFO @log_variables: train positive_distance nanmean: 0.584607 2021-04-14 13:31:19.108: INFO @log_variables: train negative_distance nanmean: 1.419204 2021-04-14 13:31:19.108: INFO @log_variables: train distance_margin nanmean: 0.834597 2021-04-14 13:31:19.108: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:32:52.365: INFO @metrics_hook: train matching accuracy: 0.988135930407104, N=110586, DB_SIZE=110592, N_IDS=11584 2021-04-14 13:32:52.536: INFO @decay_lr : LR updated to `0.00086905307` 2021-04-14 13:32:52.537: INFO @log_profile : T train: 130.644784 2021-04-14 13:32:52.537: INFO @log_profile : T read data: 2.491912 2021-04-14 13:32:52.537: INFO @log_profile : T hooks: 93.714297 2021-04-14 13:32:52.537: INFO @main_loop : Epoch 28 done 2021-04-14 13:32:52.537: INFO @main_loop : Training epoch 29 2021-04-14 13:35:07.880: INFO @log_variables: train loss nanmean: 0.075811 2021-04-14 13:35:07.880: INFO @log_variables: train positive_distance nanmean: 0.583663 2021-04-14 13:35:07.880: INFO @log_variables: train negative_distance nanmean: 1.419224 2021-04-14 13:35:07.880: INFO @log_variables: train distance_margin nanmean: 0.835561 2021-04-14 13:35:07.881: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:36:37.465: INFO @metrics_hook: train matching accuracy: 0.9891308279379318, N=110588, DB_SIZE=110592, N_IDS=11539 2021-04-14 13:36:37.622: INFO @decay_lr : LR updated to `0.0008647078` 2021-04-14 13:36:37.623: INFO @log_profile : T train: 132.553652 2021-04-14 13:36:37.623: INFO @log_profile : T read data: 2.434162 2021-04-14 13:36:37.623: INFO @log_profile : T hooks: 90.040917 2021-04-14 13:36:37.623: INFO @main_loop : Epoch 29 done 2021-04-14 13:36:37.623: INFO @main_loop : Training epoch 30 2021-04-14 13:38:49.590: INFO @log_variables: train loss nanmean: 0.074661 2021-04-14 13:38:49.590: INFO @log_variables: train positive_distance nanmean: 0.583715 2021-04-14 13:38:49.590: INFO @log_variables: train negative_distance nanmean: 1.419207 2021-04-14 13:38:49.590: INFO @log_variables: train distance_margin nanmean: 0.835492 2021-04-14 13:38:49.591: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:40:18.731: INFO @metrics_hook: train matching accuracy: 0.9892393388071038, N=110588, DB_SIZE=110592, N_IDS=11555 2021-04-14 13:40:18.897: INFO @decay_lr : LR updated to `0.00086038426` 2021-04-14 13:40:18.899: INFO @log_profile : T train: 130.704774 2021-04-14 13:40:18.899: INFO @log_profile : T read data: 0.897811 2021-04-14 13:40:18.899: INFO @log_profile : T hooks: 89.616422 2021-04-14 13:40:18.899: INFO @main_loop : Epoch 30 done 2021-04-14 13:40:18.899: INFO @main_loop : Training epoch 31 2021-04-14 13:42:33.457: INFO @log_variables: train loss nanmean: 0.074239 2021-04-14 13:42:33.458: INFO @log_variables: train positive_distance nanmean: 0.584916 2021-04-14 13:42:33.458: INFO @log_variables: train negative_distance nanmean: 1.419197 2021-04-14 13:42:33.458: INFO @log_variables: train distance_margin nanmean: 0.834281 2021-04-14 13:42:33.458: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:44:06.112: INFO @metrics_hook: train matching accuracy: 0.9891215886278553, N=110586, DB_SIZE=110592, N_IDS=11545 2021-04-14 13:44:06.277: INFO @decay_lr : LR updated to `0.00085608236` 2021-04-14 13:44:06.278: INFO @log_profile : T train: 131.700566 2021-04-14 13:44:06.278: INFO @log_profile : T read data: 2.498877 2021-04-14 13:44:06.278: INFO @log_profile : T hooks: 93.124416 2021-04-14 13:44:06.278: INFO @main_loop : Epoch 31 done 2021-04-14 13:44:06.278: INFO @main_loop : Training epoch 32 2021-04-14 13:46:21.722: INFO @log_variables: train loss nanmean: 0.070872 2021-04-14 13:46:21.723: INFO @log_variables: train positive_distance nanmean: 0.583457 2021-04-14 13:46:21.723: INFO @log_variables: train negative_distance nanmean: 1.419235 2021-04-14 13:46:21.723: INFO @log_variables: train distance_margin nanmean: 0.835778 2021-04-14 13:46:21.723: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:47:52.864: INFO @metrics_hook: train matching accuracy: 0.9890582894760639, N=110586, DB_SIZE=110592, N_IDS=11544 2021-04-14 13:47:53.032: INFO @decay_lr : LR updated to `0.00085180195` 2021-04-14 13:47:53.033: INFO @log_profile : T train: 132.584217 2021-04-14 13:47:53.033: INFO @log_profile : T read data: 2.499381 2021-04-14 13:47:53.033: INFO @log_profile : T hooks: 91.613298 2021-04-14 13:47:53.033: INFO @main_loop : Epoch 32 done 2021-04-14 13:47:53.033: INFO @main_loop : Training epoch 33 2021-04-14 13:50:07.066: INFO @log_variables: train loss nanmean: 0.069096 2021-04-14 13:50:07.067: INFO @log_variables: train positive_distance nanmean: 0.581182 2021-04-14 13:50:07.067: INFO @log_variables: train negative_distance nanmean: 1.419254 2021-04-14 13:50:07.067: INFO @log_variables: train distance_margin nanmean: 0.838071 2021-04-14 13:50:07.067: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:51:39.487: INFO @metrics_hook: train matching accuracy: 0.9884345498607444, N=110588, DB_SIZE=110592, N_IDS=11554 2021-04-14 13:51:39.684: INFO @decay_lr : LR updated to `0.00084754295` 2021-04-14 13:51:39.685: INFO @log_profile : T train: 131.131352 2021-04-14 13:51:39.685: INFO @log_profile : T read data: 2.572284 2021-04-14 13:51:39.685: INFO @log_profile : T hooks: 92.895350 2021-04-14 13:51:39.686: INFO @main_loop : Epoch 33 done 2021-04-14 13:51:39.686: INFO @main_loop : Training epoch 34 2021-04-14 13:53:58.826: INFO @log_variables: train loss nanmean: 0.069081 2021-04-14 13:53:58.826: INFO @log_variables: train positive_distance nanmean: 0.582433 2021-04-14 13:53:58.827: INFO @log_variables: train negative_distance nanmean: 1.419237 2021-04-14 13:53:58.827: INFO @log_variables: train distance_margin nanmean: 0.836804 2021-04-14 13:53:58.827: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:55:30.572: INFO @metrics_hook: train matching accuracy: 0.9892843578753188, N=110586, DB_SIZE=110592, N_IDS=11558 2021-04-14 13:55:30.748: INFO @decay_lr : LR updated to `0.00084330526` 2021-04-14 13:55:30.750: INFO @log_profile : T train: 136.283308 2021-04-14 13:55:30.750: INFO @log_profile : T read data: 2.508703 2021-04-14 13:55:30.750: INFO @log_profile : T hooks: 92.218160 2021-04-14 13:55:30.750: INFO @main_loop : Epoch 34 done 2021-04-14 13:55:30.750: INFO @main_loop : Training epoch 35 2021-04-14 13:57:49.400: INFO @log_variables: train loss nanmean: 0.066530 2021-04-14 13:57:49.400: INFO @log_variables: train positive_distance nanmean: 0.581845 2021-04-14 13:57:49.401: INFO @log_variables: train negative_distance nanmean: 1.419273 2021-04-14 13:57:49.401: INFO @log_variables: train distance_margin nanmean: 0.837427 2021-04-14 13:57:49.401: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 13:59:21.938: INFO @metrics_hook: train matching accuracy: 0.9889673632902578, N=110581, DB_SIZE=110592, N_IDS=11552 2021-04-14 13:59:22.117: INFO @decay_lr : LR updated to `0.00083908875` 2021-04-14 13:59:22.118: INFO @log_profile : T train: 135.879583 2021-04-14 13:59:22.118: INFO @log_profile : T read data: 2.420819 2021-04-14 13:59:22.118: INFO @log_profile : T hooks: 93.013811 2021-04-14 13:59:22.118: INFO @main_loop : Epoch 35 done 2021-04-14 13:59:22.118: INFO @main_loop : Training epoch 36 2021-04-14 14:01:37.514: INFO @log_variables: train loss nanmean: 0.066695 2021-04-14 14:01:37.514: INFO @log_variables: train positive_distance nanmean: 0.581441 2021-04-14 14:01:37.514: INFO @log_variables: train negative_distance nanmean: 1.419254 2021-04-14 14:01:37.514: INFO @log_variables: train distance_margin nanmean: 0.837813 2021-04-14 14:01:37.514: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:03:06.555: INFO @metrics_hook: train matching accuracy: 0.9887505312751508, N=110583, DB_SIZE=110592, N_IDS=11539 2021-04-14 14:03:06.714: INFO @decay_lr : LR updated to `0.0008348933` 2021-04-14 14:03:06.716: INFO @log_profile : T train: 132.487012 2021-04-14 14:03:06.716: INFO @log_profile : T read data: 2.577039 2021-04-14 14:03:06.716: INFO @log_profile : T hooks: 89.478367 2021-04-14 14:03:06.716: INFO @main_loop : Epoch 36 done 2021-04-14 14:03:06.716: INFO @main_loop : Training epoch 37 2021-04-14 14:05:24.382: INFO @log_variables: train loss nanmean: 0.064872 2021-04-14 14:05:24.383: INFO @log_variables: train positive_distance nanmean: 0.580627 2021-04-14 14:05:24.383: INFO @log_variables: train negative_distance nanmean: 1.419285 2021-04-14 14:05:24.383: INFO @log_variables: train distance_margin nanmean: 0.838658 2021-04-14 14:05:24.383: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:06:53.587: INFO @metrics_hook: train matching accuracy: 0.9888865779883711, N=110587, DB_SIZE=110592, N_IDS=11568 2021-04-14 14:06:53.745: INFO @decay_lr : LR updated to `0.00083071884` 2021-04-14 14:06:53.747: INFO @log_profile : T train: 136.425097 2021-04-14 14:06:53.747: INFO @log_profile : T read data: 0.898020 2021-04-14 14:06:53.747: INFO @log_profile : T hooks: 89.654203 2021-04-14 14:06:53.747: INFO @main_loop : Epoch 37 done 2021-04-14 14:06:53.748: INFO @main_loop : Training epoch 38 2021-04-14 14:09:09.265: INFO @log_variables: train loss nanmean: 0.062663 2021-04-14 14:09:09.267: INFO @log_variables: train positive_distance nanmean: 0.579494 2021-04-14 14:09:09.268: INFO @log_variables: train negative_distance nanmean: 1.419295 2021-04-14 14:09:09.268: INFO @log_variables: train distance_margin nanmean: 0.839802 2021-04-14 14:09:09.268: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:10:37.200: INFO @metrics_hook: train matching accuracy: 0.9895559232835092, N=110589, DB_SIZE=110592, N_IDS=11530 2021-04-14 14:10:37.361: INFO @decay_lr : LR updated to `0.00082656526` 2021-04-14 14:10:37.362: INFO @log_profile : T train: 132.892452 2021-04-14 14:10:37.362: INFO @log_profile : T read data: 2.292933 2021-04-14 14:10:37.362: INFO @log_profile : T hooks: 88.375478 2021-04-14 14:10:37.362: INFO @main_loop : Epoch 38 done 2021-04-14 14:10:37.362: INFO @main_loop : Training epoch 39 2021-04-14 14:12:52.864: INFO @log_variables: train loss nanmean: 0.062730 2021-04-14 14:12:52.864: INFO @log_variables: train positive_distance nanmean: 0.579409 2021-04-14 14:12:52.864: INFO @log_variables: train negative_distance nanmean: 1.419270 2021-04-14 14:12:52.864: INFO @log_variables: train distance_margin nanmean: 0.839862 2021-04-14 14:12:52.864: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:14:22.532: INFO @metrics_hook: train matching accuracy: 0.9902068055015508, N=110587, DB_SIZE=110592, N_IDS=11546 2021-04-14 14:14:22.693: INFO @decay_lr : LR updated to `0.00082243246` 2021-04-14 14:14:22.695: INFO @log_profile : T train: 132.618127 2021-04-14 14:14:22.695: INFO @log_profile : T read data: 2.534338 2021-04-14 14:14:22.695: INFO @log_profile : T hooks: 90.126772 2021-04-14 14:14:22.695: INFO @main_loop : Epoch 39 done 2021-04-14 14:14:22.695: INFO @main_loop : Training epoch 40 2021-04-14 14:16:36.104: INFO @log_variables: train loss nanmean: 0.061032 2021-04-14 14:16:36.104: INFO @log_variables: train positive_distance nanmean: 0.578602 2021-04-14 14:16:36.104: INFO @log_variables: train negative_distance nanmean: 1.419300 2021-04-14 14:16:36.104: INFO @log_variables: train distance_margin nanmean: 0.840698 2021-04-14 14:16:36.105: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:18:04.916: INFO @metrics_hook: train matching accuracy: 0.9896370245781564, N=110586, DB_SIZE=110592, N_IDS=11550 2021-04-14 14:18:05.083: INFO @decay_lr : LR updated to `0.0008183203` 2021-04-14 14:18:05.085: INFO @log_profile : T train: 130.589005 2021-04-14 14:18:05.085: INFO @log_profile : T read data: 2.473560 2021-04-14 14:18:05.085: INFO @log_profile : T hooks: 89.271169 2021-04-14 14:18:05.085: INFO @main_loop : Epoch 40 done 2021-04-14 14:18:05.085: INFO @main_loop : Training epoch 41 2021-04-14 14:20:18.393: INFO @log_variables: train loss nanmean: 0.059786 2021-04-14 14:20:18.393: INFO @log_variables: train positive_distance nanmean: 0.578295 2021-04-14 14:20:18.393: INFO @log_variables: train negative_distance nanmean: 1.419278 2021-04-14 14:20:18.393: INFO @log_variables: train distance_margin nanmean: 0.840984 2021-04-14 14:20:18.393: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:21:48.055: INFO @metrics_hook: train matching accuracy: 0.9893207219589829, N=110588, DB_SIZE=110592, N_IDS=11563 2021-04-14 14:21:48.218: INFO @decay_lr : LR updated to `0.0008142287` 2021-04-14 14:21:48.220: INFO @log_profile : T train: 130.417336 2021-04-14 14:21:48.220: INFO @log_profile : T read data: 2.547500 2021-04-14 14:21:48.220: INFO @log_profile : T hooks: 90.115494 2021-04-14 14:21:48.220: INFO @main_loop : Epoch 41 done 2021-04-14 14:21:48.220: INFO @main_loop : Training epoch 42 2021-04-14 14:24:01.421: INFO @log_variables: train loss nanmean: 0.059063 2021-04-14 14:24:01.421: INFO @log_variables: train positive_distance nanmean: 0.578706 2021-04-14 14:24:01.422: INFO @log_variables: train negative_distance nanmean: 1.419280 2021-04-14 14:24:01.422: INFO @log_variables: train distance_margin nanmean: 0.840574 2021-04-14 14:24:01.422: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:25:31.220: INFO @metrics_hook: train matching accuracy: 0.989447317973017, N=110588, DB_SIZE=110592, N_IDS=11573 2021-04-14 14:25:31.374: INFO @decay_lr : LR updated to `0.00081015757` 2021-04-14 14:25:31.376: INFO @log_profile : T train: 130.433170 2021-04-14 14:25:31.376: INFO @log_profile : T read data: 2.432960 2021-04-14 14:25:31.376: INFO @log_profile : T hooks: 90.236012 2021-04-14 14:25:31.377: INFO @main_loop : Epoch 42 done 2021-04-14 14:25:31.377: INFO @main_loop : Training epoch 43 2021-04-14 14:27:46.931: INFO @log_variables: train loss nanmean: 0.059372 2021-04-14 14:27:46.932: INFO @log_variables: train positive_distance nanmean: 0.578521 2021-04-14 14:27:46.932: INFO @log_variables: train negative_distance nanmean: 1.419293 2021-04-14 14:27:46.932: INFO @log_variables: train distance_margin nanmean: 0.840773 2021-04-14 14:27:46.932: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:29:16.002: INFO @metrics_hook: train matching accuracy: 0.989221156193766, N=110587, DB_SIZE=110592, N_IDS=11525 2021-04-14 14:29:16.176: INFO @decay_lr : LR updated to `0.0008061068` 2021-04-14 14:29:16.178: INFO @log_profile : T train: 132.878249 2021-04-14 14:29:16.178: INFO @log_profile : T read data: 2.326252 2021-04-14 14:29:16.179: INFO @log_profile : T hooks: 89.543753 2021-04-14 14:29:16.179: INFO @main_loop : Epoch 43 done 2021-04-14 14:29:16.179: INFO @main_loop : Training epoch 44 2021-04-14 14:31:30.693: INFO @log_variables: train loss nanmean: 0.056108 2021-04-14 14:31:30.694: INFO @log_variables: train positive_distance nanmean: 0.576149 2021-04-14 14:31:30.694: INFO @log_variables: train negative_distance nanmean: 1.419285 2021-04-14 14:31:30.694: INFO @log_variables: train distance_margin nanmean: 0.843136 2021-04-14 14:31:30.694: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:32:59.734: INFO @metrics_hook: train matching accuracy: 0.9897099247685185, N=110592, DB_SIZE=110592, N_IDS=11514 2021-04-14 14:32:59.895: INFO @decay_lr : LR updated to `0.00080207625` 2021-04-14 14:32:59.896: INFO @log_profile : T train: 131.824000 2021-04-14 14:32:59.896: INFO @log_profile : T read data: 2.354257 2021-04-14 14:32:59.896: INFO @log_profile : T hooks: 89.485242 2021-04-14 14:32:59.896: INFO @main_loop : Epoch 44 done 2021-04-14 14:32:59.896: INFO @main_loop : Training epoch 45 2021-04-14 14:35:14.347: INFO @log_variables: train loss nanmean: 0.057702 2021-04-14 14:35:14.347: INFO @log_variables: train positive_distance nanmean: 0.577123 2021-04-14 14:35:14.347: INFO @log_variables: train negative_distance nanmean: 1.419293 2021-04-14 14:35:14.347: INFO @log_variables: train distance_margin nanmean: 0.842170 2021-04-14 14:35:14.347: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:36:44.455: INFO @metrics_hook: train matching accuracy: 0.9899532477867303, N=110583, DB_SIZE=110592, N_IDS=11531 2021-04-14 14:36:44.614: INFO @decay_lr : LR updated to `0.00079806586` 2021-04-14 14:36:44.615: INFO @log_profile : T train: 133.176740 2021-04-14 14:36:44.615: INFO @log_profile : T read data: 0.906486 2021-04-14 14:36:44.615: INFO @log_profile : T hooks: 90.582229 2021-04-14 14:36:44.616: INFO @main_loop : Epoch 45 done 2021-04-14 14:36:44.616: INFO @main_loop : Training epoch 46 2021-04-14 14:38:58.642: INFO @log_variables: train loss nanmean: 0.056623 2021-04-14 14:38:58.642: INFO @log_variables: train positive_distance nanmean: 0.576344 2021-04-14 14:38:58.642: INFO @log_variables: train negative_distance nanmean: 1.419291 2021-04-14 14:38:58.643: INFO @log_variables: train distance_margin nanmean: 0.842948 2021-04-14 14:38:58.643: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:40:27.410: INFO @metrics_hook: train matching accuracy: 0.9898719525428633, N=110584, DB_SIZE=110592, N_IDS=11544 2021-04-14 14:40:27.561: INFO @decay_lr : LR updated to `0.00079407555` 2021-04-14 14:40:27.562: INFO @log_profile : T train: 131.206139 2021-04-14 14:40:27.562: INFO @log_profile : T read data: 2.480695 2021-04-14 14:40:27.562: INFO @log_profile : T hooks: 89.205697 2021-04-14 14:40:27.562: INFO @main_loop : Epoch 46 done 2021-04-14 14:40:27.562: INFO @main_loop : Training epoch 47 2021-04-14 14:42:44.069: INFO @log_variables: train loss nanmean: 0.055439 2021-04-14 14:42:44.069: INFO @log_variables: train positive_distance nanmean: 0.576974 2021-04-14 14:42:44.069: INFO @log_variables: train negative_distance nanmean: 1.419321 2021-04-14 14:42:44.069: INFO @log_variables: train distance_margin nanmean: 0.842347 2021-04-14 14:42:44.069: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:44:13.047: INFO @metrics_hook: train matching accuracy: 0.9899174405671552, N=110587, DB_SIZE=110592, N_IDS=11573 2021-04-14 14:44:13.215: INFO @decay_lr : LR updated to `0.00079010514` 2021-04-14 14:44:13.217: INFO @log_profile : T train: 133.651383 2021-04-14 14:44:13.217: INFO @log_profile : T read data: 2.506228 2021-04-14 14:44:13.217: INFO @log_profile : T hooks: 89.443527 2021-04-14 14:44:13.217: INFO @main_loop : Epoch 47 done 2021-04-14 14:44:13.217: INFO @main_loop : Training epoch 48 2021-04-14 14:46:25.944: INFO @log_variables: train loss nanmean: 0.054684 2021-04-14 14:46:25.944: INFO @log_variables: train positive_distance nanmean: 0.575204 2021-04-14 14:46:25.944: INFO @log_variables: train negative_distance nanmean: 1.419282 2021-04-14 14:46:25.944: INFO @log_variables: train distance_margin nanmean: 0.844078 2021-04-14 14:46:25.944: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:47:54.913: INFO @metrics_hook: train matching accuracy: 0.9896730087535267, N=110584, DB_SIZE=110592, N_IDS=11559 2021-04-14 14:47:55.072: INFO @decay_lr : LR updated to `0.0007861546` 2021-04-14 14:47:55.073: INFO @log_profile : T train: 129.959816 2021-04-14 14:47:55.073: INFO @log_profile : T read data: 2.402393 2021-04-14 14:47:55.073: INFO @log_profile : T hooks: 89.439962 2021-04-14 14:47:55.073: INFO @main_loop : Epoch 48 done 2021-04-14 14:47:55.073: INFO @main_loop : Training epoch 49 2021-04-14 14:50:09.517: INFO @log_variables: train loss nanmean: 0.055421 2021-04-14 14:50:09.517: INFO @log_variables: train positive_distance nanmean: 0.576331 2021-04-14 14:50:09.517: INFO @log_variables: train negative_distance nanmean: 1.419306 2021-04-14 14:50:09.517: INFO @log_variables: train distance_margin nanmean: 0.842975 2021-04-14 14:50:09.517: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:51:38.475: INFO @metrics_hook: train matching accuracy: 0.989528511746514, N=110586, DB_SIZE=110592, N_IDS=11583 2021-04-14 14:51:38.635: INFO @decay_lr : LR updated to `0.0007822238` 2021-04-14 14:51:38.637: INFO @log_profile : T train: 131.656573 2021-04-14 14:51:38.637: INFO @log_profile : T read data: 2.433630 2021-04-14 14:51:38.637: INFO @log_profile : T hooks: 89.419732 2021-04-14 14:51:38.637: INFO @main_loop : Epoch 49 done 2021-04-14 14:51:38.637: INFO @main_loop : Training epoch 50 2021-04-14 14:53:51.886: INFO @log_variables: train loss nanmean: 0.052429 2021-04-14 14:53:51.886: INFO @log_variables: train positive_distance nanmean: 0.574512 2021-04-14 14:53:51.887: INFO @log_variables: train negative_distance nanmean: 1.419329 2021-04-14 14:53:51.887: INFO @log_variables: train distance_margin nanmean: 0.844817 2021-04-14 14:53:51.887: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:55:22.780: INFO @metrics_hook: train matching accuracy: 0.9896823315579589, N=110587, DB_SIZE=110592, N_IDS=11552 2021-04-14 14:55:22.948: INFO @decay_lr : LR updated to `0.0007783127` 2021-04-14 14:55:22.949: WARNING @module_wrapper: From /home/kopalja/.local/lib/python3.8/site-packages/emloop_tensorflow/model.py:304: The name tf.train.write_graph is deprecated. Please use tf.io.write_graph instead. 2021-04-14 14:55:23.313: INFO @save : Model saved to: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f/model_50.ckpt 2021-04-14 14:55:23.314: INFO @log_profile : T train: 130.429463 2021-04-14 14:55:23.314: INFO @log_profile : T read data: 2.486196 2021-04-14 14:55:23.314: INFO @log_profile : T hooks: 91.709176 2021-04-14 14:55:23.314: INFO @main_loop : Epoch 50 done 2021-04-14 14:55:23.314: INFO @main_loop : Training epoch 51 2021-04-14 14:57:37.211: INFO @log_variables: train loss nanmean: 0.051596 2021-04-14 14:57:37.211: INFO @log_variables: train positive_distance nanmean: 0.574328 2021-04-14 14:57:37.211: INFO @log_variables: train negative_distance nanmean: 1.419314 2021-04-14 14:57:37.211: INFO @log_variables: train distance_margin nanmean: 0.844986 2021-04-14 14:57:37.211: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 14:59:05.043: INFO @metrics_hook: train matching accuracy: 0.990125421613752, N=110587, DB_SIZE=110592, N_IDS=11573 2021-04-14 14:59:05.211: INFO @decay_lr : LR updated to `0.0007744211` 2021-04-14 14:59:05.213: INFO @log_profile : T train: 131.198980 2021-04-14 14:59:05.213: INFO @log_profile : T read data: 2.338065 2021-04-14 14:59:05.213: INFO @log_profile : T hooks: 88.306887 2021-04-14 14:59:05.213: INFO @main_loop : Epoch 51 done 2021-04-14 14:59:05.213: INFO @main_loop : Training epoch 52 2021-04-14 15:01:18.379: INFO @log_variables: train loss nanmean: 0.052485 2021-04-14 15:01:18.379: INFO @log_variables: train positive_distance nanmean: 0.575223 2021-04-14 15:01:18.379: INFO @log_variables: train negative_distance nanmean: 1.419317 2021-04-14 15:01:18.379: INFO @log_variables: train distance_margin nanmean: 0.844094 2021-04-14 15:01:18.379: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:02:48.540: INFO @metrics_hook: train matching accuracy: 0.9892115282010472, N=110581, DB_SIZE=110592, N_IDS=11555 2021-04-14 15:02:48.721: INFO @decay_lr : LR updated to `0.000770549` 2021-04-14 15:02:48.722: INFO @log_profile : T train: 131.961400 2021-04-14 15:02:48.723: INFO @log_profile : T read data: 0.880102 2021-04-14 15:02:48.723: INFO @log_profile : T hooks: 90.614608 2021-04-14 15:02:48.723: INFO @main_loop : Epoch 52 done 2021-04-14 15:02:48.723: INFO @main_loop : Training epoch 53 2021-04-14 15:05:03.380: INFO @log_variables: train loss nanmean: 0.050798 2021-04-14 15:05:03.381: INFO @log_variables: train positive_distance nanmean: 0.574235 2021-04-14 15:05:03.381: INFO @log_variables: train negative_distance nanmean: 1.419299 2021-04-14 15:05:03.381: INFO @log_variables: train distance_margin nanmean: 0.845065 2021-04-14 15:05:03.381: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:06:32.174: INFO @metrics_hook: train matching accuracy: 0.989989691280994, N=110586, DB_SIZE=110592, N_IDS=11538 2021-04-14 15:06:32.313: INFO @decay_lr : LR updated to `0.00076669623` 2021-04-14 15:06:32.315: INFO @log_profile : T train: 132.005437 2021-04-14 15:06:32.315: INFO @log_profile : T read data: 2.311126 2021-04-14 15:06:32.315: INFO @log_profile : T hooks: 89.221917 2021-04-14 15:06:32.315: INFO @main_loop : Epoch 53 done 2021-04-14 15:06:32.315: INFO @main_loop : Training epoch 54 2021-04-14 15:08:44.954: INFO @log_variables: train loss nanmean: 0.050306 2021-04-14 15:08:44.954: INFO @log_variables: train positive_distance nanmean: 0.574809 2021-04-14 15:08:44.954: INFO @log_variables: train negative_distance nanmean: 1.419328 2021-04-14 15:08:44.954: INFO @log_variables: train distance_margin nanmean: 0.844519 2021-04-14 15:08:44.954: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:10:09.148: INFO @metrics_hook: train matching accuracy: 0.988805396558428, N=110589, DB_SIZE=110592, N_IDS=11562 2021-04-14 15:10:09.308: INFO @decay_lr : LR updated to `0.00076286274` 2021-04-14 15:10:09.310: INFO @log_profile : T train: 129.772866 2021-04-14 15:10:09.310: INFO @log_profile : T read data: 2.513966 2021-04-14 15:10:09.310: INFO @log_profile : T hooks: 84.654584 2021-04-14 15:10:09.310: INFO @main_loop : Epoch 54 done 2021-04-14 15:10:09.310: INFO @main_loop : Training epoch 55 2021-04-14 15:12:20.352: INFO @log_variables: train loss nanmean: 0.049444 2021-04-14 15:12:20.353: INFO @log_variables: train positive_distance nanmean: 0.573701 2021-04-14 15:12:20.353: INFO @log_variables: train negative_distance nanmean: 1.419332 2021-04-14 15:12:20.353: INFO @log_variables: train distance_margin nanmean: 0.845631 2021-04-14 15:12:20.353: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:13:45.530: INFO @metrics_hook: train matching accuracy: 0.9898810869466926, N=110585, DB_SIZE=110592, N_IDS=11526 2021-04-14 15:13:45.694: INFO @decay_lr : LR updated to `0.00075904845` 2021-04-14 15:13:45.695: INFO @log_profile : T train: 128.334041 2021-04-14 15:13:45.695: INFO @log_profile : T read data: 2.367803 2021-04-14 15:13:45.695: INFO @log_profile : T hooks: 85.629344 2021-04-14 15:13:45.695: INFO @main_loop : Epoch 55 done 2021-04-14 15:13:45.695: INFO @main_loop : Training epoch 56 2021-04-14 15:15:56.278: INFO @log_variables: train loss nanmean: 0.049352 2021-04-14 15:15:56.279: INFO @log_variables: train positive_distance nanmean: 0.573265 2021-04-14 15:15:56.279: INFO @log_variables: train negative_distance nanmean: 1.419325 2021-04-14 15:15:56.279: INFO @log_variables: train distance_margin nanmean: 0.846060 2021-04-14 15:15:56.279: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:17:20.982: INFO @metrics_hook: train matching accuracy: 0.989862909643348, N=110584, DB_SIZE=110592, N_IDS=11561 2021-04-14 15:17:21.147: INFO @decay_lr : LR updated to `0.0007552532` 2021-04-14 15:17:21.148: INFO @log_profile : T train: 127.934298 2021-04-14 15:17:21.149: INFO @log_profile : T read data: 2.283438 2021-04-14 15:17:21.149: INFO @log_profile : T hooks: 85.182468 2021-04-14 15:17:21.149: INFO @main_loop : Epoch 56 done 2021-04-14 15:17:21.149: INFO @main_loop : Training epoch 57 2021-04-14 15:19:33.246: INFO @log_variables: train loss nanmean: 0.047129 2021-04-14 15:19:33.246: INFO @log_variables: train positive_distance nanmean: 0.573141 2021-04-14 15:19:33.246: INFO @log_variables: train negative_distance nanmean: 1.419344 2021-04-14 15:19:33.246: INFO @log_variables: train distance_margin nanmean: 0.846203 2021-04-14 15:19:33.246: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:20:56.902: INFO @metrics_hook: train matching accuracy: 0.9902881015680104, N=110586, DB_SIZE=110592, N_IDS=11561 2021-04-14 15:20:57.064: INFO @decay_lr : LR updated to `0.0007514769` 2021-04-14 15:20:57.066: INFO @log_profile : T train: 129.390892 2021-04-14 15:20:57.066: INFO @log_profile : T read data: 2.357066 2021-04-14 15:20:57.066: INFO @log_profile : T hooks: 84.117753 2021-04-14 15:20:57.066: INFO @main_loop : Epoch 57 done 2021-04-14 15:20:57.066: INFO @main_loop : Training epoch 58 2021-04-14 15:23:09.249: INFO @log_variables: train loss nanmean: 0.048987 2021-04-14 15:23:09.249: INFO @log_variables: train positive_distance nanmean: 0.573310 2021-04-14 15:23:09.249: INFO @log_variables: train negative_distance nanmean: 1.419320 2021-04-14 15:23:09.249: INFO @log_variables: train distance_margin nanmean: 0.846010 2021-04-14 15:23:09.249: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:24:33.035: INFO @metrics_hook: train matching accuracy: 0.9903064500085904, N=110589, DB_SIZE=110592, N_IDS=11555 2021-04-14 15:24:33.181: INFO @decay_lr : LR updated to `0.00074771955` 2021-04-14 15:24:33.183: INFO @log_profile : T train: 129.518567 2021-04-14 15:24:33.183: INFO @log_profile : T read data: 2.312931 2021-04-14 15:24:33.183: INFO @log_profile : T hooks: 84.232259 2021-04-14 15:24:33.183: INFO @main_loop : Epoch 58 done 2021-04-14 15:24:33.183: INFO @main_loop : Training epoch 59 2021-04-14 15:26:44.118: INFO @log_variables: train loss nanmean: 0.048314 2021-04-14 15:26:44.118: INFO @log_variables: train positive_distance nanmean: 0.573230 2021-04-14 15:26:44.118: INFO @log_variables: train negative_distance nanmean: 1.419340 2021-04-14 15:26:44.118: INFO @log_variables: train distance_margin nanmean: 0.846110 2021-04-14 15:26:44.118: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:28:07.925: INFO @metrics_hook: train matching accuracy: 0.9897457228632401, N=110588, DB_SIZE=110592, N_IDS=11579 2021-04-14 15:28:08.074: INFO @decay_lr : LR updated to `0.0007439809` 2021-04-14 15:28:08.076: INFO @log_profile : T train: 129.752436 2021-04-14 15:28:08.076: INFO @log_profile : T read data: 0.848332 2021-04-14 15:28:08.076: INFO @log_profile : T hooks: 84.240381 2021-04-14 15:28:08.076: INFO @main_loop : Epoch 59 done 2021-04-14 15:28:08.076: INFO @main_loop : Training epoch 60 2021-04-14 15:30:17.689: INFO @log_variables: train loss nanmean: 0.045899 2021-04-14 15:30:17.689: INFO @log_variables: train positive_distance nanmean: 0.570898 2021-04-14 15:30:17.689: INFO @log_variables: train negative_distance nanmean: 1.419347 2021-04-14 15:30:17.689: INFO @log_variables: train distance_margin nanmean: 0.848449 2021-04-14 15:30:17.689: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:31:41.855: INFO @metrics_hook: train matching accuracy: 0.9902516661693027, N=110583, DB_SIZE=110592, N_IDS=11553 2021-04-14 15:31:42.003: INFO @decay_lr : LR updated to `0.00074026105` 2021-04-14 15:31:42.005: INFO @log_profile : T train: 126.928684 2021-04-14 15:31:42.005: INFO @log_profile : T read data: 2.324545 2021-04-14 15:31:42.005: INFO @log_profile : T hooks: 84.621789 2021-04-14 15:31:42.005: INFO @main_loop : Epoch 60 done 2021-04-14 15:31:42.005: INFO @main_loop : Training epoch 61 2021-04-14 15:33:52.592: INFO @log_variables: train loss nanmean: 0.045372 2021-04-14 15:33:52.593: INFO @log_variables: train positive_distance nanmean: 0.571577 2021-04-14 15:33:52.593: INFO @log_variables: train negative_distance nanmean: 1.419366 2021-04-14 15:33:52.593: INFO @log_variables: train distance_margin nanmean: 0.847789 2021-04-14 15:33:52.593: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:35:16.018: INFO @metrics_hook: train matching accuracy: 0.990224714020889, N=110585, DB_SIZE=110592, N_IDS=11572 2021-04-14 15:35:16.181: INFO @decay_lr : LR updated to `0.00073655974` 2021-04-14 15:35:16.183: INFO @log_profile : T train: 128.045677 2021-04-14 15:35:16.183: INFO @log_profile : T read data: 2.210708 2021-04-14 15:35:16.184: INFO @log_profile : T hooks: 83.870771 2021-04-14 15:35:16.184: INFO @main_loop : Epoch 61 done 2021-04-14 15:35:16.184: INFO @main_loop : Training epoch 62 2021-04-14 15:37:27.627: INFO @log_variables: train loss nanmean: 0.045906 2021-04-14 15:37:27.627: INFO @log_variables: train positive_distance nanmean: 0.571318 2021-04-14 15:37:27.627: INFO @log_variables: train negative_distance nanmean: 1.419329 2021-04-14 15:37:27.627: INFO @log_variables: train distance_margin nanmean: 0.848011 2021-04-14 15:37:27.627: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:38:51.105: INFO @metrics_hook: train matching accuracy: 0.9899625630730834, N=110586, DB_SIZE=110592, N_IDS=11570 2021-04-14 15:38:51.265: INFO @decay_lr : LR updated to `0.00073287694` 2021-04-14 15:38:51.266: INFO @log_profile : T train: 128.731226 2021-04-14 15:38:51.266: INFO @log_profile : T read data: 2.373377 2021-04-14 15:38:51.266: INFO @log_profile : T hooks: 83.923050 2021-04-14 15:38:51.266: INFO @main_loop : Epoch 62 done 2021-04-14 15:38:51.266: INFO @main_loop : Training epoch 63 2021-04-14 15:41:02.896: INFO @log_variables: train loss nanmean: 0.044487 2021-04-14 15:41:02.896: INFO @log_variables: train positive_distance nanmean: 0.571866 2021-04-14 15:41:02.897: INFO @log_variables: train negative_distance nanmean: 1.419347 2021-04-14 15:41:02.897: INFO @log_variables: train distance_margin nanmean: 0.847481 2021-04-14 15:41:02.897: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:42:27.942: INFO @metrics_hook: train matching accuracy: 0.9905504263611455, N=110587, DB_SIZE=110592, N_IDS=11569 2021-04-14 15:42:28.097: INFO @decay_lr : LR updated to `0.00072921254` 2021-04-14 15:42:28.099: INFO @log_profile : T train: 129.000385 2021-04-14 15:42:28.099: INFO @log_profile : T read data: 2.284235 2021-04-14 15:42:28.099: INFO @log_profile : T hooks: 85.496919 2021-04-14 15:42:28.099: INFO @main_loop : Epoch 63 done 2021-04-14 15:42:28.099: INFO @main_loop : Training epoch 64 2021-04-14 15:44:37.616: INFO @log_variables: train loss nanmean: 0.045002 2021-04-14 15:44:37.617: INFO @log_variables: train positive_distance nanmean: 0.572224 2021-04-14 15:44:37.617: INFO @log_variables: train negative_distance nanmean: 1.419327 2021-04-14 15:44:37.617: INFO @log_variables: train distance_margin nanmean: 0.847102 2021-04-14 15:44:37.617: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:46:02.550: INFO @metrics_hook: train matching accuracy: 0.9906946881047548, N=110582, DB_SIZE=110592, N_IDS=11551 2021-04-14 15:46:02.702: INFO @decay_lr : LR updated to `0.00072556647` 2021-04-14 15:46:02.704: INFO @log_profile : T train: 126.697489 2021-04-14 15:46:02.704: INFO @log_profile : T read data: 2.454432 2021-04-14 15:46:02.704: INFO @log_profile : T hooks: 85.400192 2021-04-14 15:46:02.705: INFO @main_loop : Epoch 64 done 2021-04-14 15:46:02.705: INFO @main_loop : Training epoch 65 2021-04-14 15:48:14.436: INFO @log_variables: train loss nanmean: 0.044629 2021-04-14 15:48:14.436: INFO @log_variables: train positive_distance nanmean: 0.571570 2021-04-14 15:48:14.436: INFO @log_variables: train negative_distance nanmean: 1.419341 2021-04-14 15:48:14.436: INFO @log_variables: train distance_margin nanmean: 0.847771 2021-04-14 15:48:14.436: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:49:38.850: INFO @metrics_hook: train matching accuracy: 0.9902070711637581, N=110590, DB_SIZE=110592, N_IDS=11579 2021-04-14 15:49:38.996: INFO @decay_lr : LR updated to `0.0007219386` 2021-04-14 15:49:38.998: INFO @log_profile : T train: 129.031110 2021-04-14 15:49:38.998: INFO @log_profile : T read data: 2.343256 2021-04-14 15:49:38.998: INFO @log_profile : T hooks: 84.866559 2021-04-14 15:49:38.998: INFO @main_loop : Epoch 65 done 2021-04-14 15:49:38.998: INFO @main_loop : Training epoch 66 2021-04-14 15:51:50.523: INFO @log_variables: train loss nanmean: 0.044419 2021-04-14 15:51:50.524: INFO @log_variables: train positive_distance nanmean: 0.571757 2021-04-14 15:51:50.524: INFO @log_variables: train negative_distance nanmean: 1.419338 2021-04-14 15:51:50.524: INFO @log_variables: train distance_margin nanmean: 0.847581 2021-04-14 15:51:50.524: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:53:15.918: INFO @metrics_hook: train matching accuracy: 0.9900170000361703, N=110588, DB_SIZE=110592, N_IDS=11571 2021-04-14 15:53:16.087: INFO @decay_lr : LR updated to `0.0007183289` 2021-04-14 15:53:16.089: INFO @log_profile : T train: 128.809192 2021-04-14 15:53:16.089: INFO @log_profile : T read data: 2.385439 2021-04-14 15:53:16.089: INFO @log_profile : T hooks: 85.844110 2021-04-14 15:53:16.089: INFO @main_loop : Epoch 66 done 2021-04-14 15:53:16.089: INFO @main_loop : Training epoch 67 2021-04-14 15:55:24.994: INFO @log_variables: train loss nanmean: 0.043706 2021-04-14 15:55:24.995: INFO @log_variables: train positive_distance nanmean: 0.570998 2021-04-14 15:55:24.995: INFO @log_variables: train negative_distance nanmean: 1.419363 2021-04-14 15:55:24.995: INFO @log_variables: train distance_margin nanmean: 0.848365 2021-04-14 15:55:24.995: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 15:56:50.257: INFO @metrics_hook: train matching accuracy: 0.9900802083427528, N=110587, DB_SIZE=110592, N_IDS=11527 2021-04-14 15:56:50.419: INFO @decay_lr : LR updated to `0.0007147373` 2021-04-14 15:56:50.420: INFO @log_profile : T train: 127.699725 2021-04-14 15:56:50.420: INFO @log_profile : T read data: 0.859336 2021-04-14 15:56:50.420: INFO @log_profile : T hooks: 85.719907 2021-04-14 15:56:50.420: INFO @main_loop : Epoch 67 done 2021-04-14 15:56:50.420: INFO @main_loop : Training epoch 68 2021-04-14 15:58:59.766: INFO @log_variables: train loss nanmean: 0.043115 2021-04-14 15:58:59.766: INFO @log_variables: train positive_distance nanmean: 0.571045 2021-04-14 15:58:59.766: INFO @log_variables: train negative_distance nanmean: 1.419370 2021-04-14 15:58:59.766: INFO @log_variables: train distance_margin nanmean: 0.848326 2021-04-14 15:58:59.766: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:00:24.193: INFO @metrics_hook: train matching accuracy: 0.990586171347959, N=110582, DB_SIZE=110592, N_IDS=11513 2021-04-14 16:00:24.360: INFO @decay_lr : LR updated to `0.0007111636` 2021-04-14 16:00:24.361: INFO @log_profile : T train: 126.789117 2021-04-14 16:00:24.361: INFO @log_profile : T read data: 2.198671 2021-04-14 16:00:24.362: INFO @log_profile : T hooks: 84.902098 2021-04-14 16:00:24.362: INFO @main_loop : Epoch 68 done 2021-04-14 16:00:24.362: INFO @main_loop : Training epoch 69 2021-04-14 16:02:36.948: INFO @log_variables: train loss nanmean: 0.042445 2021-04-14 16:02:36.949: INFO @log_variables: train positive_distance nanmean: 0.570126 2021-04-14 16:02:36.949: INFO @log_variables: train negative_distance nanmean: 1.419356 2021-04-14 16:02:36.949: INFO @log_variables: train distance_margin nanmean: 0.849230 2021-04-14 16:02:36.949: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:04:00.945: INFO @metrics_hook: train matching accuracy: 0.9898087444047565, N=110585, DB_SIZE=110592, N_IDS=11581 2021-04-14 16:04:01.102: INFO @decay_lr : LR updated to `0.00070760783` 2021-04-14 16:04:01.103: INFO @log_profile : T train: 129.967515 2021-04-14 16:04:01.104: INFO @log_profile : T read data: 2.282755 2021-04-14 16:04:01.104: INFO @log_profile : T hooks: 84.440242 2021-04-14 16:04:01.104: INFO @main_loop : Epoch 69 done 2021-04-14 16:04:01.104: INFO @main_loop : Training epoch 70 2021-04-14 16:06:12.013: INFO @log_variables: train loss nanmean: 0.042500 2021-04-14 16:06:12.013: INFO @log_variables: train positive_distance nanmean: 0.570159 2021-04-14 16:06:12.013: INFO @log_variables: train negative_distance nanmean: 1.419376 2021-04-14 16:06:12.013: INFO @log_variables: train distance_margin nanmean: 0.849217 2021-04-14 16:06:12.014: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:07:35.760: INFO @metrics_hook: train matching accuracy: 0.9901163789595522, N=110587, DB_SIZE=110592, N_IDS=11570 2021-04-14 16:07:35.916: INFO @decay_lr : LR updated to `0.0007040698` 2021-04-14 16:07:35.917: INFO @log_profile : T train: 128.421972 2021-04-14 16:07:35.917: INFO @log_profile : T read data: 2.118338 2021-04-14 16:07:35.917: INFO @log_profile : T hooks: 84.220985 2021-04-14 16:07:35.917: INFO @main_loop : Epoch 70 done 2021-04-14 16:07:35.917: INFO @main_loop : Training epoch 71 2021-04-14 16:09:48.368: INFO @log_variables: train loss nanmean: 0.042508 2021-04-14 16:09:48.369: INFO @log_variables: train positive_distance nanmean: 0.569768 2021-04-14 16:09:48.369: INFO @log_variables: train negative_distance nanmean: 1.419374 2021-04-14 16:09:48.369: INFO @log_variables: train distance_margin nanmean: 0.849607 2021-04-14 16:09:48.369: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:11:12.144: INFO @metrics_hook: train matching accuracy: 0.9898358728579826, N=110585, DB_SIZE=110592, N_IDS=11583 2021-04-14 16:11:12.310: INFO @decay_lr : LR updated to `0.00070054946` 2021-04-14 16:11:12.312: INFO @log_profile : T train: 129.662617 2021-04-14 16:11:12.312: INFO @log_profile : T read data: 2.438431 2021-04-14 16:11:12.312: INFO @log_profile : T hooks: 84.241776 2021-04-14 16:11:12.312: INFO @main_loop : Epoch 71 done 2021-04-14 16:11:12.312: INFO @main_loop : Training epoch 72 2021-04-14 16:13:24.033: INFO @log_variables: train loss nanmean: 0.041476 2021-04-14 16:13:24.033: INFO @log_variables: train positive_distance nanmean: 0.569022 2021-04-14 16:13:24.033: INFO @log_variables: train negative_distance nanmean: 1.419385 2021-04-14 16:13:24.033: INFO @log_variables: train distance_margin nanmean: 0.850362 2021-04-14 16:13:24.033: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:14:47.995: INFO @metrics_hook: train matching accuracy: 0.9899625630730834, N=110586, DB_SIZE=110592, N_IDS=11552 2021-04-14 16:14:48.136: INFO @decay_lr : LR updated to `0.0006970467` 2021-04-14 16:14:48.137: INFO @log_profile : T train: 129.027669 2021-04-14 16:14:48.138: INFO @log_profile : T read data: 2.352557 2021-04-14 16:14:48.138: INFO @log_profile : T hooks: 84.393222 2021-04-14 16:14:48.138: INFO @main_loop : Epoch 72 done 2021-04-14 16:14:48.138: INFO @main_loop : Training epoch 73 2021-04-14 16:16:58.567: INFO @log_variables: train loss nanmean: 0.041672 2021-04-14 16:16:58.568: INFO @log_variables: train positive_distance nanmean: 0.569603 2021-04-14 16:16:58.568: INFO @log_variables: train negative_distance nanmean: 1.419369 2021-04-14 16:16:58.568: INFO @log_variables: train distance_margin nanmean: 0.849766 2021-04-14 16:16:58.568: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:18:22.383: INFO @metrics_hook: train matching accuracy: 0.9893116793865518, N=110588, DB_SIZE=110592, N_IDS=11544 2021-04-14 16:18:22.539: INFO @decay_lr : LR updated to `0.00069356145` 2021-04-14 16:18:22.540: INFO @log_profile : T train: 127.630802 2021-04-14 16:18:22.541: INFO @log_profile : T read data: 2.457930 2021-04-14 16:18:22.541: INFO @log_profile : T hooks: 84.261889 2021-04-14 16:18:22.541: INFO @main_loop : Epoch 73 done 2021-04-14 16:18:22.541: INFO @main_loop : Training epoch 74 2021-04-14 16:20:32.450: INFO @log_variables: train loss nanmean: 0.040295 2021-04-14 16:20:32.450: INFO @log_variables: train positive_distance nanmean: 0.568181 2021-04-14 16:20:32.450: INFO @log_variables: train negative_distance nanmean: 1.419379 2021-04-14 16:20:32.450: INFO @log_variables: train distance_margin nanmean: 0.851198 2021-04-14 16:20:32.450: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:21:57.643: INFO @metrics_hook: train matching accuracy: 0.9910296059175664, N=110586, DB_SIZE=110592, N_IDS=11534 2021-04-14 16:21:57.831: INFO @decay_lr : LR updated to `0.0006900937` 2021-04-14 16:21:57.832: INFO @log_profile : T train: 128.729057 2021-04-14 16:21:57.832: INFO @log_profile : T read data: 0.839184 2021-04-14 16:21:57.832: INFO @log_profile : T hooks: 85.671765 2021-04-14 16:21:57.832: INFO @main_loop : Epoch 74 done 2021-04-14 16:21:57.832: INFO @main_loop : Training epoch 75 2021-04-14 16:24:09.733: INFO @log_variables: train loss nanmean: 0.040404 2021-04-14 16:24:09.733: INFO @log_variables: train positive_distance nanmean: 0.568844 2021-04-14 16:24:09.733: INFO @log_variables: train negative_distance nanmean: 1.419362 2021-04-14 16:24:09.733: INFO @log_variables: train distance_margin nanmean: 0.850518 2021-04-14 16:24:09.733: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:25:34.571: INFO @metrics_hook: train matching accuracy: 0.9902244488253061, N=110582, DB_SIZE=110592, N_IDS=11592 2021-04-14 16:25:34.734: INFO @decay_lr : LR updated to `0.0006866432` 2021-04-14 16:25:34.736: INFO @log_profile : T train: 129.087131 2021-04-14 16:25:34.736: INFO @log_profile : T read data: 2.452511 2021-04-14 16:25:34.736: INFO @log_profile : T hooks: 85.311856 2021-04-14 16:25:34.736: INFO @main_loop : Epoch 75 done 2021-04-14 16:25:34.736: INFO @main_loop : Training epoch 76 2021-04-14 16:27:45.727: INFO @log_variables: train loss nanmean: 0.041867 2021-04-14 16:27:45.727: INFO @log_variables: train positive_distance nanmean: 0.569132 2021-04-14 16:27:45.727: INFO @log_variables: train negative_distance nanmean: 1.419369 2021-04-14 16:27:45.727: INFO @log_variables: train distance_margin nanmean: 0.850238 2021-04-14 16:27:45.727: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:29:10.183: INFO @metrics_hook: train matching accuracy: 0.9901885427499209, N=110585, DB_SIZE=110592, N_IDS=11570 2021-04-14 16:29:10.355: INFO @decay_lr : LR updated to `0.00068321` 2021-04-14 16:29:10.356: INFO @log_profile : T train: 128.320110 2021-04-14 16:29:10.356: INFO @log_profile : T read data: 2.315264 2021-04-14 16:29:10.356: INFO @log_profile : T hooks: 84.933633 2021-04-14 16:29:10.356: INFO @main_loop : Epoch 76 done 2021-04-14 16:29:10.356: INFO @main_loop : Training epoch 77 2021-04-14 16:31:22.914: INFO @log_variables: train loss nanmean: 0.038085 2021-04-14 16:31:22.915: INFO @log_variables: train positive_distance nanmean: 0.567764 2021-04-14 16:31:22.915: INFO @log_variables: train negative_distance nanmean: 1.419391 2021-04-14 16:31:22.915: INFO @log_variables: train distance_margin nanmean: 0.851627 2021-04-14 16:31:22.915: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:32:47.240: INFO @metrics_hook: train matching accuracy: 0.9899264832213551, N=110587, DB_SIZE=110592, N_IDS=11554 2021-04-14 16:32:47.396: INFO @decay_lr : LR updated to `0.00067979394` 2021-04-14 16:32:47.397: INFO @log_profile : T train: 129.830490 2021-04-14 16:32:47.397: INFO @log_profile : T read data: 2.404205 2021-04-14 16:32:47.397: INFO @log_profile : T hooks: 84.753977 2021-04-14 16:32:47.398: INFO @main_loop : Epoch 77 done 2021-04-14 16:32:47.398: INFO @main_loop : Training epoch 78 2021-04-14 16:34:57.957: INFO @log_variables: train loss nanmean: 0.039085 2021-04-14 16:34:57.957: INFO @log_variables: train positive_distance nanmean: 0.569065 2021-04-14 16:34:57.957: INFO @log_variables: train negative_distance nanmean: 1.419389 2021-04-14 16:34:57.957: INFO @log_variables: train distance_margin nanmean: 0.850324 2021-04-14 16:34:57.957: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:36:22.616: INFO @metrics_hook: train matching accuracy: 0.9904779990233845, N=110586, DB_SIZE=110592, N_IDS=11535 2021-04-14 16:36:22.795: INFO @decay_lr : LR updated to `0.00067639496` 2021-04-14 16:36:22.797: INFO @log_profile : T train: 128.014437 2021-04-14 16:36:22.797: INFO @log_profile : T read data: 2.181003 2021-04-14 16:36:22.797: INFO @log_profile : T hooks: 85.151241 2021-04-14 16:36:22.797: INFO @main_loop : Epoch 78 done 2021-04-14 16:36:22.797: INFO @main_loop : Training epoch 79 2021-04-14 16:38:34.105: INFO @log_variables: train loss nanmean: 0.038552 2021-04-14 16:38:34.106: INFO @log_variables: train positive_distance nanmean: 0.568444 2021-04-14 16:38:34.106: INFO @log_variables: train negative_distance nanmean: 1.419396 2021-04-14 16:38:34.106: INFO @log_variables: train distance_margin nanmean: 0.850952 2021-04-14 16:38:34.106: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:39:57.932: INFO @metrics_hook: train matching accuracy: 0.9906222587967192, N=110581, DB_SIZE=110592, N_IDS=11541 2021-04-14 16:39:58.107: INFO @decay_lr : LR updated to `0.000673013` 2021-04-14 16:39:58.108: INFO @log_profile : T train: 128.655158 2021-04-14 16:39:58.108: INFO @log_profile : T read data: 2.319380 2021-04-14 16:39:58.108: INFO @log_profile : T hooks: 84.284497 2021-04-14 16:39:58.108: INFO @main_loop : Epoch 79 done 2021-04-14 16:39:58.108: INFO @main_loop : Training epoch 80 2021-04-14 16:42:08.892: INFO @log_variables: train loss nanmean: 0.038990 2021-04-14 16:42:08.892: INFO @log_variables: train positive_distance nanmean: 0.568785 2021-04-14 16:42:08.892: INFO @log_variables: train negative_distance nanmean: 1.419382 2021-04-14 16:42:08.892: INFO @log_variables: train distance_margin nanmean: 0.850597 2021-04-14 16:42:08.892: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:43:32.280: INFO @metrics_hook: train matching accuracy: 0.9895194690105439, N=110586, DB_SIZE=110592, N_IDS=11524 2021-04-14 16:43:32.436: INFO @decay_lr : LR updated to `0.00066964794` 2021-04-14 16:43:32.437: INFO @log_profile : T train: 128.284903 2021-04-14 16:43:32.437: INFO @log_profile : T read data: 2.171208 2021-04-14 16:43:32.437: INFO @log_profile : T hooks: 83.822232 2021-04-14 16:43:32.437: INFO @main_loop : Epoch 80 done 2021-04-14 16:43:32.437: INFO @main_loop : Training epoch 81 2021-04-14 16:45:43.161: INFO @log_variables: train loss nanmean: 0.038561 2021-04-14 16:45:43.162: INFO @log_variables: train positive_distance nanmean: 0.567576 2021-04-14 16:45:43.162: INFO @log_variables: train negative_distance nanmean: 1.419397 2021-04-14 16:45:43.162: INFO @log_variables: train distance_margin nanmean: 0.851821 2021-04-14 16:45:43.162: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:47:07.428: INFO @metrics_hook: train matching accuracy: 0.9908396256273455, N=110585, DB_SIZE=110592, N_IDS=11580 2021-04-14 16:47:07.604: INFO @decay_lr : LR updated to `0.0006662997` 2021-04-14 16:47:07.606: INFO @log_profile : T train: 129.523604 2021-04-14 16:47:07.606: INFO @log_profile : T read data: 0.841980 2021-04-14 16:47:07.606: INFO @log_profile : T hooks: 84.751425 2021-04-14 16:47:07.606: INFO @main_loop : Epoch 81 done 2021-04-14 16:47:07.606: INFO @main_loop : Training epoch 82 2021-04-14 16:49:22.774: INFO @log_variables: train loss nanmean: 0.037240 2021-04-14 16:49:22.775: INFO @log_variables: train positive_distance nanmean: 0.566207 2021-04-14 16:49:22.775: INFO @log_variables: train negative_distance nanmean: 1.419406 2021-04-14 16:49:22.775: INFO @log_variables: train distance_margin nanmean: 0.853199 2021-04-14 16:49:22.775: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:50:46.550: INFO @metrics_hook: train matching accuracy: 0.990034904960845, N=110586, DB_SIZE=110592, N_IDS=11582 2021-04-14 16:50:46.705: INFO @decay_lr : LR updated to `0.0006629682` 2021-04-14 16:50:46.706: INFO @log_profile : T train: 132.525608 2021-04-14 16:50:46.706: INFO @log_profile : T read data: 2.304557 2021-04-14 16:50:46.706: INFO @log_profile : T hooks: 84.217866 2021-04-14 16:50:46.706: INFO @main_loop : Epoch 82 done 2021-04-14 16:50:46.706: INFO @main_loop : Training epoch 83 2021-04-14 16:52:57.653: INFO @log_variables: train loss nanmean: 0.037887 2021-04-14 16:52:57.654: INFO @log_variables: train positive_distance nanmean: 0.566732 2021-04-14 16:52:57.655: INFO @log_variables: train negative_distance nanmean: 1.419388 2021-04-14 16:52:57.655: INFO @log_variables: train distance_margin nanmean: 0.852655 2021-04-14 16:52:57.655: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:54:20.280: INFO @metrics_hook: train matching accuracy: 0.9905592982773432, N=110585, DB_SIZE=110592, N_IDS=11531 2021-04-14 16:54:20.446: INFO @decay_lr : LR updated to `0.00065965333` 2021-04-14 16:54:20.447: INFO @log_profile : T train: 128.174409 2021-04-14 16:54:20.447: INFO @log_profile : T read data: 2.413379 2021-04-14 16:54:20.447: INFO @log_profile : T hooks: 83.101477 2021-04-14 16:54:20.447: INFO @main_loop : Epoch 83 done 2021-04-14 16:54:20.447: INFO @main_loop : Training epoch 84 2021-04-14 16:56:33.718: INFO @log_variables: train loss nanmean: 0.036924 2021-04-14 16:56:33.718: INFO @log_variables: train positive_distance nanmean: 0.566790 2021-04-14 16:56:33.718: INFO @log_variables: train negative_distance nanmean: 1.419421 2021-04-14 16:56:33.718: INFO @log_variables: train distance_margin nanmean: 0.852631 2021-04-14 16:56:33.718: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 16:57:58.542: INFO @metrics_hook: train matching accuracy: 0.990333227833793, N=110585, DB_SIZE=110592, N_IDS=11517 2021-04-14 16:57:58.709: INFO @decay_lr : LR updated to `0.00065635506` 2021-04-14 16:57:58.710: INFO @log_profile : T train: 130.519217 2021-04-14 16:57:58.710: INFO @log_profile : T read data: 2.389400 2021-04-14 16:57:58.710: INFO @log_profile : T hooks: 85.302739 2021-04-14 16:57:58.710: INFO @main_loop : Epoch 84 done 2021-04-14 16:57:58.710: INFO @main_loop : Training epoch 85 2021-04-14 17:00:10.993: INFO @log_variables: train loss nanmean: 0.035143 2021-04-14 17:00:10.993: INFO @log_variables: train positive_distance nanmean: 0.566523 2021-04-14 17:00:10.993: INFO @log_variables: train negative_distance nanmean: 1.419412 2021-04-14 17:00:10.993: INFO @log_variables: train distance_margin nanmean: 0.852889 2021-04-14 17:00:10.993: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:01:35.344: INFO @metrics_hook: train matching accuracy: 0.9902609733600999, N=110586, DB_SIZE=110592, N_IDS=11558 2021-04-14 17:01:35.505: INFO @decay_lr : LR updated to `0.0006530733` 2021-04-14 17:01:35.507: INFO @log_profile : T train: 129.689485 2021-04-14 17:01:35.507: INFO @log_profile : T read data: 2.238443 2021-04-14 17:01:35.507: INFO @log_profile : T hooks: 84.816662 2021-04-14 17:01:35.507: INFO @main_loop : Epoch 85 done 2021-04-14 17:01:35.507: INFO @main_loop : Training epoch 86 2021-04-14 17:03:46.445: INFO @log_variables: train loss nanmean: 0.036683 2021-04-14 17:03:46.445: INFO @log_variables: train positive_distance nanmean: 0.565972 2021-04-14 17:03:46.445: INFO @log_variables: train negative_distance nanmean: 1.419388 2021-04-14 17:03:46.445: INFO @log_variables: train distance_margin nanmean: 0.853416 2021-04-14 17:03:46.445: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:05:10.615: INFO @metrics_hook: train matching accuracy: 0.9900712554707564, N=110588, DB_SIZE=110592, N_IDS=11531 2021-04-14 17:05:10.769: INFO @decay_lr : LR updated to `0.0006498079` 2021-04-14 17:05:10.770: INFO @log_profile : T train: 128.281619 2021-04-14 17:05:10.770: INFO @log_profile : T read data: 2.309948 2021-04-14 17:05:10.770: INFO @log_profile : T hooks: 84.617587 2021-04-14 17:05:10.770: INFO @main_loop : Epoch 86 done 2021-04-14 17:05:10.770: INFO @main_loop : Training epoch 87 2021-04-14 17:07:23.557: INFO @log_variables: train loss nanmean: 0.036674 2021-04-14 17:07:23.557: INFO @log_variables: train positive_distance nanmean: 0.566887 2021-04-14 17:07:23.557: INFO @log_variables: train negative_distance nanmean: 1.419387 2021-04-14 17:07:23.557: INFO @log_variables: train distance_margin nanmean: 0.852500 2021-04-14 17:07:23.557: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:08:47.406: INFO @metrics_hook: train matching accuracy: 0.9899171670404399, N=110584, DB_SIZE=110592, N_IDS=11587 2021-04-14 17:08:47.576: INFO @decay_lr : LR updated to `0.0006465589` 2021-04-14 17:08:47.577: INFO @log_profile : T train: 130.113338 2021-04-14 17:08:47.577: INFO @log_profile : T read data: 2.328154 2021-04-14 17:08:47.577: INFO @log_profile : T hooks: 84.313831 2021-04-14 17:08:47.577: INFO @main_loop : Epoch 87 done 2021-04-14 17:08:47.577: INFO @main_loop : Training epoch 88 2021-04-14 17:10:58.692: INFO @log_variables: train loss nanmean: 0.036046 2021-04-14 17:10:58.692: INFO @log_variables: train positive_distance nanmean: 0.566229 2021-04-14 17:10:58.692: INFO @log_variables: train negative_distance nanmean: 1.419412 2021-04-14 17:10:58.692: INFO @log_variables: train distance_margin nanmean: 0.853183 2021-04-14 17:10:58.692: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:12:23.325: INFO @metrics_hook: train matching accuracy: 0.9909571004846994, N=110584, DB_SIZE=110592, N_IDS=11555 2021-04-14 17:12:23.516: INFO @decay_lr : LR updated to `0.0006433261` 2021-04-14 17:12:23.517: INFO @log_profile : T train: 128.317570 2021-04-14 17:12:23.517: INFO @log_profile : T read data: 2.439289 2021-04-14 17:12:23.517: INFO @log_profile : T hooks: 85.132175 2021-04-14 17:12:23.518: INFO @main_loop : Epoch 88 done 2021-04-14 17:12:23.518: INFO @main_loop : Training epoch 89 2021-04-14 17:14:34.381: INFO @log_variables: train loss nanmean: 0.035048 2021-04-14 17:14:34.382: INFO @log_variables: train positive_distance nanmean: 0.566094 2021-04-14 17:14:34.382: INFO @log_variables: train negative_distance nanmean: 1.419405 2021-04-14 17:14:34.382: INFO @log_variables: train distance_margin nanmean: 0.853311 2021-04-14 17:14:34.382: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:15:58.488: INFO @metrics_hook: train matching accuracy: 0.9908037038376677, N=110588, DB_SIZE=110592, N_IDS=11553 2021-04-14 17:15:58.651: INFO @decay_lr : LR updated to `0.0006401095` 2021-04-14 17:15:58.652: INFO @log_profile : T train: 129.655152 2021-04-14 17:15:58.652: INFO @log_profile : T read data: 0.853138 2021-04-14 17:15:58.652: INFO @log_profile : T hooks: 84.574936 2021-04-14 17:15:58.652: INFO @main_loop : Epoch 89 done 2021-04-14 17:15:58.652: INFO @main_loop : Training epoch 90 2021-04-14 17:18:11.795: INFO @log_variables: train loss nanmean: 0.036342 2021-04-14 17:18:11.795: INFO @log_variables: train positive_distance nanmean: 0.567210 2021-04-14 17:18:11.795: INFO @log_variables: train negative_distance nanmean: 1.419405 2021-04-14 17:18:11.795: INFO @log_variables: train distance_margin nanmean: 0.852195 2021-04-14 17:18:11.795: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:19:38.452: INFO @metrics_hook: train matching accuracy: 0.9898541419877562, N=110587, DB_SIZE=110592, N_IDS=11534 2021-04-14 17:19:38.616: INFO @decay_lr : LR updated to `0.0006369089` 2021-04-14 17:19:38.618: INFO @log_profile : T train: 130.598552 2021-04-14 17:19:38.618: INFO @log_profile : T read data: 2.207526 2021-04-14 17:19:38.618: INFO @log_profile : T hooks: 87.107906 2021-04-14 17:19:38.618: INFO @main_loop : Epoch 90 done 2021-04-14 17:19:38.618: INFO @main_loop : Training epoch 91 2021-04-14 17:21:50.268: INFO @log_variables: train loss nanmean: 0.033689 2021-04-14 17:21:50.268: INFO @log_variables: train positive_distance nanmean: 0.564384 2021-04-14 17:21:50.268: INFO @log_variables: train negative_distance nanmean: 1.419405 2021-04-14 17:21:50.268: INFO @log_variables: train distance_margin nanmean: 0.855021 2021-04-14 17:21:50.268: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:23:15.236: INFO @metrics_hook: train matching accuracy: 0.9900076863950807, N=110585, DB_SIZE=110592, N_IDS=11528 2021-04-14 17:23:15.421: INFO @decay_lr : LR updated to `0.0006337244` 2021-04-14 17:23:15.422: INFO @log_profile : T train: 128.990185 2021-04-14 17:23:15.422: INFO @log_profile : T read data: 2.305164 2021-04-14 17:23:15.422: INFO @log_profile : T hooks: 85.456130 2021-04-14 17:23:15.422: INFO @main_loop : Epoch 91 done 2021-04-14 17:23:15.422: INFO @main_loop : Training epoch 92 2021-04-14 17:25:26.119: INFO @log_variables: train loss nanmean: 0.034922 2021-04-14 17:25:26.119: INFO @log_variables: train positive_distance nanmean: 0.565878 2021-04-14 17:25:26.119: INFO @log_variables: train negative_distance nanmean: 1.419415 2021-04-14 17:25:26.119: INFO @log_variables: train distance_margin nanmean: 0.853536 2021-04-14 17:25:26.119: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:26:50.520: INFO @metrics_hook: train matching accuracy: 0.9908304998914852, N=110584, DB_SIZE=110592, N_IDS=11541 2021-04-14 17:26:50.694: INFO @decay_lr : LR updated to `0.00063055573` 2021-04-14 17:26:50.695: INFO @log_profile : T train: 127.936718 2021-04-14 17:26:50.695: INFO @log_profile : T read data: 2.396207 2021-04-14 17:26:50.695: INFO @log_profile : T hooks: 84.886983 2021-04-14 17:26:50.695: INFO @main_loop : Epoch 92 done 2021-04-14 17:26:50.695: INFO @main_loop : Training epoch 93 2021-04-14 17:29:03.460: INFO @log_variables: train loss nanmean: 0.033341 2021-04-14 17:29:03.460: INFO @log_variables: train positive_distance nanmean: 0.564521 2021-04-14 17:29:03.460: INFO @log_variables: train negative_distance nanmean: 1.419417 2021-04-14 17:29:03.460: INFO @log_variables: train distance_margin nanmean: 0.854895 2021-04-14 17:29:03.460: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:30:26.532: INFO @metrics_hook: train matching accuracy: 0.9901882765730409, N=110582, DB_SIZE=110592, N_IDS=11547 2021-04-14 17:30:26.695: INFO @decay_lr : LR updated to `0.000627403` 2021-04-14 17:30:26.697: INFO @log_profile : T train: 130.138305 2021-04-14 17:30:26.697: INFO @log_profile : T read data: 2.292857 2021-04-14 17:30:26.697: INFO @log_profile : T hooks: 83.519018 2021-04-14 17:30:26.698: INFO @main_loop : Epoch 93 done 2021-04-14 17:30:26.698: INFO @main_loop : Training epoch 94 2021-04-14 17:32:37.893: INFO @log_variables: train loss nanmean: 0.034952 2021-04-14 17:32:37.893: INFO @log_variables: train positive_distance nanmean: 0.565742 2021-04-14 17:32:37.893: INFO @log_variables: train negative_distance nanmean: 1.419408 2021-04-14 17:32:37.893: INFO @log_variables: train distance_margin nanmean: 0.853666 2021-04-14 17:32:37.893: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:34:02.282: INFO @metrics_hook: train matching accuracy: 0.9900256818346235, N=110584, DB_SIZE=110592, N_IDS=11579 2021-04-14 17:34:02.440: INFO @decay_lr : LR updated to `0.00062426593` 2021-04-14 17:34:02.442: INFO @log_profile : T train: 128.509429 2021-04-14 17:34:02.442: INFO @log_profile : T read data: 2.345464 2021-04-14 17:34:02.442: INFO @log_profile : T hooks: 84.837974 2021-04-14 17:34:02.443: INFO @main_loop : Epoch 94 done 2021-04-14 17:34:02.443: INFO @main_loop : Training epoch 95 2021-04-14 17:36:14.210: INFO @log_variables: train loss nanmean: 0.034173 2021-04-14 17:36:14.211: INFO @log_variables: train positive_distance nanmean: 0.564117 2021-04-14 17:36:14.211: INFO @log_variables: train negative_distance nanmean: 1.419427 2021-04-14 17:36:14.211: INFO @log_variables: train distance_margin nanmean: 0.855310 2021-04-14 17:36:14.211: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:37:38.548: INFO @metrics_hook: train matching accuracy: 0.9909124612310224, N=110591, DB_SIZE=110592, N_IDS=11558 2021-04-14 17:37:38.717: INFO @decay_lr : LR updated to `0.0006211446` 2021-04-14 17:37:38.719: INFO @log_profile : T train: 129.147744 2021-04-14 17:37:38.719: INFO @log_profile : T read data: 2.260270 2021-04-14 17:37:38.719: INFO @log_profile : T hooks: 84.816999 2021-04-14 17:37:38.719: INFO @main_loop : Epoch 95 done 2021-04-14 17:37:38.719: INFO @main_loop : Training epoch 96 2021-04-14 17:39:49.816: INFO @log_variables: train loss nanmean: 0.033443 2021-04-14 17:39:49.817: INFO @log_variables: train positive_distance nanmean: 0.565495 2021-04-14 17:39:49.817: INFO @log_variables: train negative_distance nanmean: 1.419423 2021-04-14 17:39:49.817: INFO @log_variables: train distance_margin nanmean: 0.853929 2021-04-14 17:39:49.817: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:41:13.187: INFO @metrics_hook: train matching accuracy: 0.9901434177924873, N=110586, DB_SIZE=110592, N_IDS=11529 2021-04-14 17:41:13.354: INFO @decay_lr : LR updated to `0.0006180389` 2021-04-14 17:41:13.355: INFO @log_profile : T train: 129.900544 2021-04-14 17:41:13.356: INFO @log_profile : T read data: 0.834873 2021-04-14 17:41:13.356: INFO @log_profile : T hooks: 83.849207 2021-04-14 17:41:13.356: INFO @main_loop : Epoch 96 done 2021-04-14 17:41:13.356: INFO @main_loop : Training epoch 97 2021-04-14 17:43:25.504: INFO @log_variables: train loss nanmean: 0.032737 2021-04-14 17:43:25.504: INFO @log_variables: train positive_distance nanmean: 0.563061 2021-04-14 17:43:25.504: INFO @log_variables: train negative_distance nanmean: 1.419417 2021-04-14 17:43:25.504: INFO @log_variables: train distance_margin nanmean: 0.856357 2021-04-14 17:43:25.504: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:44:49.742: INFO @metrics_hook: train matching accuracy: 0.991210619597063, N=110588, DB_SIZE=110592, N_IDS=11559 2021-04-14 17:44:49.897: INFO @decay_lr : LR updated to `0.0006149487` 2021-04-14 17:44:49.899: INFO @log_profile : T train: 129.448396 2021-04-14 17:44:49.899: INFO @log_profile : T read data: 2.362783 2021-04-14 17:44:49.899: INFO @log_profile : T hooks: 84.680729 2021-04-14 17:44:49.899: INFO @main_loop : Epoch 97 done 2021-04-14 17:44:49.899: INFO @main_loop : Training epoch 98 2021-04-14 17:47:01.168: INFO @log_variables: train loss nanmean: 0.032950 2021-04-14 17:47:01.169: INFO @log_variables: train positive_distance nanmean: 0.565113 2021-04-14 17:47:01.169: INFO @log_variables: train negative_distance nanmean: 1.419423 2021-04-14 17:47:01.169: INFO @log_variables: train distance_margin nanmean: 0.854310 2021-04-14 17:47:01.169: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:48:25.073: INFO @metrics_hook: train matching accuracy: 0.9906226827989076, N=110586, DB_SIZE=110592, N_IDS=11541 2021-04-14 17:48:25.240: INFO @decay_lr : LR updated to `0.0006118739` 2021-04-14 17:48:25.242: INFO @log_profile : T train: 128.637825 2021-04-14 17:48:25.242: INFO @log_profile : T read data: 2.282173 2021-04-14 17:48:25.242: INFO @log_profile : T hooks: 84.371058 2021-04-14 17:48:25.242: INFO @main_loop : Epoch 98 done 2021-04-14 17:48:25.242: INFO @main_loop : Training epoch 99 2021-04-14 17:50:36.965: INFO @log_variables: train loss nanmean: 0.033173 2021-04-14 17:50:36.965: INFO @log_variables: train positive_distance nanmean: 0.564807 2021-04-14 17:50:36.965: INFO @log_variables: train negative_distance nanmean: 1.419424 2021-04-14 17:50:36.965: INFO @log_variables: train distance_margin nanmean: 0.854617 2021-04-14 17:50:36.965: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:52:00.889: INFO @metrics_hook: train matching accuracy: 0.9902428878881594, N=110586, DB_SIZE=110592, N_IDS=11541 2021-04-14 17:52:01.066: INFO @decay_lr : LR updated to `0.00060881453` 2021-04-14 17:52:01.067: INFO @log_profile : T train: 129.115776 2021-04-14 17:52:01.067: INFO @log_profile : T read data: 2.238252 2021-04-14 17:52:01.067: INFO @log_profile : T hooks: 84.419044 2021-04-14 17:52:01.067: INFO @main_loop : Epoch 99 done 2021-04-14 17:52:01.067: INFO @main_loop : Training epoch 100 2021-04-14 17:54:14.046: INFO @log_variables: train loss nanmean: 0.032454 2021-04-14 17:54:14.047: INFO @log_variables: train positive_distance nanmean: 0.563873 2021-04-14 17:54:14.047: INFO @log_variables: train negative_distance nanmean: 1.419449 2021-04-14 17:54:14.047: INFO @log_variables: train distance_margin nanmean: 0.855576 2021-04-14 17:54:14.047: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:55:37.790: INFO @metrics_hook: train matching accuracy: 0.9896100842767751, N=110588, DB_SIZE=110592, N_IDS=11537 2021-04-14 17:55:37.952: INFO @decay_lr : LR updated to `0.00060577044` 2021-04-14 17:55:38.232: INFO @save : Model saved to: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f/model_100.ckpt 2021-04-14 17:55:38.254: INFO @log_profile : T train: 130.399339 2021-04-14 17:55:38.254: INFO @log_profile : T read data: 2.244171 2021-04-14 17:55:38.254: INFO @log_profile : T hooks: 84.490249 2021-04-14 17:55:38.254: INFO @main_loop : Epoch 100 done 2021-04-14 17:55:38.254: INFO @main_loop : Training epoch 101 2021-04-14 17:57:48.304: INFO @log_variables: train loss nanmean: 0.032565 2021-04-14 17:57:48.304: INFO @log_variables: train positive_distance nanmean: 0.562697 2021-04-14 17:57:48.304: INFO @log_variables: train negative_distance nanmean: 1.419449 2021-04-14 17:57:48.304: INFO @log_variables: train distance_margin nanmean: 0.856752 2021-04-14 17:57:48.304: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 17:59:12.143: INFO @metrics_hook: train matching accuracy: 0.9899170758615701, N=110583, DB_SIZE=110592, N_IDS=11524 2021-04-14 17:59:12.311: INFO @decay_lr : LR updated to `0.0006027416` 2021-04-14 17:59:12.312: INFO @log_profile : T train: 127.365232 2021-04-14 17:59:12.312: INFO @log_profile : T read data: 2.358566 2021-04-14 17:59:12.312: INFO @log_profile : T hooks: 84.282377 2021-04-14 17:59:12.312: INFO @main_loop : Epoch 101 done 2021-04-14 17:59:12.312: INFO @main_loop : Training epoch 102 2021-04-14 18:01:26.750: INFO @log_variables: train loss nanmean: 0.032281 2021-04-14 18:01:26.750: INFO @log_variables: train positive_distance nanmean: 0.563501 2021-04-14 18:01:26.750: INFO @log_variables: train negative_distance nanmean: 1.419378 2021-04-14 18:01:26.750: INFO @log_variables: train distance_margin nanmean: 0.855877 2021-04-14 18:01:26.750: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:02:49.640: INFO @metrics_hook: train matching accuracy: 0.9900351752886816, N=110589, DB_SIZE=110592, N_IDS=11587 2021-04-14 18:02:49.806: INFO @decay_lr : LR updated to `0.0005997279` 2021-04-14 18:02:49.807: INFO @log_profile : T train: 131.736080 2021-04-14 18:02:49.807: INFO @log_profile : T read data: 2.352272 2021-04-14 18:02:49.808: INFO @log_profile : T hooks: 83.355546 2021-04-14 18:02:49.808: INFO @main_loop : Epoch 102 done 2021-04-14 18:02:49.808: INFO @main_loop : Training epoch 103 2021-04-14 18:05:00.015: INFO @log_variables: train loss nanmean: 0.033359 2021-04-14 18:05:00.015: INFO @log_variables: train positive_distance nanmean: 0.564974 2021-04-14 18:05:00.015: INFO @log_variables: train negative_distance nanmean: 1.419407 2021-04-14 18:05:00.015: INFO @log_variables: train distance_margin nanmean: 0.854433 2021-04-14 18:05:00.015: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:06:23.542: INFO @metrics_hook: train matching accuracy: 0.9900616736901123, N=110582, DB_SIZE=110592, N_IDS=11584 2021-04-14 18:06:23.705: INFO @decay_lr : LR updated to `0.0005967293` 2021-04-14 18:06:23.706: INFO @log_profile : T train: 129.017281 2021-04-14 18:06:23.706: INFO @log_profile : T read data: 0.831535 2021-04-14 18:06:23.706: INFO @log_profile : T hooks: 83.997595 2021-04-14 18:06:23.706: INFO @main_loop : Epoch 103 done 2021-04-14 18:06:23.706: INFO @main_loop : Training epoch 104 2021-04-14 18:08:38.101: INFO @log_variables: train loss nanmean: 0.031674 2021-04-14 18:08:38.101: INFO @log_variables: train positive_distance nanmean: 0.563627 2021-04-14 18:08:38.101: INFO @log_variables: train negative_distance nanmean: 1.419450 2021-04-14 18:08:38.101: INFO @log_variables: train distance_margin nanmean: 0.855823 2021-04-14 18:08:38.101: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:10:00.559: INFO @metrics_hook: train matching accuracy: 0.9901161108297765, N=110584, DB_SIZE=110592, N_IDS=11552 2021-04-14 18:10:00.723: INFO @decay_lr : LR updated to `0.0005937456` 2021-04-14 18:10:00.725: INFO @log_profile : T train: 131.846117 2021-04-14 18:10:00.725: INFO @log_profile : T read data: 2.216595 2021-04-14 18:10:00.725: INFO @log_profile : T hooks: 82.904499 2021-04-14 18:10:00.725: INFO @main_loop : Epoch 104 done 2021-04-14 18:10:00.725: INFO @main_loop : Training epoch 105 2021-04-14 18:12:13.454: INFO @log_variables: train loss nanmean: 0.031492 2021-04-14 18:12:13.454: INFO @log_variables: train positive_distance nanmean: 0.564191 2021-04-14 18:12:13.454: INFO @log_variables: train negative_distance nanmean: 1.419444 2021-04-14 18:12:13.454: INFO @log_variables: train distance_margin nanmean: 0.855253 2021-04-14 18:12:13.454: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:13:37.532: INFO @metrics_hook: train matching accuracy: 0.9909480575851841, N=110584, DB_SIZE=110592, N_IDS=11554 2021-04-14 18:13:37.694: INFO @decay_lr : LR updated to `0.0005907769` 2021-04-14 18:13:37.695: INFO @log_profile : T train: 129.959545 2021-04-14 18:13:37.695: INFO @log_profile : T read data: 2.436227 2021-04-14 18:13:37.695: INFO @log_profile : T hooks: 84.522773 2021-04-14 18:13:37.695: INFO @main_loop : Epoch 105 done 2021-04-14 18:13:37.695: INFO @main_loop : Training epoch 106 2021-04-14 18:15:49.255: INFO @log_variables: train loss nanmean: 0.031193 2021-04-14 18:15:49.255: INFO @log_variables: train positive_distance nanmean: 0.562952 2021-04-14 18:15:49.256: INFO @log_variables: train negative_distance nanmean: 1.419442 2021-04-14 18:15:49.256: INFO @log_variables: train distance_margin nanmean: 0.856491 2021-04-14 18:15:49.256: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:17:12.872: INFO @metrics_hook: train matching accuracy: 0.9903785289277124, N=110586, DB_SIZE=110592, N_IDS=11567 2021-04-14 18:17:13.050: INFO @decay_lr : LR updated to `0.00058782304` 2021-04-14 18:17:13.051: INFO @log_profile : T train: 128.892464 2021-04-14 18:17:13.051: INFO @log_profile : T read data: 2.311141 2021-04-14 18:17:13.051: INFO @log_profile : T hooks: 84.099577 2021-04-14 18:17:13.051: INFO @main_loop : Epoch 106 done 2021-04-14 18:17:13.051: INFO @main_loop : Training epoch 107 2021-04-14 18:19:24.770: INFO @log_variables: train loss nanmean: 0.031552 2021-04-14 18:19:24.770: INFO @log_variables: train positive_distance nanmean: 0.562326 2021-04-14 18:19:24.770: INFO @log_variables: train negative_distance nanmean: 1.419437 2021-04-14 18:19:24.770: INFO @log_variables: train distance_margin nanmean: 0.857111 2021-04-14 18:19:24.770: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:20:49.660: INFO @metrics_hook: train matching accuracy: 0.9907674500619422, N=110587, DB_SIZE=110592, N_IDS=11544 2021-04-14 18:20:49.846: INFO @decay_lr : LR updated to `0.0005848839` 2021-04-14 18:20:49.847: INFO @log_profile : T train: 129.128813 2021-04-14 18:20:49.848: INFO @log_profile : T read data: 2.222455 2021-04-14 18:20:49.848: INFO @log_profile : T hooks: 85.393142 2021-04-14 18:20:49.848: INFO @main_loop : Epoch 107 done 2021-04-14 18:20:49.848: INFO @main_loop : Training epoch 108 2021-04-14 18:23:02.476: INFO @log_variables: train loss nanmean: 0.030408 2021-04-14 18:23:02.476: INFO @log_variables: train positive_distance nanmean: 0.562808 2021-04-14 18:23:02.476: INFO @log_variables: train negative_distance nanmean: 1.419488 2021-04-14 18:23:02.476: INFO @log_variables: train distance_margin nanmean: 0.856680 2021-04-14 18:23:02.476: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:24:26.811: INFO @metrics_hook: train matching accuracy: 0.9905685116695453, N=110587, DB_SIZE=110592, N_IDS=11569 2021-04-14 18:24:26.983: INFO @decay_lr : LR updated to `0.0005819595` 2021-04-14 18:24:26.984: INFO @log_profile : T train: 129.908455 2021-04-14 18:24:26.984: INFO @log_profile : T read data: 2.363935 2021-04-14 18:24:26.984: INFO @log_profile : T hooks: 84.812266 2021-04-14 18:24:26.984: INFO @main_loop : Epoch 108 done 2021-04-14 18:24:26.984: INFO @main_loop : Training epoch 109 2021-04-14 18:26:40.955: INFO @log_variables: train loss nanmean: 0.029597 2021-04-14 18:26:40.957: INFO @log_variables: train positive_distance nanmean: 0.561436 2021-04-14 18:26:40.957: INFO @log_variables: train negative_distance nanmean: 1.419448 2021-04-14 18:26:40.957: INFO @log_variables: train distance_margin nanmean: 0.858012 2021-04-14 18:26:40.957: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:28:04.761: INFO @metrics_hook: train matching accuracy: 0.9907763259031515, N=110585, DB_SIZE=110592, N_IDS=11559 2021-04-14 18:28:04.929: INFO @decay_lr : LR updated to `0.0005790497` 2021-04-14 18:28:04.930: INFO @log_profile : T train: 131.203931 2021-04-14 18:28:04.931: INFO @log_profile : T read data: 2.420209 2021-04-14 18:28:04.931: INFO @log_profile : T hooks: 84.270029 2021-04-14 18:28:04.931: INFO @main_loop : Epoch 109 done 2021-04-14 18:28:04.931: INFO @main_loop : Training epoch 110 2021-04-14 18:30:15.787: INFO @log_variables: train loss nanmean: 0.030740 2021-04-14 18:30:15.787: INFO @log_variables: train positive_distance nanmean: 0.562111 2021-04-14 18:30:15.787: INFO @log_variables: train negative_distance nanmean: 1.419440 2021-04-14 18:30:15.787: INFO @log_variables: train distance_margin nanmean: 0.857329 2021-04-14 18:30:15.788: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:31:38.465: INFO @metrics_hook: train matching accuracy: 0.9907671995948781, N=110584, DB_SIZE=110592, N_IDS=11552 2021-04-14 18:31:38.630: INFO @decay_lr : LR updated to `0.00057615445` 2021-04-14 18:31:38.631: INFO @log_profile : T train: 128.228842 2021-04-14 18:31:38.631: INFO @log_profile : T read data: 2.281327 2021-04-14 18:31:38.631: INFO @log_profile : T hooks: 83.138215 2021-04-14 18:31:38.631: INFO @main_loop : Epoch 110 done 2021-04-14 18:31:38.631: INFO @main_loop : Training epoch 111 2021-04-14 18:33:51.689: INFO @log_variables: train loss nanmean: 0.030478 2021-04-14 18:33:51.689: INFO @log_variables: train positive_distance nanmean: 0.562694 2021-04-14 18:33:51.689: INFO @log_variables: train negative_distance nanmean: 1.419461 2021-04-14 18:33:51.689: INFO @log_variables: train distance_margin nanmean: 0.856767 2021-04-14 18:33:51.689: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:35:15.521: INFO @metrics_hook: train matching accuracy: 0.9905958947463605, N=110590, DB_SIZE=110592, N_IDS=11570 2021-04-14 18:35:15.689: INFO @decay_lr : LR updated to `0.0005732737` 2021-04-14 18:35:15.690: INFO @log_profile : T train: 131.859132 2021-04-14 18:35:15.690: INFO @log_profile : T read data: 0.852556 2021-04-14 18:35:15.690: INFO @log_profile : T hooks: 84.295164 2021-04-14 18:35:15.690: INFO @main_loop : Epoch 111 done 2021-04-14 18:35:15.690: INFO @main_loop : Training epoch 112 2021-04-14 18:37:28.785: INFO @log_variables: train loss nanmean: 0.030432 2021-04-14 18:37:28.785: INFO @log_variables: train positive_distance nanmean: 0.561256 2021-04-14 18:37:28.785: INFO @log_variables: train negative_distance nanmean: 1.419465 2021-04-14 18:37:28.785: INFO @log_variables: train distance_margin nanmean: 0.858210 2021-04-14 18:37:28.785: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:38:52.481: INFO @metrics_hook: train matching accuracy: 0.9902792346366694, N=110588, DB_SIZE=110592, N_IDS=11586 2021-04-14 18:38:52.651: INFO @decay_lr : LR updated to `0.0005704073` 2021-04-14 18:38:52.653: INFO @log_profile : T train: 130.484666 2021-04-14 18:38:52.653: INFO @log_profile : T read data: 2.277889 2021-04-14 18:38:52.653: INFO @log_profile : T hooks: 84.148444 2021-04-14 18:38:52.653: INFO @main_loop : Epoch 112 done 2021-04-14 18:38:52.653: INFO @main_loop : Training epoch 113 2021-04-14 18:41:05.539: INFO @log_variables: train loss nanmean: 0.030134 2021-04-14 18:41:05.539: INFO @log_variables: train positive_distance nanmean: 0.562082 2021-04-14 18:41:05.539: INFO @log_variables: train negative_distance nanmean: 1.419439 2021-04-14 18:41:05.539: INFO @log_variables: train distance_margin nanmean: 0.857357 2021-04-14 18:41:05.540: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:42:29.599: INFO @metrics_hook: train matching accuracy: 0.990224714020889, N=110585, DB_SIZE=110592, N_IDS=11532 2021-04-14 18:42:29.758: INFO @decay_lr : LR updated to `0.00056755525` 2021-04-14 18:42:29.759: INFO @log_profile : T train: 130.255293 2021-04-14 18:42:29.759: INFO @log_profile : T read data: 2.284729 2021-04-14 18:42:29.759: INFO @log_profile : T hooks: 84.514726 2021-04-14 18:42:29.759: INFO @main_loop : Epoch 113 done 2021-04-14 18:42:29.760: INFO @main_loop : Training epoch 114 2021-04-14 18:44:43.568: INFO @log_variables: train loss nanmean: 0.030787 2021-04-14 18:44:43.568: INFO @log_variables: train positive_distance nanmean: 0.563675 2021-04-14 18:44:43.568: INFO @log_variables: train negative_distance nanmean: 1.419456 2021-04-14 18:44:43.568: INFO @log_variables: train distance_margin nanmean: 0.855781 2021-04-14 18:44:43.568: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:46:07.058: INFO @metrics_hook: train matching accuracy: 0.9899174405671552, N=110587, DB_SIZE=110592, N_IDS=11565 2021-04-14 18:46:07.235: INFO @decay_lr : LR updated to `0.00056471745` 2021-04-14 18:46:07.237: INFO @log_profile : T train: 131.122887 2021-04-14 18:46:07.237: INFO @log_profile : T read data: 2.341697 2021-04-14 18:46:07.237: INFO @log_profile : T hooks: 83.961452 2021-04-14 18:46:07.237: INFO @main_loop : Epoch 114 done 2021-04-14 18:46:07.237: INFO @main_loop : Training epoch 115 2021-04-14 18:48:18.979: INFO @log_variables: train loss nanmean: 0.029776 2021-04-14 18:48:18.979: INFO @log_variables: train positive_distance nanmean: 0.563011 2021-04-14 18:48:18.979: INFO @log_variables: train negative_distance nanmean: 1.419419 2021-04-14 18:48:18.979: INFO @log_variables: train distance_margin nanmean: 0.856408 2021-04-14 18:48:18.979: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:49:42.846: INFO @metrics_hook: train matching accuracy: 0.9902339334641505, N=110587, DB_SIZE=110592, N_IDS=11567 2021-04-14 18:49:43.024: INFO @decay_lr : LR updated to `0.00056189386` 2021-04-14 18:49:43.025: INFO @log_profile : T train: 129.179590 2021-04-14 18:49:43.025: INFO @log_profile : T read data: 2.215478 2021-04-14 18:49:43.025: INFO @log_profile : T hooks: 84.339809 2021-04-14 18:49:43.025: INFO @main_loop : Epoch 115 done 2021-04-14 18:49:43.025: INFO @main_loop : Training epoch 116 2021-04-14 18:51:56.126: INFO @log_variables: train loss nanmean: 0.028933 2021-04-14 18:51:56.126: INFO @log_variables: train positive_distance nanmean: 0.562340 2021-04-14 18:51:56.126: INFO @log_variables: train negative_distance nanmean: 1.419433 2021-04-14 18:51:56.126: INFO @log_variables: train distance_margin nanmean: 0.857093 2021-04-14 18:51:56.126: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:53:20.039: INFO @metrics_hook: train matching accuracy: 0.9895827681623351, N=110586, DB_SIZE=110592, N_IDS=11564 2021-04-14 18:53:20.202: INFO @decay_lr : LR updated to `0.0005590844` 2021-04-14 18:53:20.203: INFO @log_profile : T train: 130.453339 2021-04-14 18:53:20.203: INFO @log_profile : T read data: 2.311252 2021-04-14 18:53:20.203: INFO @log_profile : T hooks: 84.360701 2021-04-14 18:53:20.203: INFO @main_loop : Epoch 116 done 2021-04-14 18:53:20.204: INFO @main_loop : Training epoch 117 2021-04-14 18:55:32.900: INFO @log_variables: train loss nanmean: 0.028656 2021-04-14 18:55:32.900: INFO @log_variables: train positive_distance nanmean: 0.561604 2021-04-14 18:55:32.900: INFO @log_variables: train negative_distance nanmean: 1.419448 2021-04-14 18:55:32.900: INFO @log_variables: train distance_margin nanmean: 0.857843 2021-04-14 18:55:32.900: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 18:56:56.192: INFO @metrics_hook: train matching accuracy: 0.9912193445706431, N=110584, DB_SIZE=110592, N_IDS=11525 2021-04-14 18:56:56.358: INFO @decay_lr : LR updated to `0.000556289` 2021-04-14 18:56:56.359: INFO @log_profile : T train: 130.081015 2021-04-14 18:56:56.359: INFO @log_profile : T read data: 2.281999 2021-04-14 18:56:56.359: INFO @log_profile : T hooks: 83.739662 2021-04-14 18:56:56.360: INFO @main_loop : Epoch 117 done 2021-04-14 18:56:56.360: INFO @main_loop : Training epoch 118 2021-04-14 18:59:06.397: INFO @log_variables: train loss nanmean: 0.029610 2021-04-14 18:59:06.397: INFO @log_variables: train positive_distance nanmean: 0.561419 2021-04-14 18:59:06.397: INFO @log_variables: train negative_distance nanmean: 1.419445 2021-04-14 18:59:06.397: INFO @log_variables: train distance_margin nanmean: 0.858026 2021-04-14 18:59:06.397: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:00:30.236: INFO @metrics_hook: train matching accuracy: 0.990278970927341, N=110585, DB_SIZE=110592, N_IDS=11547 2021-04-14 19:00:30.406: INFO @decay_lr : LR updated to `0.00055350753` 2021-04-14 19:00:30.407: INFO @log_profile : T train: 128.866835 2021-04-14 19:00:30.407: INFO @log_profile : T read data: 0.841048 2021-04-14 19:00:30.407: INFO @log_profile : T hooks: 84.287656 2021-04-14 19:00:30.407: INFO @main_loop : Epoch 118 done 2021-04-14 19:00:30.407: INFO @main_loop : Training epoch 119 2021-04-14 19:02:42.780: INFO @log_variables: train loss nanmean: 0.028002 2021-04-14 19:02:42.780: INFO @log_variables: train positive_distance nanmean: 0.560909 2021-04-14 19:02:42.780: INFO @log_variables: train negative_distance nanmean: 1.419467 2021-04-14 19:02:42.780: INFO @log_variables: train distance_margin nanmean: 0.858558 2021-04-14 19:02:42.781: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:04:06.391: INFO @metrics_hook: train matching accuracy: 0.9904599135514441, N=110586, DB_SIZE=110592, N_IDS=11544 2021-04-14 19:04:06.594: INFO @decay_lr : LR updated to `0.00055074` 2021-04-14 19:04:06.595: INFO @log_profile : T train: 129.525132 2021-04-14 19:04:06.595: INFO @log_profile : T read data: 2.495381 2021-04-14 19:04:06.595: INFO @log_profile : T hooks: 84.116180 2021-04-14 19:04:06.595: INFO @main_loop : Epoch 119 done 2021-04-14 19:04:06.595: INFO @main_loop : Training epoch 120 2021-04-14 19:06:20.297: INFO @log_variables: train loss nanmean: 0.029124 2021-04-14 19:06:20.297: INFO @log_variables: train positive_distance nanmean: 0.561418 2021-04-14 19:06:20.298: INFO @log_variables: train negative_distance nanmean: 1.419435 2021-04-14 19:06:20.298: INFO @log_variables: train distance_margin nanmean: 0.858016 2021-04-14 19:06:20.298: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:07:44.548: INFO @metrics_hook: train matching accuracy: 0.9906860661741435, N=110587, DB_SIZE=110592, N_IDS=11569 2021-04-14 19:07:44.730: INFO @decay_lr : LR updated to `0.0005479863` 2021-04-14 19:07:44.731: INFO @log_profile : T train: 130.959423 2021-04-14 19:07:44.731: INFO @log_profile : T read data: 2.392586 2021-04-14 19:07:44.731: INFO @log_profile : T hooks: 84.733057 2021-04-14 19:07:44.732: INFO @main_loop : Epoch 120 done 2021-04-14 19:07:44.732: INFO @main_loop : Training epoch 121 2021-04-14 19:09:57.773: INFO @log_variables: train loss nanmean: 0.028490 2021-04-14 19:09:57.773: INFO @log_variables: train positive_distance nanmean: 0.561220 2021-04-14 19:09:57.773: INFO @log_variables: train negative_distance nanmean: 1.419475 2021-04-14 19:09:57.773: INFO @log_variables: train distance_margin nanmean: 0.858255 2021-04-14 19:09:57.773: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:11:22.645: INFO @metrics_hook: train matching accuracy: 0.9911741300730667, N=110584, DB_SIZE=110592, N_IDS=11557 2021-04-14 19:11:22.804: INFO @decay_lr : LR updated to `0.00054524635` 2021-04-14 19:11:22.805: INFO @log_profile : T train: 130.402267 2021-04-14 19:11:22.805: INFO @log_profile : T read data: 2.282097 2021-04-14 19:11:22.805: INFO @log_profile : T hooks: 85.336699 2021-04-14 19:11:22.805: INFO @main_loop : Epoch 121 done 2021-04-14 19:11:22.805: INFO @main_loop : Training epoch 122 2021-04-14 19:13:36.727: INFO @log_variables: train loss nanmean: 0.028483 2021-04-14 19:13:36.727: INFO @log_variables: train positive_distance nanmean: 0.561143 2021-04-14 19:13:36.727: INFO @log_variables: train negative_distance nanmean: 1.419481 2021-04-14 19:13:36.727: INFO @log_variables: train distance_margin nanmean: 0.858338 2021-04-14 19:13:36.727: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:15:03.266: INFO @metrics_hook: train matching accuracy: 0.9907763259031515, N=110585, DB_SIZE=110592, N_IDS=11547 2021-04-14 19:15:03.427: INFO @decay_lr : LR updated to `0.00054252014` 2021-04-14 19:15:03.428: INFO @log_profile : T train: 131.172517 2021-04-14 19:15:03.428: INFO @log_profile : T read data: 2.390682 2021-04-14 19:15:03.428: INFO @log_profile : T hooks: 87.006278 2021-04-14 19:15:03.428: INFO @main_loop : Epoch 122 done 2021-04-14 19:15:03.428: INFO @main_loop : Training epoch 123 2021-04-14 19:17:13.425: INFO @log_variables: train loss nanmean: 0.028278 2021-04-14 19:17:13.425: INFO @log_variables: train positive_distance nanmean: 0.560905 2021-04-14 19:17:13.425: INFO @log_variables: train negative_distance nanmean: 1.419499 2021-04-14 19:17:13.425: INFO @log_variables: train distance_margin nanmean: 0.858593 2021-04-14 19:17:13.425: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:18:40.260: INFO @metrics_hook: train matching accuracy: 0.9908304169718672, N=110583, DB_SIZE=110592, N_IDS=11550 2021-04-14 19:18:40.433: INFO @decay_lr : LR updated to `0.00053980754` 2021-04-14 19:18:40.434: INFO @log_profile : T train: 127.368125 2021-04-14 19:18:40.434: INFO @log_profile : T read data: 2.290908 2021-04-14 19:18:40.434: INFO @log_profile : T hooks: 87.294004 2021-04-14 19:18:40.434: INFO @main_loop : Epoch 123 done 2021-04-14 19:18:40.434: INFO @main_loop : Training epoch 124 2021-04-14 19:20:52.044: INFO @log_variables: train loss nanmean: 0.027733 2021-04-14 19:20:52.044: INFO @log_variables: train positive_distance nanmean: 0.560856 2021-04-14 19:20:52.044: INFO @log_variables: train negative_distance nanmean: 1.419458 2021-04-14 19:20:52.044: INFO @log_variables: train distance_margin nanmean: 0.858602 2021-04-14 19:20:52.044: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:22:20.322: INFO @metrics_hook: train matching accuracy: 0.991237747314356, N=110588, DB_SIZE=110592, N_IDS=11555 2021-04-14 19:22:20.520: INFO @decay_lr : LR updated to `0.0005371085` 2021-04-14 19:22:20.521: INFO @log_profile : T train: 128.877974 2021-04-14 19:22:20.521: INFO @log_profile : T read data: 2.376969 2021-04-14 19:22:20.521: INFO @log_profile : T hooks: 88.778183 2021-04-14 19:22:20.521: INFO @main_loop : Epoch 124 done 2021-04-14 19:22:20.521: INFO @main_loop : Training epoch 125 2021-04-14 19:24:32.725: INFO @log_variables: train loss nanmean: 0.027225 2021-04-14 19:24:32.725: INFO @log_variables: train positive_distance nanmean: 0.560055 2021-04-14 19:24:32.725: INFO @log_variables: train negative_distance nanmean: 1.419464 2021-04-14 19:24:32.725: INFO @log_variables: train distance_margin nanmean: 0.859409 2021-04-14 19:24:32.725: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:25:57.872: INFO @metrics_hook: train matching accuracy: 0.9907309441776765, N=110583, DB_SIZE=110592, N_IDS=11550 2021-04-14 19:25:58.037: INFO @decay_lr : LR updated to `0.000534423` 2021-04-14 19:25:58.038: INFO @log_profile : T train: 130.968011 2021-04-14 19:25:58.039: INFO @log_profile : T read data: 0.880833 2021-04-14 19:25:58.039: INFO @log_profile : T hooks: 85.615919 2021-04-14 19:25:58.039: INFO @main_loop : Epoch 125 done 2021-04-14 19:25:58.039: INFO @main_loop : Training epoch 126 2021-04-14 19:28:12.154: INFO @log_variables: train loss nanmean: 0.027241 2021-04-14 19:28:12.154: INFO @log_variables: train positive_distance nanmean: 0.560528 2021-04-14 19:28:12.154: INFO @log_variables: train negative_distance nanmean: 1.419477 2021-04-14 19:28:12.154: INFO @log_variables: train distance_margin nanmean: 0.858949 2021-04-14 19:28:12.154: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:29:37.529: INFO @metrics_hook: train matching accuracy: 0.9910564915041191, N=110583, DB_SIZE=110592, N_IDS=11518 2021-04-14 19:29:37.719: INFO @decay_lr : LR updated to `0.00053175085` 2021-04-14 19:29:37.721: INFO @log_profile : T train: 131.482258 2021-04-14 19:29:37.721: INFO @log_profile : T read data: 2.285605 2021-04-14 19:29:37.721: INFO @log_profile : T hooks: 85.860551 2021-04-14 19:29:37.721: INFO @main_loop : Epoch 126 done 2021-04-14 19:29:37.721: INFO @main_loop : Training epoch 127 2021-04-14 19:31:51.764: INFO @log_variables: train loss nanmean: 0.027429 2021-04-14 19:31:51.765: INFO @log_variables: train positive_distance nanmean: 0.560272 2021-04-14 19:31:51.765: INFO @log_variables: train negative_distance nanmean: 1.419482 2021-04-14 19:31:51.765: INFO @log_variables: train distance_margin nanmean: 0.859210 2021-04-14 19:31:51.765: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:33:17.104: INFO @metrics_hook: train matching accuracy: 0.9905686822378356, N=110589, DB_SIZE=110592, N_IDS=11561 2021-04-14 19:33:17.281: INFO @decay_lr : LR updated to `0.0005290921` 2021-04-14 19:33:17.282: INFO @log_profile : T train: 131.409650 2021-04-14 19:33:17.282: INFO @log_profile : T read data: 2.272689 2021-04-14 19:33:17.282: INFO @log_profile : T hooks: 85.826794 2021-04-14 19:33:17.282: INFO @main_loop : Epoch 127 done 2021-04-14 19:33:17.282: INFO @main_loop : Training epoch 128 2021-04-14 19:35:27.332: INFO @log_variables: train loss nanmean: 0.026710 2021-04-14 19:35:27.332: INFO @log_variables: train positive_distance nanmean: 0.559026 2021-04-14 19:35:27.332: INFO @log_variables: train negative_distance nanmean: 1.419477 2021-04-14 19:35:27.332: INFO @log_variables: train distance_margin nanmean: 0.860451 2021-04-14 19:35:27.333: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:36:51.510: INFO @metrics_hook: train matching accuracy: 0.9907130261789574, N=110585, DB_SIZE=110592, N_IDS=11559 2021-04-14 19:36:51.701: INFO @decay_lr : LR updated to `0.0005264466` 2021-04-14 19:36:51.702: INFO @log_profile : T train: 127.448467 2021-04-14 19:36:51.702: INFO @log_profile : T read data: 2.269736 2021-04-14 19:36:51.702: INFO @log_profile : T hooks: 84.649072 2021-04-14 19:36:51.702: INFO @main_loop : Epoch 128 done 2021-04-14 19:36:51.702: INFO @main_loop : Training epoch 129 2021-04-14 19:39:03.862: INFO @log_variables: train loss nanmean: 0.026984 2021-04-14 19:39:03.862: INFO @log_variables: train positive_distance nanmean: 0.560412 2021-04-14 19:39:03.863: INFO @log_variables: train negative_distance nanmean: 1.419464 2021-04-14 19:39:03.863: INFO @log_variables: train distance_margin nanmean: 0.859052 2021-04-14 19:39:03.863: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:40:27.736: INFO @metrics_hook: train matching accuracy: 0.9911834121551357, N=110587, DB_SIZE=110592, N_IDS=11522 2021-04-14 19:40:27.904: INFO @decay_lr : LR updated to `0.0005238144` 2021-04-14 19:40:27.905: INFO @log_profile : T train: 129.573470 2021-04-14 19:40:27.905: INFO @log_profile : T read data: 2.245936 2021-04-14 19:40:27.905: INFO @log_profile : T hooks: 84.331647 2021-04-14 19:40:27.905: INFO @main_loop : Epoch 129 done 2021-04-14 19:40:27.905: INFO @main_loop : Training epoch 130 2021-04-14 19:42:41.338: INFO @log_variables: train loss nanmean: 0.027013 2021-04-14 19:42:41.338: INFO @log_variables: train positive_distance nanmean: 0.560316 2021-04-14 19:42:41.338: INFO @log_variables: train negative_distance nanmean: 1.419452 2021-04-14 19:42:41.338: INFO @log_variables: train distance_margin nanmean: 0.859136 2021-04-14 19:42:41.338: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:44:05.441: INFO @metrics_hook: train matching accuracy: 0.9904957407172957, N=110582, DB_SIZE=110592, N_IDS=11525 2021-04-14 19:44:05.635: INFO @decay_lr : LR updated to `0.00052119535` 2021-04-14 19:44:05.637: INFO @log_profile : T train: 130.579068 2021-04-14 19:44:05.637: INFO @log_profile : T read data: 2.485317 2021-04-14 19:44:05.637: INFO @log_profile : T hooks: 84.614396 2021-04-14 19:44:05.637: INFO @main_loop : Epoch 130 done 2021-04-14 19:44:05.637: INFO @main_loop : Training epoch 131 2021-04-14 19:46:18.531: INFO @log_variables: train loss nanmean: 0.028171 2021-04-14 19:46:18.531: INFO @log_variables: train positive_distance nanmean: 0.560846 2021-04-14 19:46:18.531: INFO @log_variables: train negative_distance nanmean: 1.419446 2021-04-14 19:46:18.531: INFO @log_variables: train distance_margin nanmean: 0.858600 2021-04-14 19:46:18.531: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:47:43.452: INFO @metrics_hook: train matching accuracy: 0.9904779990233845, N=110586, DB_SIZE=110592, N_IDS=11563 2021-04-14 19:47:43.624: INFO @decay_lr : LR updated to `0.0005185894` 2021-04-14 19:47:43.625: INFO @log_profile : T train: 130.185967 2021-04-14 19:47:43.625: INFO @log_profile : T read data: 2.340267 2021-04-14 19:47:43.625: INFO @log_profile : T hooks: 85.409329 2021-04-14 19:47:43.626: INFO @main_loop : Epoch 131 done 2021-04-14 19:47:43.626: INFO @main_loop : Training epoch 132 2021-04-14 19:49:56.299: INFO @log_variables: train loss nanmean: 0.025978 2021-04-14 19:49:56.299: INFO @log_variables: train positive_distance nanmean: 0.559106 2021-04-14 19:49:56.299: INFO @log_variables: train negative_distance nanmean: 1.419484 2021-04-14 19:49:56.299: INFO @log_variables: train distance_margin nanmean: 0.860378 2021-04-14 19:49:56.299: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:51:19.308: INFO @metrics_hook: train matching accuracy: 0.991120193872753, N=110588, DB_SIZE=110592, N_IDS=11529 2021-04-14 19:51:19.521: INFO @decay_lr : LR updated to `0.0005159965` 2021-04-14 19:51:19.522: INFO @log_profile : T train: 129.935507 2021-04-14 19:51:19.522: INFO @log_profile : T read data: 2.394101 2021-04-14 19:51:19.522: INFO @log_profile : T hooks: 83.514260 2021-04-14 19:51:19.522: INFO @main_loop : Epoch 132 done 2021-04-14 19:51:19.522: INFO @main_loop : Training epoch 133 2021-04-14 19:53:30.370: INFO @log_variables: train loss nanmean: 0.026238 2021-04-14 19:53:30.370: INFO @log_variables: train positive_distance nanmean: 0.559520 2021-04-14 19:53:30.370: INFO @log_variables: train negative_distance nanmean: 1.419451 2021-04-14 19:53:30.370: INFO @log_variables: train distance_margin nanmean: 0.859931 2021-04-14 19:53:30.370: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:54:54.535: INFO @metrics_hook: train matching accuracy: 0.9904780851275466, N=110587, DB_SIZE=110592, N_IDS=11524 2021-04-14 19:54:54.699: INFO @decay_lr : LR updated to `0.0005134165` 2021-04-14 19:54:54.700: INFO @log_profile : T train: 129.670990 2021-04-14 19:54:54.700: INFO @log_profile : T read data: 0.824801 2021-04-14 19:54:54.700: INFO @log_profile : T hooks: 84.630790 2021-04-14 19:54:54.700: INFO @main_loop : Epoch 133 done 2021-04-14 19:54:54.700: INFO @main_loop : Training epoch 134 2021-04-14 19:57:09.033: INFO @log_variables: train loss nanmean: 0.026399 2021-04-14 19:57:09.033: INFO @log_variables: train positive_distance nanmean: 0.560235 2021-04-14 19:57:09.033: INFO @log_variables: train negative_distance nanmean: 1.419464 2021-04-14 19:57:09.033: INFO @log_variables: train distance_margin nanmean: 0.859229 2021-04-14 19:57:09.033: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 19:58:33.792: INFO @metrics_hook: train matching accuracy: 0.9904960844953249, N=110586, DB_SIZE=110592, N_IDS=11532 2021-04-14 19:58:33.978: INFO @decay_lr : LR updated to `0.0005108494` 2021-04-14 19:58:33.979: INFO @log_profile : T train: 131.605359 2021-04-14 19:58:33.979: INFO @log_profile : T read data: 2.384299 2021-04-14 19:58:33.979: INFO @log_profile : T hooks: 85.237441 2021-04-14 19:58:33.979: INFO @main_loop : Epoch 134 done 2021-04-14 19:58:33.979: INFO @main_loop : Training epoch 135 2021-04-14 20:00:47.717: INFO @log_variables: train loss nanmean: 0.026229 2021-04-14 20:00:47.717: INFO @log_variables: train positive_distance nanmean: 0.559495 2021-04-14 20:00:47.717: INFO @log_variables: train negative_distance nanmean: 1.419448 2021-04-14 20:00:47.717: INFO @log_variables: train distance_margin nanmean: 0.859953 2021-04-14 20:00:47.717: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:02:12.113: INFO @metrics_hook: train matching accuracy: 0.9906410221631446, N=110589, DB_SIZE=110592, N_IDS=11587 2021-04-14 20:02:12.304: INFO @decay_lr : LR updated to `0.00050829514` 2021-04-14 20:02:12.306: INFO @log_profile : T train: 131.097666 2021-04-14 20:02:12.306: INFO @log_profile : T read data: 2.298754 2021-04-14 20:02:12.306: INFO @log_profile : T hooks: 84.878305 2021-04-14 20:02:12.306: INFO @main_loop : Epoch 135 done 2021-04-14 20:02:12.306: INFO @main_loop : Training epoch 136 2021-04-14 20:04:26.208: INFO @log_variables: train loss nanmean: 0.025472 2021-04-14 20:04:26.208: INFO @log_variables: train positive_distance nanmean: 0.559318 2021-04-14 20:04:26.208: INFO @log_variables: train negative_distance nanmean: 1.419469 2021-04-14 20:04:26.208: INFO @log_variables: train distance_margin nanmean: 0.860152 2021-04-14 20:04:26.208: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:05:50.506: INFO @metrics_hook: train matching accuracy: 0.9907491974499254, N=110585, DB_SIZE=110592, N_IDS=11565 2021-04-14 20:05:50.688: INFO @decay_lr : LR updated to `0.0005057537` 2021-04-14 20:05:50.689: INFO @log_profile : T train: 131.239727 2021-04-14 20:05:50.689: INFO @log_profile : T read data: 2.305310 2021-04-14 20:05:50.689: INFO @log_profile : T hooks: 84.785407 2021-04-14 20:05:50.689: INFO @main_loop : Epoch 136 done 2021-04-14 20:05:50.689: INFO @main_loop : Training epoch 137 2021-04-14 20:08:05.639: INFO @log_variables: train loss nanmean: 0.026787 2021-04-14 20:08:05.639: INFO @log_variables: train positive_distance nanmean: 0.559612 2021-04-14 20:08:05.639: INFO @log_variables: train negative_distance nanmean: 1.419469 2021-04-14 20:08:05.639: INFO @log_variables: train distance_margin nanmean: 0.859858 2021-04-14 20:08:05.639: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:09:29.604: INFO @metrics_hook: train matching accuracy: 0.990631979672481, N=110589, DB_SIZE=110592, N_IDS=11541 2021-04-14 20:09:29.782: INFO @decay_lr : LR updated to `0.0005032249` 2021-04-14 20:09:29.783: INFO @log_profile : T train: 132.269245 2021-04-14 20:09:29.783: INFO @log_profile : T read data: 2.357907 2021-04-14 20:09:29.783: INFO @log_profile : T hooks: 84.415528 2021-04-14 20:09:29.783: INFO @main_loop : Epoch 137 done 2021-04-14 20:09:29.783: INFO @main_loop : Training epoch 138 2021-04-14 20:11:43.411: INFO @log_variables: train loss nanmean: 0.025222 2021-04-14 20:11:43.413: INFO @log_variables: train positive_distance nanmean: 0.558408 2021-04-14 20:11:43.413: INFO @log_variables: train negative_distance nanmean: 1.419503 2021-04-14 20:11:43.413: INFO @log_variables: train distance_margin nanmean: 0.861095 2021-04-14 20:11:43.413: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:13:08.049: INFO @metrics_hook: train matching accuracy: 0.990703899298271, N=110584, DB_SIZE=110592, N_IDS=11564 2021-04-14 20:13:08.218: INFO @decay_lr : LR updated to `0.00050070876` 2021-04-14 20:13:08.219: INFO @log_profile : T train: 131.018835 2021-04-14 20:13:08.219: INFO @log_profile : T read data: 2.279160 2021-04-14 20:13:08.219: INFO @log_profile : T hooks: 85.086265 2021-04-14 20:13:08.219: INFO @main_loop : Epoch 138 done 2021-04-14 20:13:08.219: INFO @main_loop : Training epoch 139 2021-04-14 20:15:21.614: INFO @log_variables: train loss nanmean: 0.026031 2021-04-14 20:15:21.615: INFO @log_variables: train positive_distance nanmean: 0.559669 2021-04-14 20:15:21.615: INFO @log_variables: train negative_distance nanmean: 1.419490 2021-04-14 20:15:21.615: INFO @log_variables: train distance_margin nanmean: 0.859821 2021-04-14 20:15:21.615: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:16:45.279: INFO @metrics_hook: train matching accuracy: 0.990342183317659, N=110584, DB_SIZE=110592, N_IDS=11527 2021-04-14 20:16:45.451: INFO @decay_lr : LR updated to `0.0004982052` 2021-04-14 20:16:45.453: INFO @log_profile : T train: 130.750335 2021-04-14 20:16:45.453: INFO @log_profile : T read data: 2.300270 2021-04-14 20:16:45.453: INFO @log_profile : T hooks: 84.130857 2021-04-14 20:16:45.453: INFO @main_loop : Epoch 139 done 2021-04-14 20:16:45.453: INFO @main_loop : Training epoch 140 2021-04-14 20:18:57.887: INFO @log_variables: train loss nanmean: 0.026169 2021-04-14 20:18:57.888: INFO @log_variables: train positive_distance nanmean: 0.559703 2021-04-14 20:18:57.888: INFO @log_variables: train negative_distance nanmean: 1.419486 2021-04-14 20:18:57.888: INFO @log_variables: train distance_margin nanmean: 0.859783 2021-04-14 20:18:57.888: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:20:21.473: INFO @metrics_hook: train matching accuracy: 0.990966306765775, N=110586, DB_SIZE=110592, N_IDS=11538 2021-04-14 20:20:21.644: INFO @decay_lr : LR updated to `0.00049571414` 2021-04-14 20:20:21.645: INFO @log_profile : T train: 131.231617 2021-04-14 20:20:21.645: INFO @log_profile : T read data: 0.840219 2021-04-14 20:20:21.645: INFO @log_profile : T hooks: 84.068528 2021-04-14 20:20:21.645: INFO @main_loop : Epoch 140 done 2021-04-14 20:20:21.645: INFO @main_loop : Training epoch 141 2021-04-14 20:22:36.998: INFO @log_variables: train loss nanmean: 0.025917 2021-04-14 20:22:36.998: INFO @log_variables: train positive_distance nanmean: 0.559917 2021-04-14 20:22:36.998: INFO @log_variables: train negative_distance nanmean: 1.419501 2021-04-14 20:22:36.998: INFO @log_variables: train distance_margin nanmean: 0.859584 2021-04-14 20:22:36.998: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:24:00.231: INFO @metrics_hook: train matching accuracy: 0.9899083979129554, N=110587, DB_SIZE=110592, N_IDS=11586 2021-04-14 20:24:00.394: INFO @decay_lr : LR updated to `0.00049323554` 2021-04-14 20:24:00.396: INFO @log_profile : T train: 132.737689 2021-04-14 20:24:00.396: INFO @log_profile : T read data: 2.286687 2021-04-14 20:24:00.396: INFO @log_profile : T hooks: 83.673845 2021-04-14 20:24:00.396: INFO @main_loop : Epoch 141 done 2021-04-14 20:24:00.396: INFO @main_loop : Training epoch 142 2021-04-14 20:26:12.960: INFO @log_variables: train loss nanmean: 0.026715 2021-04-14 20:26:12.960: INFO @log_variables: train positive_distance nanmean: 0.559899 2021-04-14 20:26:12.960: INFO @log_variables: train negative_distance nanmean: 1.419512 2021-04-14 20:26:12.960: INFO @log_variables: train distance_margin nanmean: 0.859613 2021-04-14 20:26:12.960: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:27:37.236: INFO @metrics_hook: train matching accuracy: 0.9901528167103717, N=110590, DB_SIZE=110592, N_IDS=11559 2021-04-14 20:27:37.404: INFO @decay_lr : LR updated to `0.00049076934` 2021-04-14 20:27:37.405: INFO @log_profile : T train: 129.800693 2021-04-14 20:27:37.405: INFO @log_profile : T read data: 2.426528 2021-04-14 20:27:37.405: INFO @log_profile : T hooks: 84.729734 2021-04-14 20:27:37.405: INFO @main_loop : Epoch 142 done 2021-04-14 20:27:37.406: INFO @main_loop : Training epoch 143 2021-04-14 20:29:49.701: INFO @log_variables: train loss nanmean: 0.025664 2021-04-14 20:29:49.701: INFO @log_variables: train positive_distance nanmean: 0.558512 2021-04-14 20:29:49.701: INFO @log_variables: train negative_distance nanmean: 1.419513 2021-04-14 20:29:49.701: INFO @log_variables: train distance_margin nanmean: 0.861001 2021-04-14 20:29:49.701: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:31:13.599: INFO @metrics_hook: train matching accuracy: 0.9907405732887241, N=110590, DB_SIZE=110592, N_IDS=11573 2021-04-14 20:31:13.768: INFO @decay_lr : LR updated to `0.0004883155` 2021-04-14 20:31:13.769: INFO @log_profile : T train: 129.758792 2021-04-14 20:31:13.770: INFO @log_profile : T read data: 2.196281 2021-04-14 20:31:13.770: INFO @log_profile : T hooks: 84.356981 2021-04-14 20:31:13.770: INFO @main_loop : Epoch 143 done 2021-04-14 20:31:13.770: INFO @main_loop : Training epoch 144 2021-04-14 20:33:26.296: INFO @log_variables: train loss nanmean: 0.025661 2021-04-14 20:33:26.297: INFO @log_variables: train positive_distance nanmean: 0.558525 2021-04-14 20:33:26.297: INFO @log_variables: train negative_distance nanmean: 1.419486 2021-04-14 20:33:26.297: INFO @log_variables: train distance_margin nanmean: 0.860961 2021-04-14 20:33:26.297: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:34:50.547: INFO @metrics_hook: train matching accuracy: 0.9904870417593548, N=110586, DB_SIZE=110592, N_IDS=11516 2021-04-14 20:34:50.722: INFO @decay_lr : LR updated to `0.0004858739` 2021-04-14 20:34:50.723: INFO @log_profile : T train: 129.956833 2021-04-14 20:34:50.723: INFO @log_profile : T read data: 2.233625 2021-04-14 20:34:50.723: INFO @log_profile : T hooks: 84.711714 2021-04-14 20:34:50.723: INFO @main_loop : Epoch 144 done 2021-04-14 20:34:50.723: INFO @main_loop : Training epoch 145 2021-04-14 20:37:03.263: INFO @log_variables: train loss nanmean: 0.025019 2021-04-14 20:37:03.263: INFO @log_variables: train positive_distance nanmean: 0.559357 2021-04-14 20:37:03.263: INFO @log_variables: train negative_distance nanmean: 1.419489 2021-04-14 20:37:03.263: INFO @log_variables: train distance_margin nanmean: 0.860132 2021-04-14 20:37:03.263: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:38:27.614: INFO @metrics_hook: train matching accuracy: 0.990071345251336, N=110589, DB_SIZE=110592, N_IDS=11571 2021-04-14 20:38:27.792: INFO @decay_lr : LR updated to `0.00048344454` 2021-04-14 20:38:27.793: INFO @log_profile : T train: 129.858253 2021-04-14 20:38:27.793: INFO @log_profile : T read data: 2.337795 2021-04-14 20:38:27.793: INFO @log_profile : T hooks: 84.821829 2021-04-14 20:38:27.793: INFO @main_loop : Epoch 145 done 2021-04-14 20:38:27.793: INFO @main_loop : Training epoch 146 2021-04-14 20:40:40.807: INFO @log_variables: train loss nanmean: 0.024573 2021-04-14 20:40:40.807: INFO @log_variables: train positive_distance nanmean: 0.558443 2021-04-14 20:40:40.807: INFO @log_variables: train negative_distance nanmean: 1.419495 2021-04-14 20:40:40.807: INFO @log_variables: train distance_margin nanmean: 0.861052 2021-04-14 20:40:40.807: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:42:04.805: INFO @metrics_hook: train matching accuracy: 0.9908397084621923, N=110586, DB_SIZE=110592, N_IDS=11551 2021-04-14 20:42:04.995: INFO @decay_lr : LR updated to `0.00048102732` 2021-04-14 20:42:04.996: INFO @log_profile : T train: 130.403301 2021-04-14 20:42:04.996: INFO @log_profile : T read data: 2.247411 2021-04-14 20:42:04.996: INFO @log_profile : T hooks: 84.500083 2021-04-14 20:42:04.996: INFO @main_loop : Epoch 146 done 2021-04-14 20:42:04.996: INFO @main_loop : Training epoch 147 2021-04-14 20:44:19.425: INFO @log_variables: train loss nanmean: 0.024476 2021-04-14 20:44:19.426: INFO @log_variables: train positive_distance nanmean: 0.557785 2021-04-14 20:44:19.426: INFO @log_variables: train negative_distance nanmean: 1.419485 2021-04-14 20:44:19.426: INFO @log_variables: train distance_margin nanmean: 0.861700 2021-04-14 20:44:19.426: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:45:44.113: INFO @metrics_hook: train matching accuracy: 0.9902157596802489, N=110586, DB_SIZE=110592, N_IDS=11539 2021-04-14 20:45:44.287: INFO @decay_lr : LR updated to `0.00047862218` 2021-04-14 20:45:44.288: INFO @log_profile : T train: 131.678135 2021-04-14 20:45:44.288: INFO @log_profile : T read data: 2.384082 2021-04-14 20:45:44.288: INFO @log_profile : T hooks: 85.178487 2021-04-14 20:45:44.289: INFO @main_loop : Epoch 147 done 2021-04-14 20:45:44.289: INFO @main_loop : Training epoch 148 2021-04-14 20:47:57.726: INFO @log_variables: train loss nanmean: 0.024653 2021-04-14 20:47:57.727: INFO @log_variables: train positive_distance nanmean: 0.558413 2021-04-14 20:47:57.727: INFO @log_variables: train negative_distance nanmean: 1.419514 2021-04-14 20:47:57.727: INFO @log_variables: train distance_margin nanmean: 0.861102 2021-04-14 20:47:57.727: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:49:21.860: INFO @metrics_hook: train matching accuracy: 0.9903878324245631, N=110589, DB_SIZE=110592, N_IDS=11539 2021-04-14 20:49:22.024: INFO @decay_lr : LR updated to `0.00047622906` 2021-04-14 20:49:22.025: INFO @log_profile : T train: 132.254303 2021-04-14 20:49:22.025: INFO @log_profile : T read data: 0.840992 2021-04-14 20:49:22.025: INFO @log_profile : T hooks: 84.589422 2021-04-14 20:49:22.025: INFO @main_loop : Epoch 148 done 2021-04-14 20:49:22.025: INFO @main_loop : Training epoch 149 2021-04-14 20:51:37.722: INFO @log_variables: train loss nanmean: 0.025313 2021-04-14 20:51:37.722: INFO @log_variables: train positive_distance nanmean: 0.558451 2021-04-14 20:51:37.722: INFO @log_variables: train negative_distance nanmean: 1.419503 2021-04-14 20:51:37.722: INFO @log_variables: train distance_margin nanmean: 0.861053 2021-04-14 20:51:37.722: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:53:01.700: INFO @metrics_hook: train matching accuracy: 0.9901976742083085, N=110586, DB_SIZE=110592, N_IDS=11542 2021-04-14 20:53:01.874: INFO @decay_lr : LR updated to `0.0004738479` 2021-04-14 20:53:01.875: INFO @log_profile : T train: 133.031359 2021-04-14 20:53:01.875: INFO @log_profile : T read data: 2.312250 2021-04-14 20:53:01.875: INFO @log_profile : T hooks: 84.455036 2021-04-14 20:53:01.875: INFO @main_loop : Epoch 149 done 2021-04-14 20:53:01.875: INFO @main_loop : Training epoch 150 2021-04-14 20:55:14.322: INFO @log_variables: train loss nanmean: 0.024656 2021-04-14 20:55:14.322: INFO @log_variables: train positive_distance nanmean: 0.557937 2021-04-14 20:55:14.322: INFO @log_variables: train negative_distance nanmean: 1.419506 2021-04-14 20:55:14.322: INFO @log_variables: train distance_margin nanmean: 0.861569 2021-04-14 20:55:14.322: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 20:56:38.219: INFO @metrics_hook: train matching accuracy: 0.991264875031649, N=110588, DB_SIZE=110592, N_IDS=11547 2021-04-14 20:56:38.402: INFO @decay_lr : LR updated to `0.00047147868` 2021-04-14 20:56:38.675: INFO @save : Model saved to: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f/model_150.ckpt 2021-04-14 20:56:38.697: INFO @log_profile : T train: 129.636145 2021-04-14 20:56:38.697: INFO @log_profile : T read data: 2.443057 2021-04-14 20:56:38.697: INFO @log_profile : T hooks: 84.690775 2021-04-14 20:56:38.698: INFO @main_loop : Epoch 150 done 2021-04-14 20:56:38.698: INFO @main_loop : Training epoch 151 2021-04-14 20:58:52.222: INFO @log_variables: train loss nanmean: 0.024663 2021-04-14 20:58:52.222: INFO @log_variables: train positive_distance nanmean: 0.557809 2021-04-14 20:58:52.222: INFO @log_variables: train negative_distance nanmean: 1.419515 2021-04-14 20:58:52.222: INFO @log_variables: train distance_margin nanmean: 0.861706 2021-04-14 20:58:52.223: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:00:16.794: INFO @metrics_hook: train matching accuracy: 0.9907130261789574, N=110585, DB_SIZE=110592, N_IDS=11554 2021-04-14 21:00:16.987: INFO @decay_lr : LR updated to `0.0004691213` 2021-04-14 21:00:16.988: INFO @log_profile : T train: 130.957145 2021-04-14 21:00:16.988: INFO @log_profile : T read data: 2.219025 2021-04-14 21:00:16.988: INFO @log_profile : T hooks: 85.062258 2021-04-14 21:00:16.989: INFO @main_loop : Epoch 151 done 2021-04-14 21:00:16.989: INFO @main_loop : Training epoch 152 2021-04-14 21:02:32.027: INFO @log_variables: train loss nanmean: 0.024432 2021-04-14 21:02:32.028: INFO @log_variables: train positive_distance nanmean: 0.558844 2021-04-14 21:02:32.028: INFO @log_variables: train negative_distance nanmean: 1.419512 2021-04-14 21:02:32.028: INFO @log_variables: train distance_margin nanmean: 0.860669 2021-04-14 21:02:32.028: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:03:56.495: INFO @metrics_hook: train matching accuracy: 0.9899172582176606, N=110585, DB_SIZE=110592, N_IDS=11555 2021-04-14 21:03:56.684: INFO @decay_lr : LR updated to `0.0004667757` 2021-04-14 21:03:56.685: INFO @log_profile : T train: 132.311736 2021-04-14 21:03:56.686: INFO @log_profile : T read data: 2.382181 2021-04-14 21:03:56.686: INFO @log_profile : T hooks: 84.951618 2021-04-14 21:03:56.686: INFO @main_loop : Epoch 152 done 2021-04-14 21:03:56.686: INFO @main_loop : Training epoch 153 2021-04-14 21:06:10.003: INFO @log_variables: train loss nanmean: 0.023453 2021-04-14 21:06:10.004: INFO @log_variables: train positive_distance nanmean: 0.557464 2021-04-14 21:06:10.004: INFO @log_variables: train negative_distance nanmean: 1.419518 2021-04-14 21:06:10.004: INFO @log_variables: train distance_margin nanmean: 0.862054 2021-04-14 21:06:10.004: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:07:34.634: INFO @metrics_hook: train matching accuracy: 0.9907489464831528, N=110582, DB_SIZE=110592, N_IDS=11556 2021-04-14 21:07:34.803: INFO @decay_lr : LR updated to `0.00046444184` 2021-04-14 21:07:34.804: INFO @log_profile : T train: 130.883291 2021-04-14 21:07:34.804: INFO @log_profile : T read data: 2.109828 2021-04-14 21:07:34.804: INFO @log_profile : T hooks: 85.074334 2021-04-14 21:07:34.804: INFO @main_loop : Epoch 153 done 2021-04-14 21:07:34.804: INFO @main_loop : Training epoch 154 2021-04-14 21:09:47.761: INFO @log_variables: train loss nanmean: 0.023389 2021-04-14 21:09:47.761: INFO @log_variables: train positive_distance nanmean: 0.557013 2021-04-14 21:09:47.761: INFO @log_variables: train negative_distance nanmean: 1.419523 2021-04-14 21:09:47.761: INFO @log_variables: train distance_margin nanmean: 0.862511 2021-04-14 21:09:47.761: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:11:12.494: INFO @metrics_hook: train matching accuracy: 0.9904961704359464, N=110587, DB_SIZE=110592, N_IDS=11564 2021-04-14 21:11:12.693: INFO @decay_lr : LR updated to `0.00046211964` 2021-04-14 21:11:12.694: INFO @log_profile : T train: 130.469676 2021-04-14 21:11:12.694: INFO @log_profile : T read data: 2.143528 2021-04-14 21:11:12.694: INFO @log_profile : T hooks: 85.224502 2021-04-14 21:11:12.694: INFO @main_loop : Epoch 154 done 2021-04-14 21:11:12.694: INFO @main_loop : Training epoch 155 2021-04-14 21:13:23.543: INFO @log_variables: train loss nanmean: 0.023937 2021-04-14 21:13:23.543: INFO @log_variables: train positive_distance nanmean: 0.558237 2021-04-14 21:13:23.543: INFO @log_variables: train negative_distance nanmean: 1.419526 2021-04-14 21:13:23.543: INFO @log_variables: train distance_margin nanmean: 0.861289 2021-04-14 21:13:23.543: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:14:47.693: INFO @metrics_hook: train matching accuracy: 0.9903512262171743, N=110584, DB_SIZE=110592, N_IDS=11544 2021-04-14 21:14:47.878: INFO @decay_lr : LR updated to `0.00045980903` 2021-04-14 21:14:47.879: INFO @log_profile : T train: 129.668728 2021-04-14 21:14:47.879: INFO @log_profile : T read data: 0.847428 2021-04-14 21:14:47.879: INFO @log_profile : T hooks: 84.617192 2021-04-14 21:14:47.879: INFO @main_loop : Epoch 155 done 2021-04-14 21:14:47.879: INFO @main_loop : Training epoch 156 2021-04-14 21:17:02.553: INFO @log_variables: train loss nanmean: 0.024267 2021-04-14 21:17:02.553: INFO @log_variables: train positive_distance nanmean: 0.557273 2021-04-14 21:17:02.553: INFO @log_variables: train negative_distance nanmean: 1.419510 2021-04-14 21:17:02.553: INFO @log_variables: train distance_margin nanmean: 0.862237 2021-04-14 21:17:02.553: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:18:27.510: INFO @metrics_hook: train matching accuracy: 0.9900799392317152, N=110584, DB_SIZE=110592, N_IDS=11539 2021-04-14 21:18:27.696: INFO @decay_lr : LR updated to `0.00045750997` 2021-04-14 21:18:27.697: INFO @log_profile : T train: 132.110262 2021-04-14 21:18:27.697: INFO @log_profile : T read data: 2.223668 2021-04-14 21:18:27.697: INFO @log_profile : T hooks: 85.432905 2021-04-14 21:18:27.697: INFO @main_loop : Epoch 156 done 2021-04-14 21:18:27.698: INFO @main_loop : Training epoch 157 2021-04-14 21:20:41.899: INFO @log_variables: train loss nanmean: 0.024184 2021-04-14 21:20:41.900: INFO @log_variables: train positive_distance nanmean: 0.558393 2021-04-14 21:20:41.900: INFO @log_variables: train negative_distance nanmean: 1.419512 2021-04-14 21:20:41.900: INFO @log_variables: train distance_margin nanmean: 0.861119 2021-04-14 21:20:41.900: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:22:06.241: INFO @metrics_hook: train matching accuracy: 0.9897545801457689, N=110586, DB_SIZE=110592, N_IDS=11562 2021-04-14 21:22:06.421: INFO @decay_lr : LR updated to `0.0004552224` 2021-04-14 21:22:06.423: INFO @log_profile : T train: 131.310904 2021-04-14 21:22:06.423: INFO @log_profile : T read data: 2.528806 2021-04-14 21:22:06.423: INFO @log_profile : T hooks: 84.835326 2021-04-14 21:22:06.423: INFO @main_loop : Epoch 157 done 2021-04-14 21:22:06.423: INFO @main_loop : Training epoch 158 2021-04-14 21:24:19.841: INFO @log_variables: train loss nanmean: 0.023017 2021-04-14 21:24:19.841: INFO @log_variables: train positive_distance nanmean: 0.556936 2021-04-14 21:24:19.841: INFO @log_variables: train negative_distance nanmean: 1.419530 2021-04-14 21:24:19.841: INFO @log_variables: train distance_margin nanmean: 0.862594 2021-04-14 21:24:19.841: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:25:44.927: INFO @metrics_hook: train matching accuracy: 0.9903878324245631, N=110589, DB_SIZE=110592, N_IDS=11508 2021-04-14 21:25:45.134: INFO @decay_lr : LR updated to `0.0004529463` 2021-04-14 21:25:45.135: INFO @log_profile : T train: 130.724472 2021-04-14 21:25:45.135: INFO @log_profile : T read data: 2.355504 2021-04-14 21:25:45.136: INFO @log_profile : T hooks: 85.581016 2021-04-14 21:25:45.136: INFO @main_loop : Epoch 158 done 2021-04-14 21:25:45.136: INFO @main_loop : Training epoch 159 2021-04-14 21:27:58.837: INFO @log_variables: train loss nanmean: 0.023305 2021-04-14 21:27:58.837: INFO @log_variables: train positive_distance nanmean: 0.556815 2021-04-14 21:27:58.837: INFO @log_variables: train negative_distance nanmean: 1.419498 2021-04-14 21:27:58.837: INFO @log_variables: train distance_margin nanmean: 0.862683 2021-04-14 21:27:58.837: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:29:24.401: INFO @metrics_hook: train matching accuracy: 0.9902246256239601, N=110584, DB_SIZE=110592, N_IDS=11553 2021-04-14 21:29:24.589: INFO @decay_lr : LR updated to `0.00045068155` 2021-04-14 21:29:24.590: INFO @log_profile : T train: 130.861681 2021-04-14 21:29:24.590: INFO @log_profile : T read data: 2.485019 2021-04-14 21:29:24.590: INFO @log_profile : T hooks: 86.057447 2021-04-14 21:29:24.590: INFO @main_loop : Epoch 159 done 2021-04-14 21:29:24.590: INFO @main_loop : Training epoch 160 2021-04-14 21:31:37.579: INFO @log_variables: train loss nanmean: 0.023503 2021-04-14 21:31:37.579: INFO @log_variables: train positive_distance nanmean: 0.558115 2021-04-14 21:31:37.579: INFO @log_variables: train negative_distance nanmean: 1.419507 2021-04-14 21:31:37.579: INFO @log_variables: train distance_margin nanmean: 0.861392 2021-04-14 21:31:37.579: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:33:01.008: INFO @metrics_hook: train matching accuracy: 0.9899174405671552, N=110587, DB_SIZE=110592, N_IDS=11586 2021-04-14 21:33:01.187: INFO @decay_lr : LR updated to `0.00044842815` 2021-04-14 21:33:01.189: INFO @log_profile : T train: 130.385430 2021-04-14 21:33:01.189: INFO @log_profile : T read data: 2.262516 2021-04-14 21:33:01.189: INFO @log_profile : T hooks: 83.898795 2021-04-14 21:33:01.189: INFO @main_loop : Epoch 160 done 2021-04-14 21:33:01.189: INFO @main_loop : Training epoch 161 2021-04-14 21:35:15.923: INFO @log_variables: train loss nanmean: 0.023663 2021-04-14 21:35:15.924: INFO @log_variables: train positive_distance nanmean: 0.557299 2021-04-14 21:35:15.924: INFO @log_variables: train negative_distance nanmean: 1.419507 2021-04-14 21:35:15.924: INFO @log_variables: train distance_margin nanmean: 0.862209 2021-04-14 21:35:15.924: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:36:38.719: INFO @metrics_hook: train matching accuracy: 0.9897457228632401, N=110588, DB_SIZE=110592, N_IDS=11544 2021-04-14 21:36:38.899: INFO @decay_lr : LR updated to `0.00044618602` 2021-04-14 21:36:38.900: INFO @log_profile : T train: 132.071473 2021-04-14 21:36:38.900: INFO @log_profile : T read data: 2.299807 2021-04-14 21:36:38.900: INFO @log_profile : T hooks: 83.288079 2021-04-14 21:36:38.900: INFO @main_loop : Epoch 161 done 2021-04-14 21:36:38.900: INFO @main_loop : Training epoch 162 2021-04-14 21:38:50.257: INFO @log_variables: train loss nanmean: 0.022512 2021-04-14 21:38:50.258: INFO @log_variables: train positive_distance nanmean: 0.556290 2021-04-14 21:38:50.258: INFO @log_variables: train negative_distance nanmean: 1.419514 2021-04-14 21:38:50.258: INFO @log_variables: train distance_margin nanmean: 0.863224 2021-04-14 21:38:50.258: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:40:14.975: INFO @metrics_hook: train matching accuracy: 0.9907039833612153, N=110585, DB_SIZE=110592, N_IDS=11571 2021-04-14 21:40:15.157: INFO @decay_lr : LR updated to `0.0004439551` 2021-04-14 21:40:15.158: INFO @log_profile : T train: 130.200093 2021-04-14 21:40:15.158: INFO @log_profile : T read data: 0.821558 2021-04-14 21:40:15.158: INFO @log_profile : T hooks: 85.185076 2021-04-14 21:40:15.159: INFO @main_loop : Epoch 162 done 2021-04-14 21:40:15.159: INFO @main_loop : Training epoch 163 2021-04-14 21:42:28.191: INFO @log_variables: train loss nanmean: 0.022785 2021-04-14 21:42:28.191: INFO @log_variables: train positive_distance nanmean: 0.557035 2021-04-14 21:42:28.191: INFO @log_variables: train negative_distance nanmean: 1.419501 2021-04-14 21:42:28.191: INFO @log_variables: train distance_margin nanmean: 0.862466 2021-04-14 21:42:28.191: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:43:52.594: INFO @metrics_hook: train matching accuracy: 0.9904327853435335, N=110586, DB_SIZE=110592, N_IDS=11560 2021-04-14 21:43:52.772: INFO @decay_lr : LR updated to `0.00044173532` 2021-04-14 21:43:52.773: INFO @log_profile : T train: 130.509089 2021-04-14 21:43:52.774: INFO @log_profile : T read data: 2.159993 2021-04-14 21:43:52.774: INFO @log_profile : T hooks: 84.894756 2021-04-14 21:43:52.774: INFO @main_loop : Epoch 163 done 2021-04-14 21:43:52.774: INFO @main_loop : Training epoch 164 2021-04-14 21:46:06.057: INFO @log_variables: train loss nanmean: 0.023876 2021-04-14 21:46:06.057: INFO @log_variables: train positive_distance nanmean: 0.557227 2021-04-14 21:46:06.057: INFO @log_variables: train negative_distance nanmean: 1.419487 2021-04-14 21:46:06.057: INFO @log_variables: train distance_margin nanmean: 0.862260 2021-04-14 21:46:06.057: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:47:30.179: INFO @metrics_hook: train matching accuracy: 0.9909026785552802, N=110582, DB_SIZE=110592, N_IDS=11558 2021-04-14 21:47:30.353: INFO @decay_lr : LR updated to `0.00043952663` 2021-04-14 21:47:30.354: INFO @log_profile : T train: 130.659219 2021-04-14 21:47:30.354: INFO @log_profile : T read data: 2.256681 2021-04-14 21:47:30.354: INFO @log_profile : T hooks: 84.612141 2021-04-14 21:47:30.354: INFO @main_loop : Epoch 164 done 2021-04-14 21:47:30.354: INFO @main_loop : Training epoch 165 2021-04-14 21:49:45.928: INFO @log_variables: train loss nanmean: 0.023168 2021-04-14 21:49:45.928: INFO @log_variables: train positive_distance nanmean: 0.557649 2021-04-14 21:49:45.928: INFO @log_variables: train negative_distance nanmean: 1.419531 2021-04-14 21:49:45.928: INFO @log_variables: train distance_margin nanmean: 0.861882 2021-04-14 21:49:45.928: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:51:10.696: INFO @metrics_hook: train matching accuracy: 0.9900623926213944, N=110590, DB_SIZE=110592, N_IDS=11551 2021-04-14 21:51:10.871: INFO @decay_lr : LR updated to `0.000437329` 2021-04-14 21:51:10.872: INFO @log_profile : T train: 133.001734 2021-04-14 21:51:10.872: INFO @log_profile : T read data: 2.250386 2021-04-14 21:51:10.872: INFO @log_profile : T hooks: 85.215600 2021-04-14 21:51:10.872: INFO @main_loop : Epoch 165 done 2021-04-14 21:51:10.872: INFO @main_loop : Training epoch 166 2021-04-14 21:53:23.270: INFO @log_variables: train loss nanmean: 0.023290 2021-04-14 21:53:23.270: INFO @log_variables: train positive_distance nanmean: 0.557377 2021-04-14 21:53:23.270: INFO @log_variables: train negative_distance nanmean: 1.419546 2021-04-14 21:53:23.270: INFO @log_variables: train distance_margin nanmean: 0.862169 2021-04-14 21:53:23.270: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:54:46.943: INFO @metrics_hook: train matching accuracy: 0.9907762424943934, N=110584, DB_SIZE=110592, N_IDS=11577 2021-04-14 21:54:47.118: INFO @decay_lr : LR updated to `0.00043514234` 2021-04-14 21:54:47.119: INFO @log_profile : T train: 129.639765 2021-04-14 21:54:47.119: INFO @log_profile : T read data: 2.401642 2021-04-14 21:54:47.119: INFO @log_profile : T hooks: 84.154701 2021-04-14 21:54:47.119: INFO @main_loop : Epoch 166 done 2021-04-14 21:54:47.119: INFO @main_loop : Training epoch 167 2021-04-14 21:57:00.748: INFO @log_variables: train loss nanmean: 0.022606 2021-04-14 21:57:00.748: INFO @log_variables: train positive_distance nanmean: 0.556163 2021-04-14 21:57:00.748: INFO @log_variables: train negative_distance nanmean: 1.419527 2021-04-14 21:57:00.748: INFO @log_variables: train distance_margin nanmean: 0.863363 2021-04-14 21:57:00.748: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 21:58:25.035: INFO @metrics_hook: train matching accuracy: 0.991191737958726, N=110578, DB_SIZE=110592, N_IDS=11569 2021-04-14 21:58:25.217: INFO @decay_lr : LR updated to `0.00043296663` 2021-04-14 21:58:25.218: INFO @log_profile : T train: 130.995168 2021-04-14 21:58:25.218: INFO @log_profile : T read data: 2.275724 2021-04-14 21:58:25.218: INFO @log_profile : T hooks: 84.776690 2021-04-14 21:58:25.218: INFO @main_loop : Epoch 167 done 2021-04-14 21:58:25.218: INFO @main_loop : Training epoch 168 2021-04-14 22:00:37.447: INFO @log_variables: train loss nanmean: 0.022143 2021-04-14 22:00:37.447: INFO @log_variables: train positive_distance nanmean: 0.555921 2021-04-14 22:00:37.447: INFO @log_variables: train negative_distance nanmean: 1.419550 2021-04-14 22:00:37.447: INFO @log_variables: train distance_margin nanmean: 0.863629 2021-04-14 22:00:37.447: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:02:01.532: INFO @metrics_hook: train matching accuracy: 0.9906864030526896, N=110591, DB_SIZE=110592, N_IDS=11546 2021-04-14 22:02:01.715: INFO @decay_lr : LR updated to `0.0004308018` 2021-04-14 22:02:01.716: INFO @log_profile : T train: 129.602375 2021-04-14 22:02:01.716: INFO @log_profile : T read data: 2.264682 2021-04-14 22:02:01.716: INFO @log_profile : T hooks: 84.578495 2021-04-14 22:02:01.716: INFO @main_loop : Epoch 168 done 2021-04-14 22:02:01.716: INFO @main_loop : Training epoch 169 2021-04-14 22:04:15.476: INFO @log_variables: train loss nanmean: 0.022947 2021-04-14 22:04:15.476: INFO @log_variables: train positive_distance nanmean: 0.557655 2021-04-14 22:04:15.476: INFO @log_variables: train negative_distance nanmean: 1.419560 2021-04-14 22:04:15.476: INFO @log_variables: train distance_margin nanmean: 0.861905 2021-04-14 22:04:15.476: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:05:39.386: INFO @metrics_hook: train matching accuracy: 0.9909210109870236, N=110585, DB_SIZE=110592, N_IDS=11545 2021-04-14 22:05:39.566: INFO @decay_lr : LR updated to `0.0004286478` 2021-04-14 22:05:39.568: INFO @log_profile : T train: 131.360972 2021-04-14 22:05:39.568: INFO @log_profile : T read data: 2.073226 2021-04-14 22:05:39.568: INFO @log_profile : T hooks: 84.366579 2021-04-14 22:05:39.568: INFO @main_loop : Epoch 169 done 2021-04-14 22:05:39.568: INFO @main_loop : Training epoch 170 2021-04-14 22:07:50.846: INFO @log_variables: train loss nanmean: 0.022434 2021-04-14 22:07:50.846: INFO @log_variables: train positive_distance nanmean: 0.556403 2021-04-14 22:07:50.847: INFO @log_variables: train negative_distance nanmean: 1.419524 2021-04-14 22:07:50.847: INFO @log_variables: train distance_margin nanmean: 0.863121 2021-04-14 22:07:50.847: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:09:14.843: INFO @metrics_hook: train matching accuracy: 0.990278970927341, N=110585, DB_SIZE=110592, N_IDS=11551 2021-04-14 22:09:15.015: INFO @decay_lr : LR updated to `0.00042650456` 2021-04-14 22:09:15.016: INFO @log_profile : T train: 130.091955 2021-04-14 22:09:15.016: INFO @log_profile : T read data: 0.849370 2021-04-14 22:09:15.017: INFO @log_profile : T hooks: 84.454817 2021-04-14 22:09:15.017: INFO @main_loop : Epoch 170 done 2021-04-14 22:09:15.017: INFO @main_loop : Training epoch 171 2021-04-14 22:11:26.238: INFO @log_variables: train loss nanmean: 0.022943 2021-04-14 22:11:26.238: INFO @log_variables: train positive_distance nanmean: 0.556869 2021-04-14 22:11:26.238: INFO @log_variables: train negative_distance nanmean: 1.419534 2021-04-14 22:11:26.238: INFO @log_variables: train distance_margin nanmean: 0.862665 2021-04-14 22:11:26.239: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:12:50.441: INFO @metrics_hook: train matching accuracy: 0.9895191846848069, N=110583, DB_SIZE=110592, N_IDS=11536 2021-04-14 22:12:50.617: INFO @decay_lr : LR updated to `0.00042437203` 2021-04-14 22:12:50.618: INFO @log_profile : T train: 128.604742 2021-04-14 22:12:50.619: INFO @log_profile : T read data: 2.286273 2021-04-14 22:12:50.619: INFO @log_profile : T hooks: 84.659607 2021-04-14 22:12:50.619: INFO @main_loop : Epoch 171 done 2021-04-14 22:12:50.619: INFO @main_loop : Training epoch 172 2021-04-14 22:15:03.006: INFO @log_variables: train loss nanmean: 0.022077 2021-04-14 22:15:03.006: INFO @log_variables: train positive_distance nanmean: 0.555860 2021-04-14 22:15:03.006: INFO @log_variables: train negative_distance nanmean: 1.419542 2021-04-14 22:15:03.006: INFO @log_variables: train distance_margin nanmean: 0.863682 2021-04-14 22:15:03.006: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:16:27.520: INFO @metrics_hook: train matching accuracy: 0.9911652469571194, N=110586, DB_SIZE=110592, N_IDS=11540 2021-04-14 22:16:27.696: INFO @decay_lr : LR updated to `0.00042225016` 2021-04-14 22:16:27.697: INFO @log_profile : T train: 129.784812 2021-04-14 22:16:27.697: INFO @log_profile : T read data: 2.239798 2021-04-14 22:16:27.697: INFO @log_profile : T hooks: 85.001566 2021-04-14 22:16:27.698: INFO @main_loop : Epoch 172 done 2021-04-14 22:16:27.698: INFO @main_loop : Training epoch 173 2021-04-14 22:18:40.295: INFO @log_variables: train loss nanmean: 0.022382 2021-04-14 22:18:40.295: INFO @log_variables: train positive_distance nanmean: 0.556231 2021-04-14 22:18:40.295: INFO @log_variables: train negative_distance nanmean: 1.419501 2021-04-14 22:18:40.295: INFO @log_variables: train distance_margin nanmean: 0.863270 2021-04-14 22:18:40.295: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:20:04.465: INFO @metrics_hook: train matching accuracy: 0.9911653268467361, N=110587, DB_SIZE=110592, N_IDS=11553 2021-04-14 22:20:04.650: INFO @decay_lr : LR updated to `0.0004201389` 2021-04-14 22:20:04.652: INFO @log_profile : T train: 129.850413 2021-04-14 22:20:04.652: INFO @log_profile : T read data: 2.389923 2021-04-14 22:20:04.652: INFO @log_profile : T hooks: 84.661746 2021-04-14 22:20:04.652: INFO @main_loop : Epoch 173 done 2021-04-14 22:20:04.652: INFO @main_loop : Training epoch 174 2021-04-14 22:22:19.644: INFO @log_variables: train loss nanmean: 0.021554 2021-04-14 22:22:19.644: INFO @log_variables: train positive_distance nanmean: 0.555897 2021-04-14 22:22:19.644: INFO @log_variables: train negative_distance nanmean: 1.419527 2021-04-14 22:22:19.644: INFO @log_variables: train distance_margin nanmean: 0.863629 2021-04-14 22:22:19.644: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:23:43.278: INFO @metrics_hook: train matching accuracy: 0.9909571822579916, N=110585, DB_SIZE=110592, N_IDS=11554 2021-04-14 22:23:43.467: INFO @decay_lr : LR updated to `0.00041803822` 2021-04-14 22:23:43.469: INFO @log_profile : T train: 132.172765 2021-04-14 22:23:43.469: INFO @log_profile : T read data: 2.463413 2021-04-14 22:23:43.469: INFO @log_profile : T hooks: 84.128910 2021-04-14 22:23:43.469: INFO @main_loop : Epoch 174 done 2021-04-14 22:23:43.469: INFO @main_loop : Training epoch 175 2021-04-14 22:25:56.361: INFO @log_variables: train loss nanmean: 0.021579 2021-04-14 22:25:56.361: INFO @log_variables: train positive_distance nanmean: 0.555905 2021-04-14 22:25:56.361: INFO @log_variables: train negative_distance nanmean: 1.419531 2021-04-14 22:25:56.361: INFO @log_variables: train distance_margin nanmean: 0.863626 2021-04-14 22:25:56.361: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:27:19.780: INFO @metrics_hook: train matching accuracy: 0.9905591275331651, N=110583, DB_SIZE=110592, N_IDS=11550 2021-04-14 22:27:19.955: INFO @decay_lr : LR updated to `0.00041594804` 2021-04-14 22:27:19.957: INFO @log_profile : T train: 130.158285 2021-04-14 22:27:19.957: INFO @log_profile : T read data: 2.386192 2021-04-14 22:27:19.957: INFO @log_profile : T hooks: 83.891937 2021-04-14 22:27:19.957: INFO @main_loop : Epoch 175 done 2021-04-14 22:27:19.957: INFO @main_loop : Training epoch 176 2021-04-14 22:29:33.011: INFO @log_variables: train loss nanmean: 0.022159 2021-04-14 22:29:33.011: INFO @log_variables: train positive_distance nanmean: 0.556088 2021-04-14 22:29:33.011: INFO @log_variables: train negative_distance nanmean: 1.419500 2021-04-14 22:29:33.011: INFO @log_variables: train distance_margin nanmean: 0.863413 2021-04-14 22:29:33.011: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:30:57.991: INFO @metrics_hook: train matching accuracy: 0.9905411271069956, N=110584, DB_SIZE=110592, N_IDS=11566 2021-04-14 22:30:58.174: INFO @decay_lr : LR updated to `0.0004138683` 2021-04-14 22:30:58.176: INFO @log_profile : T train: 130.301777 2021-04-14 22:30:58.176: INFO @log_profile : T read data: 2.400353 2021-04-14 22:30:58.176: INFO @log_profile : T hooks: 85.464405 2021-04-14 22:30:58.176: INFO @main_loop : Epoch 176 done 2021-04-14 22:30:58.176: INFO @main_loop : Training epoch 177 2021-04-14 22:33:08.936: INFO @log_variables: train loss nanmean: 0.022212 2021-04-14 22:33:08.937: INFO @log_variables: train positive_distance nanmean: 0.556112 2021-04-14 22:33:08.937: INFO @log_variables: train negative_distance nanmean: 1.419553 2021-04-14 22:33:08.937: INFO @log_variables: train distance_margin nanmean: 0.863441 2021-04-14 22:33:08.937: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:34:33.674: INFO @metrics_hook: train matching accuracy: 0.9901792335099745, N=110582, DB_SIZE=110592, N_IDS=11559 2021-04-14 22:34:33.850: INFO @decay_lr : LR updated to `0.00041179897` 2021-04-14 22:34:33.851: INFO @log_profile : T train: 129.567935 2021-04-14 22:34:33.851: INFO @log_profile : T read data: 0.837844 2021-04-14 22:34:33.851: INFO @log_profile : T hooks: 85.217730 2021-04-14 22:34:33.851: INFO @main_loop : Epoch 177 done 2021-04-14 22:34:33.851: INFO @main_loop : Training epoch 178 2021-04-14 22:36:45.982: INFO @log_variables: train loss nanmean: 0.021437 2021-04-14 22:36:45.983: INFO @log_variables: train positive_distance nanmean: 0.556582 2021-04-14 22:36:45.983: INFO @log_variables: train negative_distance nanmean: 1.419513 2021-04-14 22:36:45.983: INFO @log_variables: train distance_margin nanmean: 0.862931 2021-04-14 22:36:45.983: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:38:09.912: INFO @metrics_hook: train matching accuracy: 0.9910659384381669, N=110588, DB_SIZE=110592, N_IDS=11543 2021-04-14 22:38:10.091: INFO @decay_lr : LR updated to `0.00040973997` 2021-04-14 22:38:10.092: INFO @log_profile : T train: 129.522653 2021-04-14 22:38:10.092: INFO @log_profile : T read data: 2.274613 2021-04-14 22:38:10.092: INFO @log_profile : T hooks: 84.392648 2021-04-14 22:38:10.092: INFO @main_loop : Epoch 178 done 2021-04-14 22:38:10.092: INFO @main_loop : Training epoch 179 2021-04-14 22:40:24.855: INFO @log_variables: train loss nanmean: 0.021595 2021-04-14 22:40:24.855: INFO @log_variables: train positive_distance nanmean: 0.554749 2021-04-14 22:40:24.856: INFO @log_variables: train negative_distance nanmean: 1.419541 2021-04-14 22:40:24.856: INFO @log_variables: train distance_margin nanmean: 0.864793 2021-04-14 22:40:24.856: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:41:49.488: INFO @metrics_hook: train matching accuracy: 0.9909212572792708, N=110588, DB_SIZE=110592, N_IDS=11543 2021-04-14 22:41:49.673: INFO @decay_lr : LR updated to `0.00040769126` 2021-04-14 22:41:49.674: INFO @log_profile : T train: 132.368169 2021-04-14 22:41:49.674: INFO @log_profile : T read data: 2.070357 2021-04-14 22:41:49.674: INFO @log_profile : T hooks: 85.092149 2021-04-14 22:41:49.674: INFO @main_loop : Epoch 179 done 2021-04-14 22:41:49.674: INFO @main_loop : Training epoch 180 2021-04-14 22:44:01.474: INFO @log_variables: train loss nanmean: 0.022061 2021-04-14 22:44:01.474: INFO @log_variables: train positive_distance nanmean: 0.557088 2021-04-14 22:44:01.474: INFO @log_variables: train negative_distance nanmean: 1.419516 2021-04-14 22:44:01.474: INFO @log_variables: train distance_margin nanmean: 0.862428 2021-04-14 22:44:01.474: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:45:25.881: INFO @metrics_hook: train matching accuracy: 0.9905689380787037, N=110592, DB_SIZE=110592, N_IDS=11560 2021-04-14 22:45:26.053: INFO @decay_lr : LR updated to `0.0004056528` 2021-04-14 22:45:26.054: INFO @log_profile : T train: 129.224277 2021-04-14 22:45:26.054: INFO @log_profile : T read data: 2.243732 2021-04-14 22:45:26.054: INFO @log_profile : T hooks: 84.860874 2021-04-14 22:45:26.054: INFO @main_loop : Epoch 180 done 2021-04-14 22:45:26.054: INFO @main_loop : Training epoch 181 2021-04-14 22:47:38.341: INFO @log_variables: train loss nanmean: 0.021278 2021-04-14 22:47:38.341: INFO @log_variables: train positive_distance nanmean: 0.555518 2021-04-14 22:47:38.341: INFO @log_variables: train negative_distance nanmean: 1.419528 2021-04-14 22:47:38.341: INFO @log_variables: train distance_margin nanmean: 0.864010 2021-04-14 22:47:38.341: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:49:03.352: INFO @metrics_hook: train matching accuracy: 0.9908943765767558, N=110591, DB_SIZE=110592, N_IDS=11535 2021-04-14 22:49:03.533: INFO @decay_lr : LR updated to `0.00040362452` 2021-04-14 22:49:03.534: INFO @log_profile : T train: 129.661792 2021-04-14 22:49:03.534: INFO @log_profile : T read data: 2.290531 2021-04-14 22:49:03.535: INFO @log_profile : T hooks: 85.475790 2021-04-14 22:49:03.535: INFO @main_loop : Epoch 181 done 2021-04-14 22:49:03.535: INFO @main_loop : Training epoch 182 2021-04-14 22:51:16.086: INFO @log_variables: train loss nanmean: 0.020937 2021-04-14 22:51:16.086: INFO @log_variables: train positive_distance nanmean: 0.555386 2021-04-14 22:51:16.086: INFO @log_variables: train negative_distance nanmean: 1.419528 2021-04-14 22:51:16.086: INFO @log_variables: train distance_margin nanmean: 0.864141 2021-04-14 22:51:16.086: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:52:40.467: INFO @metrics_hook: train matching accuracy: 0.9909753495017453, N=110586, DB_SIZE=110592, N_IDS=11593 2021-04-14 22:52:40.652: INFO @decay_lr : LR updated to `0.0004016064` 2021-04-14 22:52:40.654: INFO @log_profile : T train: 129.985488 2021-04-14 22:52:40.654: INFO @log_profile : T read data: 2.215305 2021-04-14 22:52:40.654: INFO @log_profile : T hooks: 84.866961 2021-04-14 22:52:40.654: INFO @main_loop : Epoch 182 done 2021-04-14 22:52:40.655: INFO @main_loop : Training epoch 183 2021-04-14 22:54:54.116: INFO @log_variables: train loss nanmean: 0.020902 2021-04-14 22:54:54.116: INFO @log_variables: train positive_distance nanmean: 0.555391 2021-04-14 22:54:54.116: INFO @log_variables: train negative_distance nanmean: 1.419534 2021-04-14 22:54:54.116: INFO @log_variables: train distance_margin nanmean: 0.864143 2021-04-14 22:54:54.116: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:56:19.137: INFO @metrics_hook: train matching accuracy: 0.9908397084621923, N=110586, DB_SIZE=110592, N_IDS=11555 2021-04-14 22:56:19.347: INFO @decay_lr : LR updated to `0.0003995984` 2021-04-14 22:56:19.348: INFO @log_profile : T train: 130.763287 2021-04-14 22:56:19.348: INFO @log_profile : T read data: 2.333428 2021-04-14 22:56:19.348: INFO @log_profile : T hooks: 85.542297 2021-04-14 22:56:19.348: INFO @main_loop : Epoch 183 done 2021-04-14 22:56:19.348: INFO @main_loop : Training epoch 184 2021-04-14 22:58:29.890: INFO @log_variables: train loss nanmean: 0.021311 2021-04-14 22:58:29.890: INFO @log_variables: train positive_distance nanmean: 0.555299 2021-04-14 22:58:29.890: INFO @log_variables: train negative_distance nanmean: 1.419564 2021-04-14 22:58:29.890: INFO @log_variables: train distance_margin nanmean: 0.864265 2021-04-14 22:58:29.890: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 22:59:53.692: INFO @metrics_hook: train matching accuracy: 0.9915358460536786, N=110584, DB_SIZE=110592, N_IDS=11555 2021-04-14 22:59:53.890: INFO @decay_lr : LR updated to `0.0003976004` 2021-04-14 22:59:53.891: INFO @log_profile : T train: 129.331609 2021-04-14 22:59:53.891: INFO @log_profile : T read data: 0.847053 2021-04-14 22:59:53.891: INFO @log_profile : T hooks: 84.312150 2021-04-14 22:59:53.891: INFO @main_loop : Epoch 184 done 2021-04-14 22:59:53.892: INFO @main_loop : Training epoch 185 2021-04-14 23:02:08.037: INFO @log_variables: train loss nanmean: 0.021349 2021-04-14 23:02:08.037: INFO @log_variables: train positive_distance nanmean: 0.554787 2021-04-14 23:02:08.037: INFO @log_variables: train negative_distance nanmean: 1.419548 2021-04-14 23:02:08.037: INFO @log_variables: train distance_margin nanmean: 0.864761 2021-04-14 23:02:08.037: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:03:32.310: INFO @metrics_hook: train matching accuracy: 0.9911651670660577, N=110585, DB_SIZE=110592, N_IDS=11553 2021-04-14 23:03:32.480: INFO @decay_lr : LR updated to `0.0003956124` 2021-04-14 23:03:32.482: INFO @log_profile : T train: 131.665634 2021-04-14 23:03:32.482: INFO @log_profile : T read data: 2.147756 2021-04-14 23:03:32.482: INFO @log_profile : T hooks: 84.724781 2021-04-14 23:03:32.482: INFO @main_loop : Epoch 185 done 2021-04-14 23:03:32.482: INFO @main_loop : Training epoch 186 2021-04-14 23:05:43.841: INFO @log_variables: train loss nanmean: 0.020775 2021-04-14 23:05:43.841: INFO @log_variables: train positive_distance nanmean: 0.553910 2021-04-14 23:05:43.841: INFO @log_variables: train negative_distance nanmean: 1.419541 2021-04-14 23:05:43.841: INFO @log_variables: train distance_margin nanmean: 0.865631 2021-04-14 23:05:43.841: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:07:08.130: INFO @metrics_hook: train matching accuracy: 0.9896915606434636, N=110589, DB_SIZE=110592, N_IDS=11544 2021-04-14 23:07:08.318: INFO @decay_lr : LR updated to `0.00039363434` 2021-04-14 23:07:08.319: INFO @log_profile : T train: 128.740488 2021-04-14 23:07:08.319: INFO @log_profile : T read data: 2.265624 2021-04-14 23:07:08.319: INFO @log_profile : T hooks: 84.780690 2021-04-14 23:07:08.319: INFO @main_loop : Epoch 186 done 2021-04-14 23:07:08.319: INFO @main_loop : Training epoch 187 2021-04-14 23:09:21.012: INFO @log_variables: train loss nanmean: 0.020636 2021-04-14 23:09:21.012: INFO @log_variables: train positive_distance nanmean: 0.554667 2021-04-14 23:09:21.013: INFO @log_variables: train negative_distance nanmean: 1.419552 2021-04-14 23:09:21.013: INFO @log_variables: train distance_margin nanmean: 0.864885 2021-04-14 23:09:21.013: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:10:44.668: INFO @metrics_hook: train matching accuracy: 0.9906317255348779, N=110586, DB_SIZE=110592, N_IDS=11544 2021-04-14 23:10:44.865: INFO @decay_lr : LR updated to `0.00039166617` 2021-04-14 23:10:44.867: INFO @log_profile : T train: 130.017736 2021-04-14 23:10:44.867: INFO @log_profile : T read data: 2.312635 2021-04-14 23:10:44.867: INFO @log_profile : T hooks: 84.165989 2021-04-14 23:10:44.867: INFO @main_loop : Epoch 187 done 2021-04-14 23:10:44.867: INFO @main_loop : Training epoch 188 2021-04-14 23:12:55.951: INFO @log_variables: train loss nanmean: 0.021395 2021-04-14 23:12:55.951: INFO @log_variables: train positive_distance nanmean: 0.555677 2021-04-14 23:12:55.951: INFO @log_variables: train negative_distance nanmean: 1.419539 2021-04-14 23:12:55.951: INFO @log_variables: train distance_margin nanmean: 0.863863 2021-04-14 23:12:55.951: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:14:19.869: INFO @metrics_hook: train matching accuracy: 0.9910472866043896, N=110581, DB_SIZE=110592, N_IDS=11546 2021-04-14 23:14:20.082: INFO @decay_lr : LR updated to `0.00038970783` 2021-04-14 23:14:20.084: INFO @log_profile : T train: 128.394933 2021-04-14 23:14:20.084: INFO @log_profile : T read data: 2.325914 2021-04-14 23:14:20.084: INFO @log_profile : T hooks: 84.444143 2021-04-14 23:14:20.084: INFO @main_loop : Epoch 188 done 2021-04-14 23:14:20.084: INFO @main_loop : Training epoch 189 2021-04-14 23:16:31.293: INFO @log_variables: train loss nanmean: 0.021665 2021-04-14 23:16:31.293: INFO @log_variables: train positive_distance nanmean: 0.556665 2021-04-14 23:16:31.293: INFO @log_variables: train negative_distance nanmean: 1.419532 2021-04-14 23:16:31.293: INFO @log_variables: train distance_margin nanmean: 0.862867 2021-04-14 23:16:31.293: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:17:55.795: INFO @metrics_hook: train matching accuracy: 0.9900890717547588, N=110585, DB_SIZE=110592, N_IDS=11547 2021-04-14 23:17:55.965: INFO @decay_lr : LR updated to `0.0003877593` 2021-04-14 23:17:55.967: INFO @log_profile : T train: 128.424825 2021-04-14 23:17:55.967: INFO @log_profile : T read data: 2.423349 2021-04-14 23:17:55.967: INFO @log_profile : T hooks: 84.983539 2021-04-14 23:17:55.968: INFO @main_loop : Epoch 189 done 2021-04-14 23:17:55.968: INFO @main_loop : Training epoch 190 2021-04-14 23:20:08.371: INFO @log_variables: train loss nanmean: 0.020409 2021-04-14 23:20:08.373: INFO @log_variables: train positive_distance nanmean: 0.554856 2021-04-14 23:20:08.373: INFO @log_variables: train negative_distance nanmean: 1.419538 2021-04-14 23:20:08.373: INFO @log_variables: train distance_margin nanmean: 0.864682 2021-04-14 23:20:08.373: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:21:33.031: INFO @metrics_hook: train matching accuracy: 0.9907584909755127, N=110588, DB_SIZE=110592, N_IDS=11551 2021-04-14 23:21:33.217: INFO @decay_lr : LR updated to `0.0003858205` 2021-04-14 23:21:33.218: INFO @log_profile : T train: 129.877205 2021-04-14 23:21:33.219: INFO @log_profile : T read data: 2.178373 2021-04-14 23:21:33.219: INFO @log_profile : T hooks: 85.143698 2021-04-14 23:21:33.219: INFO @main_loop : Epoch 190 done 2021-04-14 23:21:33.219: INFO @main_loop : Training epoch 191 2021-04-14 23:23:46.214: INFO @log_variables: train loss nanmean: 0.020043 2021-04-14 23:23:46.214: INFO @log_variables: train positive_distance nanmean: 0.554885 2021-04-14 23:23:46.214: INFO @log_variables: train negative_distance nanmean: 1.419536 2021-04-14 23:23:46.214: INFO @log_variables: train distance_margin nanmean: 0.864651 2021-04-14 23:23:46.214: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:25:09.287: INFO @metrics_hook: train matching accuracy: 0.9899809203446998, N=110589, DB_SIZE=110592, N_IDS=11557 2021-04-14 23:25:09.478: INFO @decay_lr : LR updated to `0.0003838914` 2021-04-14 23:25:09.480: INFO @log_profile : T train: 130.439014 2021-04-14 23:25:09.480: INFO @log_profile : T read data: 2.225242 2021-04-14 23:25:09.480: INFO @log_profile : T hooks: 83.545426 2021-04-14 23:25:09.480: INFO @main_loop : Epoch 191 done 2021-04-14 23:25:09.480: INFO @main_loop : Training epoch 192 2021-04-14 23:27:22.562: INFO @log_variables: train loss nanmean: 0.021098 2021-04-14 23:27:22.563: INFO @log_variables: train positive_distance nanmean: 0.554851 2021-04-14 23:27:22.563: INFO @log_variables: train negative_distance nanmean: 1.419562 2021-04-14 23:27:22.563: INFO @log_variables: train distance_margin nanmean: 0.864711 2021-04-14 23:27:22.563: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:28:47.432: INFO @metrics_hook: train matching accuracy: 0.9895559232835092, N=110589, DB_SIZE=110592, N_IDS=11562 2021-04-14 23:28:47.620: INFO @decay_lr : LR updated to `0.00038197194` 2021-04-14 23:28:47.621: INFO @log_profile : T train: 131.898592 2021-04-14 23:28:47.622: INFO @log_profile : T read data: 0.841691 2021-04-14 23:28:47.622: INFO @log_profile : T hooks: 85.349612 2021-04-14 23:28:47.622: INFO @main_loop : Epoch 192 done 2021-04-14 23:28:47.622: INFO @main_loop : Training epoch 193 2021-04-14 23:30:59.904: INFO @log_variables: train loss nanmean: 0.021196 2021-04-14 23:30:59.904: INFO @log_variables: train positive_distance nanmean: 0.555709 2021-04-14 23:30:59.904: INFO @log_variables: train negative_distance nanmean: 1.419540 2021-04-14 23:30:59.904: INFO @log_variables: train distance_margin nanmean: 0.863830 2021-04-14 23:30:59.904: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:32:24.509: INFO @metrics_hook: train matching accuracy: 0.9906863188353378, N=110590, DB_SIZE=110592, N_IDS=11524 2021-04-14 23:32:24.712: INFO @decay_lr : LR updated to `0.0003800621` 2021-04-14 23:32:24.713: INFO @log_profile : T train: 129.540705 2021-04-14 23:32:24.713: INFO @log_profile : T read data: 2.399893 2021-04-14 23:32:24.713: INFO @log_profile : T hooks: 85.099345 2021-04-14 23:32:24.714: INFO @main_loop : Epoch 193 done 2021-04-14 23:32:24.714: INFO @main_loop : Training epoch 194 2021-04-14 23:34:36.760: INFO @log_variables: train loss nanmean: 0.020264 2021-04-14 23:34:36.761: INFO @log_variables: train positive_distance nanmean: 0.554286 2021-04-14 23:34:36.761: INFO @log_variables: train negative_distance nanmean: 1.419552 2021-04-14 23:34:36.761: INFO @log_variables: train distance_margin nanmean: 0.865266 2021-04-14 23:34:36.761: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:35:59.649: INFO @metrics_hook: train matching accuracy: 0.9906410221631446, N=110589, DB_SIZE=110592, N_IDS=11563 2021-04-14 23:35:59.822: INFO @decay_lr : LR updated to `0.00037816179` 2021-04-14 23:35:59.823: INFO @log_profile : T train: 129.570909 2021-04-14 23:35:59.824: INFO @log_profile : T read data: 2.117975 2021-04-14 23:35:59.824: INFO @log_profile : T hooks: 83.368948 2021-04-14 23:35:59.824: INFO @main_loop : Epoch 194 done 2021-04-14 23:35:59.824: INFO @main_loop : Training epoch 195 2021-04-14 23:38:11.947: INFO @log_variables: train loss nanmean: 0.021189 2021-04-14 23:38:11.947: INFO @log_variables: train positive_distance nanmean: 0.554072 2021-04-14 23:38:11.948: INFO @log_variables: train negative_distance nanmean: 1.419557 2021-04-14 23:38:11.948: INFO @log_variables: train distance_margin nanmean: 0.865485 2021-04-14 23:38:11.948: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:39:36.472: INFO @metrics_hook: train matching accuracy: 0.9910565723793677, N=110584, DB_SIZE=110592, N_IDS=11554 2021-04-14 23:39:36.650: INFO @decay_lr : LR updated to `0.00037627097` 2021-04-14 23:39:36.651: INFO @log_profile : T train: 129.596574 2021-04-14 23:39:36.651: INFO @log_profile : T read data: 2.175608 2021-04-14 23:39:36.651: INFO @log_profile : T hooks: 85.003694 2021-04-14 23:39:36.651: INFO @main_loop : Epoch 195 done 2021-04-14 23:39:36.651: INFO @main_loop : Training epoch 196 2021-04-14 23:41:49.171: INFO @log_variables: train loss nanmean: 0.021492 2021-04-14 23:41:49.172: INFO @log_variables: train positive_distance nanmean: 0.555487 2021-04-14 23:41:49.172: INFO @log_variables: train negative_distance nanmean: 1.419521 2021-04-14 23:41:49.172: INFO @log_variables: train distance_margin nanmean: 0.864034 2021-04-14 23:41:49.172: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:43:13.654: INFO @metrics_hook: train matching accuracy: 0.9905954695483112, N=110585, DB_SIZE=110592, N_IDS=11565 2021-04-14 23:43:13.833: INFO @decay_lr : LR updated to `0.0003743896` 2021-04-14 23:43:13.834: INFO @log_profile : T train: 130.030693 2021-04-14 23:43:13.834: INFO @log_profile : T read data: 2.150415 2021-04-14 23:43:13.834: INFO @log_profile : T hooks: 84.950492 2021-04-14 23:43:13.834: INFO @main_loop : Epoch 196 done 2021-04-14 23:43:13.834: INFO @main_loop : Training epoch 197 2021-04-14 23:45:27.869: INFO @log_variables: train loss nanmean: 0.020772 2021-04-14 23:45:27.869: INFO @log_variables: train positive_distance nanmean: 0.554884 2021-04-14 23:45:27.869: INFO @log_variables: train negative_distance nanmean: 1.419561 2021-04-14 23:45:27.869: INFO @log_variables: train distance_margin nanmean: 0.864677 2021-04-14 23:45:27.869: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:46:50.568: INFO @metrics_hook: train matching accuracy: 0.9908304169718672, N=110583, DB_SIZE=110592, N_IDS=11530 2021-04-14 23:46:50.752: INFO @decay_lr : LR updated to `0.00037251765` 2021-04-14 23:46:50.753: INFO @log_profile : T train: 131.524417 2021-04-14 23:46:50.754: INFO @log_profile : T read data: 2.153370 2021-04-14 23:46:50.754: INFO @log_profile : T hooks: 83.190374 2021-04-14 23:46:50.754: INFO @main_loop : Epoch 197 done 2021-04-14 23:46:50.754: INFO @main_loop : Training epoch 198 2021-04-14 23:49:04.555: INFO @log_variables: train loss nanmean: 0.020065 2021-04-14 23:49:04.555: INFO @log_variables: train positive_distance nanmean: 0.554268 2021-04-14 23:49:04.555: INFO @log_variables: train negative_distance nanmean: 1.419550 2021-04-14 23:49:04.555: INFO @log_variables: train distance_margin nanmean: 0.865282 2021-04-14 23:49:04.555: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:50:28.204: INFO @metrics_hook: train matching accuracy: 0.9908577112628295, N=110585, DB_SIZE=110592, N_IDS=11541 2021-04-14 23:50:28.381: INFO @decay_lr : LR updated to `0.00037065506` 2021-04-14 23:50:28.383: INFO @log_profile : T train: 131.232270 2021-04-14 23:50:28.383: INFO @log_profile : T read data: 2.221584 2021-04-14 23:50:28.383: INFO @log_profile : T hooks: 84.123576 2021-04-14 23:50:28.383: INFO @main_loop : Epoch 198 done 2021-04-14 23:50:28.383: INFO @main_loop : Training epoch 199 2021-04-14 23:52:39.253: INFO @log_variables: train loss nanmean: 0.019712 2021-04-14 23:52:39.253: INFO @log_variables: train positive_distance nanmean: 0.555349 2021-04-14 23:52:39.253: INFO @log_variables: train negative_distance nanmean: 1.419568 2021-04-14 23:52:39.253: INFO @log_variables: train distance_margin nanmean: 0.864219 2021-04-14 23:52:39.254: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:54:02.554: INFO @metrics_hook: train matching accuracy: 0.9908394599531574, N=110583, DB_SIZE=110592, N_IDS=11578 2021-04-14 23:54:02.728: INFO @decay_lr : LR updated to `0.0003688018` 2021-04-14 23:54:02.729: INFO @log_profile : T train: 129.673247 2021-04-14 23:54:02.729: INFO @log_profile : T read data: 0.842676 2021-04-14 23:54:02.729: INFO @log_profile : T hooks: 83.777864 2021-04-14 23:54:02.729: INFO @main_loop : Epoch 199 done 2021-04-14 23:54:02.729: INFO @main_loop : Training epoch 200 2021-04-14 23:56:14.631: INFO @log_variables: train loss nanmean: 0.019574 2021-04-14 23:56:14.632: INFO @log_variables: train positive_distance nanmean: 0.554047 2021-04-14 23:56:14.632: INFO @log_variables: train negative_distance nanmean: 1.419543 2021-04-14 23:56:14.632: INFO @log_variables: train distance_margin nanmean: 0.865496 2021-04-14 23:56:14.632: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f 2021-04-14 23:57:39.428: INFO @metrics_hook: train matching accuracy: 0.9907946612652367, N=110588, DB_SIZE=110592, N_IDS=11540 2021-04-14 23:57:39.609: INFO @decay_lr : LR updated to `0.00036695777` 2021-04-14 23:57:39.880: INFO @save : Model saved to: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f/model_200.ckpt 2021-04-14 23:57:39.901: INFO @stop_after : EpochStopperHook triggered 2021-04-14 23:57:39.902: INFO @log_profile : T train: 129.287325 2021-04-14 23:57:39.902: INFO @log_profile : T read data: 2.254477 2021-04-14 23:57:39.903: INFO @log_profile : T hooks: 85.579514 2021-04-14 23:57:39.903: INFO @main_loop : Training terminated: Training terminated after epoch 200 2021-04-14 23:57:39.903: INFO @log_dir : Output dir: log-nightly/embedder/2021-04-14-nightly-embedder--Fa3f