'Performance Issue with Tensoflow Hub and Universal Sentence Encoder
I am training Top2vec models using universal-sentence-encoder_4. On larger data sets (e.g. > 500k rows), the code is dismally slow.
Here's the notebook: https://github.com/ddangelov/Top2Vec/blob/master/notebooks/CORD-19_top2vec.ipynb
top2vec_trained = Top2Vec(documents=papers_filtered_df.text.tolist(), split_documents=True, embedding_batch_size=64, embedding_model="universal-sentence-encoder", use_embedding_model_tokenizer=True, embedding_model_path="/Users/davidlaxer/Downloads/universal-sentence-encoder_4", workers=8)
Here's the code snippet I profiled from Top2vec.py:
import cProfile, pstats, io
from pstats import SortKey
pr = cProfile.Profile()
pr.enable()
for ind in range(0, batches):
document_vectors.append(self.embed(train_corpus[current:current + batch_size]))
current += batch_size
if extra > 0:
document_vectors.append(self.embed(train_corpus[current:current + extra]))
document_vectors = self._l2_normalize(np.array(np.vstack(document_vectors)))
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
print(s.getvalue())
Here's the profiler output:
3638257 function calls (3568425 primitive calls) in 154.779 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
6983 0.015 0.000 152.706 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:743(_call_attribute)
6983 0.019 0.000 152.692 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:138(error_handler)
6983 0.129 0.000 152.643 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:887(__call__)
6983 0.086 0.000 152.281 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:934(_call)
6983 0.068 0.000 152.188 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2951(__call__)
6983 0.081 0.000 149.721 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:1782(_call_flat)
6983 0.205 0.000 149.416 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:454(call)
6983 0.034 0.000 149.041 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/execute.py:29(quick_execute)
6983 148.998 0.021 148.998 0.021 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_Execute}
6983 0.048 0.000 1.928 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:3219(_maybe_define_function)
6/4 0.148 0.025 1.925 0.481 {built-in method numpy.core._multiarray_umath.implement_array_function}
1 0.000 0.000 1.882 1.882 <__array_function__ internals>:2(vstack)
1 0.000 0.000 1.877 1.877 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:223(vstack)
6987 1.833 0.000 1.833 0.000 {built-in method numpy.array}
1 0.000 0.000 1.736 1.736 <__array_function__ internals>:2(atleast_2d)
1 0.015 0.015 1.734 1.734 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:82(atleast_2d)
6984 0.004 0.000 1.718 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/_asarray.py:110(asanyarray)
6983 0.046 0.000 1.381 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2648(canonicalize_function_inputs)
6983 0.080 0.000 1.311 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2817(_convert_inputs_to_signature)
6983 0.009 0.000 0.652 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:178(wrapped)
6983 0.076 0.000 0.643 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1636(convert_to_tensor)
6983 0.015 0.000 0.472 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2051(captured_inputs)
6983 0.005 0.000 0.405 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:340(_constant_tensor_conversion_function)
6983 0.007 0.000 0.400 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:170(constant)
6983 0.016 0.000 0.393 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:271(_constant_impl)
6983 0.009 0.000 0.367 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:302(_constant_eager_impl)
6983 0.343 0.000 0.359 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:74(convert_to_eager_tensor)
27932 0.024 0.000 0.322 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:353(flatten)
27932 0.298 0.000 0.298 0.000 {built-in method tensorflow.python.util._pywrap_utils.Flatten}
6983 0.027 0.000 0.275 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:250(make_cache_key)
6983 0.169 0.000 0.232 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2058(<listcomp>)
6983 0.014 0.000 0.219 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1157(flatten_up_to)
6983 0.006 0.000 0.181 0.000 {built-in method builtins.any}
13966 0.012 0.000 0.175 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:689(pack_sequence_as)
13966 0.010 0.000 0.175 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2851(<genexpr>)
6983 0.051 0.000 0.172 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2196(_build_call_outputs)
6983 0.018 0.000 0.165 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:124(is_compatible_with)
6983 0.016 0.000 0.164 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:154(lookup)
13966 0.032 0.000 0.163 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:649(_pack_sequence_as)
13966/6983 0.058 0.000 0.158 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1029(assert_shallow_structure)
13966 0.044 0.000 0.151 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/handle_data_util.py:25(copy_handle_data)
6983 0.031 0.000 0.146 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:70(is_compatible_with)
1 0.000 0.000 0.141 0.141 <__array_function__ internals>:2(concatenate)
6983 0.030 0.000 0.137 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:185(called_without_tracing)
13966 0.020 0.000 0.119 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:98(__eq__)
6983 0.051 0.000 0.116 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:265(_make_execution_context)
251402 0.060 0.000 0.112 0.000 {built-in method builtins.isinstance}
6983 0.014 0.000 0.106 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:314(make_function_signature)
1 0.000 0.000 0.099 0.099 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py:812(_l2_normalize)
1 0.026 0.026 0.099 0.099 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/preprocessing/_data.py:1731(normalize)
6983 0.023 0.000 0.093 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:180(_get_detector)
6983 0.048 0.000 0.090 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_EncodeArg}
48881 0.089 0.000 0.089 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:175(__eq__)
13966 0.016 0.000 0.082 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:201(__eq__)
6983 0.006 0.000 0.082 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1534(_autopacking_conversion_function)
6983 0.011 0.000 0.075 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1524(_should_not_autopack)
6983 0.033 0.000 0.075 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:183(_sequence_like)
1256940 0.063 0.000 0.063 0.000 {built-in method builtins.callable}
69830/20949 0.045 0.000 0.062 0.000 {built-in method builtins.hash}
34915 0.062 0.000 0.062 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:927(executing_eagerly)
20949 0.007 0.000 0.061 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:95(__hash__)
6983 0.019 0.000 0.060 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1320(shape)
1 0.000 0.000 0.059 0.059 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:486(check_array)
6983 0.058 0.000 0.058 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/weakref.py:422(__contains__)
6983 0.015 0.000 0.053 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2087(executing_eagerly)
55866 0.016 0.000 0.052 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/abc.py:96(__instancecheck__)
41898 0.039 0.000 0.050 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1193(dtype)
41898 0.026 0.000 0.049 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:253(_yield_value)
69830 0.049 0.000 0.049 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsNestedOrComposite}
6983 0.005 0.000 0.048 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1242(<listcomp>)
13966 0.016 0.000 0.044 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:343(__eq__)
20949 0.006 0.000 0.043 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:210(__hash__)
27932/13966 0.027 0.000 0.042 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:999(_yield_flat_up_to)
6983 0.020 0.000 0.039 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:229(__tf_tracing_type__)
55866 0.035 0.000 0.035 0.000 {built-in method _abc._abc_instancecheck}
6983 0.011 0.000 0.035 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:761(__init__)
6983 0.013 0.000 0.034 0.000 {built-in method builtins.all}
69830 0.029 0.000 0.032 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:258(_yield_sorted_items)
13967 0.031 0.000 0.031 0.000 {built-in method builtins.getattr}
6983 0.006 0.000 0.030 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:32(is_traceback_filtering_enabled)
20949 0.009 0.000 0.030 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:188(__ne__)
13966 0.016 0.000 0.029 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:6285(get_default_graph)
6983 0.013 0.000 0.029 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:609(_packed_nest_with_indices)
1 0.000 0.000 0.029 0.029 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:90(_assert_all_finite)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/extmath.py:869(_safe_accumulator_op)
1 0.000 0.000 0.028 0.028 <__array_function__ internals>:2(sum)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:2111(sum)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:70(_wrapreduction)
1 0.028 0.028 0.028 0.028 {method 'reduce' of 'numpy.ufunc' objects}
20949 0.008 0.000 0.028 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:340(__hash__)
6983 0.013 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:155(is_compatible_with)
13966 0.023 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:678(as_dtype)
20949 0.009 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:63(base_dtype)
6983 0.013 0.000 0.023 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:771(<listcomp>)
6983 0.005 0.000 0.022 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:146(is_namedtuple)
118711 0.020 0.000 0.020 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1530(<genexpr>)
104747 0.019 0.000 0.019 0.000 {built-in method builtins.len}
6983 0.014 0.000 0.019 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:61(__init__)
6983 0.017 0.000 0.017 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsNamedtuple}
13966 0.010 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2968(input_signature)
6983 0.008 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5418(_distribution_strategy_stack)
13966 0.015 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:850(experimental_get_tracing_count)
20949 0.015 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:50(_is_ref_dtype)
6983 0.008 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_conversion_registry.py:112(get)
6983 0.010 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/six.py:582(iterkeys)
6983 0.013 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2865(<listcomp>)
6983 0.014 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:162(called_without_tracing)
1 0.000 0.000 0.014 0.014 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/extmath.py:51(row_norms)
1 0.000 0.000 0.014 0.014 <__array_function__ internals>:2(einsum)
1 0.000 0.000 0.014 0.014 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/einsumfunc.py:997(einsum)
1 0.014 0.014 0.014 0.014 {built-in method numpy.core._multiarray_umath.c_einsum}
6983 0.006 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/tape.py:220(could_possibly_record)
13966 0.010 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:158(_has_same_structure)
6983 0.011 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:290(__init__)
13966 0.013 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5814(get_default)
13966 0.012 0.000 0.012 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:949(device_name)
6983 0.011 0.000 0.011 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/weakref.py:382(__getitem__)
6983 0.011 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1203(function_call_options)
6983 0.008 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/save_context.py:59(in_save_context)
41898 0.011 0.000 0.011 0.000 {method '_datatype_enum' of 'tensorflow.python.framework.ops.EagerTensor' objects}
6983 0.005 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py:1016(PossibleTapeGradientTypes)
6983 0.010 0.000 0.010 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:539(__exit__)
13976 0.010 0.000 0.010 0.000 {built-in method builtins.hasattr}
6983 0.007 0.000 0.009 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:1107(is_compatible_with)
6983 0.009 0.000 0.009 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:196(__init__)
6983 0.009 0.000 0.009 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsTensor}
6983 0.004 0.000 0.008 0.000 <string>:1(__new__)
6983 0.008 0.000 0.008 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_TapeSetIsEmpty}
13966 0.007 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:546(ensure_initialized)
20949 0.007 0.000 0.007 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsCompositeTensor}
6983 0.006 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2973(flat_input_signature)
13966 0.007 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:60(dtype)
6983 0.005 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2640(_validate_inputs)
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:50(__init__)
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:60(__init__)
6983 0.006 0.000 0.006 0.000 {method '_shape_tuple' of 'tensorflow.python.framework.ops.EagerTensor' objects}
6983 0.006 0.000 0.006 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsMutableMapping}
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:38(__init__)
6983 0.006 0.000 0.006 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_TapeSetPossibleGradientTypes}
27932 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
6983 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/c_api_util.py:100(has_been_garbage_collected)
27932 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2057(context_safe)
13966 0.005 0.000 0.005 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsTypeSpec}
6983 0.005 0.000 0.005 0.000 {method 'acquire' of '_thread.RLock' objects}
13966 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2552(input_signature)
6983 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:216(__exit__)
13966 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:885(_handle)
...
When I increased 'enmbedding_batch_size=256' it crashed OS X 12.3.1
panic(cpu 8 caller 0xffffff8020d449ad): userspace watchdog timeout: no successful checkins from WindowServer in 120 seconds
service: logd, total successful checkins since wake (127621 seconds ago): 12763, last successful checkin: 0 seconds ago
service: WindowServer, total successful checkins since wake (127621 seconds ago): 12751, last successful checkin: 120 seconds ago
service: remoted, total successful checkins since wake (127621 seconds ago): 12763, last successful checkin: 0 seconds ago
service: opendirectoryd, total successful checkins since wake (127621 seconds ago): 12763, last successful checkin: 0 seconds ago
Panicked task 0xffffff95dce7f9c0: 3 threads: pid 133: watchdogd
Backtrace (CPU 8), panicked thread: 0xffffff87770d6550, Frame : Return Address
0xffffffd15babb670 : 0xffffff801d883e2d
0xffffffd15babb6c0 : 0xffffff801d9e3cb6
0xffffffd15babb700 : 0xffffff801d9d350d
0xffffffd15babb750 : 0xffffff801d823a60
0xffffffd15babb770 : 0xffffff801d8841fd
0xffffffd15babb890 : 0xffffff801d8839b6
0xffffffd15babb8f0 : 0xffffff801e116573
0xffffffd15babb9e0 : 0xffffff8020d449ad
0xffffffd15babb9f0 : 0xffffff8020d44600
0xffffffd15babba10 : 0xffffff8020d439c7
0xffffffd15babbb40 : 0xffffff801e0848cc
0xffffffd15babbca0 : 0xffffff801d98a1a6
0xffffffd15babbdb0 : 0xffffff801d88a6db
0xffffffd15babbe10 : 0xffffff801d85ed03
0xffffffd15babbe60 : 0xffffff801d875259
0xffffffd15babbef0 : 0xffffff801d9b61a8
0xffffffd15babbfa0 : 0xffffff801d824246
Kernel Extensions in backtrace:
com.apple.driver.watchdog(1.0)[01A90A91-CE41-37C4-A5C0-BBD735087472]@0xffffff8020d42000->0xffffff8020d44fff
Process name corresponding to current thread (0xffffff87770d6550): watchdogd
Boot args: chunklist-security-epoch=0 -chunklist-no-rev2-dev
Mac OS version:
21E258
Kernel version:
Darwin Kernel Version 21.4.0: Fri Mar 18 00:45:05 PDT 2022; root:xnu-8020.101.4~15/RELEASE_X86_64
Kernel UUID: B6F8637B-0844-355F-8C82-60FA06149384
KernelCache slide: 0x000000001d600000
KernelCache base: 0xffffff801d800000
Kernel slide: 0x000000001d610000
Kernel text base: 0xffffff801d810000
__HIB text base: 0xffffff801d700000
System model name: iMac20,2 (Mac-AF89B6D9451A490B)
System shutdown begun: NO
Hibernation exit count: 0
System uptime in nanoseconds: 580860938653267
Last Sleep: absolute base_tsc base_nano
Uptime : 0x0002104a39e99311
Sleep : 0x00014c05bb0263eb 0x00000000b1a4f4fc 0x00014bf3f0948456
Wake : 0x00014c05c311c37a 0x00000000b156c318 0x00014c05bfe0324e
Compressor Info: 32% of compressed pages limit (OK) and 99% of segments limit (BAD) with 8 swapfiles and OK swap space
Zone info:
Foreign : 0xffffff80354ab000 - 0xffffff80354b9000
Native : 0xffffff81102b8000 - 0xffffffa1102b8000
Readonly: 0xffffff85dcf84000 - 0xffffff877691d000
Metadata: 0xffffffe9d1f0a000 - 0xffffffe9f2cb9000
Bitmaps : 0xffffffe9f2cb9000 - 0xffffffea12cb9000
last started kext at 356398810041: @filesystems.afpfs 11.3.1 (addr 0xffffff7fb6c29000, size 282624)
last stopped kext at 580792842657981: >!AThunderboltEDMSink 5.0.3 (addr 0xffffff7fb65c3000, size 32768)
loaded kexts:
@filesystems.afpfs 11.3.1
@nke.asp_tcp 8.2.1
>!AHIDALSService 1
>!APlatformEnabler 2.7.0d0
>AGPM 127
>X86PlatformShim 1.0.0
>!AUpstreamUserClient 3.6.9
...
Tensorflow-macos & Tensorflow-metal
% pip show tensorflow-macos
WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
Name: tensorflow-macos
Version: 2.8.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: [email protected]
License: Apache 2.0
Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, setuptools, six, tensorboard, termcolor, tf-estimator-nightly, typing-extensions, wrapt
Required-by:
(tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 top2vec % pip show tensorflow-metal
WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
Name: tensorflow-metal
Version: 0.4.0
Summary: TensorFlow acceleration for Mac GPUs.
Home-page: https://developer.apple.com/metal/tensorflow-plugin/
Author:
Author-email:
License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved.
Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
Requires: six, wheel
Required-by:
Any suggestions on how to increase performance?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|