How to write cross-framework machine learning code for tensorflow and pytorch?

Question

Machine learning framework comprise, amongst other things, the following functions:

• augmentations
• metrics and losses

These functions are simple conversions of tensors and seem rather framework independent. However, for example tensorflow's categorical crossentropy loss uses some tensorflow specific functions like tf.convert_to_tensor() or tf.cast(). So it cannot be used easily in pytorch. Also tensorflow heavily prefers to work with tensorflow tensors instead of numpy ones to create tensorflow graphs to my knowledge.

Are there any existing efforts or ideas how to write such functions in a way that they can be used in both frameworks? I'm thinking of pure numpy functions which can be somehow converted to either tensorflow or pytorch.