mlflow model artifacts are not getting stored, while running the airflow dag. for that reason unable to fetch experiment details?

Question

**mlflow training code:**

import mlflow
from mlflow.tracking import MlflowClient
client = MlflowClient()

" training the model and saving the model artificats"
mlflow.set_registry_uri('postgresql://postgres:postgres@localhost/mlflow')
mlflow.set_experiment('testing_mlflow_with_airflow')
with mlflow.start_run():
    # creating the training dataframe
    train_x = self.train_data[0]
    train_y = self.train_data[1]

    # training the given model
    model.fit(train_x, train_y)
                
    mlflow.sklearn.log_model(model, "model")


" getting the experiment details by experiment name"
experiment_id = client.get_experiment_by_name('testing_mlflow_with_airflow').experiment_id
experiment_results = mlflow.search_runs(experiment_ids=experiment_id) '''


**airflow task code:**

training = BashOperator(
              task_id = 'mlflow_training',
              bash_command='python3 /home/vasanth/airflow/scripts/mlproject/src/models/train_mlflow.py',
              do_xcom_push=False
               )


**airflow error:**

[2022-04-29, 13:01:08 UTC] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'python3 /home/vasanth/airflow/scripts/mlproject/src/models/train_mlflow.py']

[2022-04-29, 13:01:08 UTC] {subprocess.py:85} INFO - Output:

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - WARNING:root:Malformed experiment '2'. Detailed error Yaml file '/tmp/airflowtmpzjvuldm6/mlruns/2/meta.yaml' does not exist.

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - Traceback (most recent call last):

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -   File "/usr/local/lib/python3.8/dist-packages/mlflow/store/tracking/file_store.py", line 262, in list_experiments

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -     experiment = self._get_experiment(exp_id, view_type)

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -   File "/usr/local/lib/python3.8/dist-packages/mlflow/store/tracking/file_store.py", line 341, in _get_experiment

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -     meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -   File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/file_utils.py", line 179, in read_yaml

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO -     raise MissingConfigException("Yaml file '%s' does not exist." % file_path)

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - mlflow.exceptions.MissingConfigException: Yaml file '/tmp/airflowtmpzjvuldm6/mlruns/2/meta.yaml' does not exist.

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - tracking uri ***ql://***:***@localhost/mlflow

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - Traceback (most recent call last):

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - File "/home/vasanth/airflow/scripts/mlproject/src/models/train_mlflow.py", line 44, in <module>


[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - experiment_id = client.get_experiment_by_name('testing_mlflow_with_airflow').experiment_id

[2022-04-29, 13:01:28 UTC] {subprocess.py:89} INFO - AttributeError: 'NoneType' object has no attribute 'experiment_id'

[2022-04-29, 13:01:29 UTC] {subprocess.py:93} INFO - Command exited with return code 1

[2022-04-29, 13:01:29 UTC] {taskinstance.py:1774} ERROR - Task failed with exception

how i can set a directory where all my experiment runs artifacts will be stored? where the mlflow artifacts are getting stored now? how i can find all runs details by the mlflow client as per the above code?

i have tried with different approaches, None of them is worked

1. setting the tracking server as below mlflow.set_tracking_uri('postgresql://postgres:postgres@localhost/mlflow') mlflow.set_tracking_uri('file:///tmp/mlruns')

2. mlflow.set_tracking_uri('http://localhost:5000')

3. mlflow.set_registry_uri('postgresql://postgres:postgres@localhost/mlflow') mlflow.set_tracking_uri('/home/vasanth/airflow/scripts/mlproject/src/models')