'Reinforcement Learning using PPO2- Why my ep_rewmean (reward) during training is always showing nan?

I am currently implementing a reinforcement learning problem similar to buying/selling shares in the stock exchange. It's an online implementation and I am creating steps based on a dataset and its a customized environment. Now, the problem is that during my training timesteps, I observe that the value of one of the performance metrics ep_rewmean and eplenmean is always nan as follows,

enter image description here

My environment is

env = DummyVecEnv([lambda: ETradingEnv(df)])
model = PPO2('MlpPolicy', env, verbose=1)  
model.learn(total_timesteps=200) 

My Step Function is,

def step(self, action): 
    # Execute one time step within the environment
    amount=self._take_action(action)  
    self.amount += amount

    self.current_step += 1
    if self.current_step > len(self.df.loc[:, 'current_l'].values) - 6:
      self.current_step = 0
    delay_modifier = (self.current_step / MAX_STEPS)  

    # Reward 1
    r1 = -1 * self.delta * amount* self.current_price 

    # Reward 2
    d=self.df.loc[self.current_step, 'd']         
    if del == 1:                                            
      r2= -1 * (self.s-MAX_S)**2
    else:
      r2=0
    reward = r1 + r2
    # reward=100
    done = d == 1
    obs = self._next_observation()
    print(f'reward: {reward}')
    return obs, reward, done, {}

How can I resolve this issue? Any help would be highly appreciated. Thanks in advance.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source