load model in DQN

Question

#The model loading is folowing:

agents = []
agents.append(TestDQN(env.s_dim_cloud, env.a1_dim_cloud, env.a1_cloud, 
env.a2_cloud, env.a3_cloud, config)) #agent1
agents.append(TestDQN(env.s_dim_n, env.a1_dim_n, env.a1_n, env.a2_n, 
env.a3_n, config)) #agent2
agents.append(TestDQN(env.s_dim, env.a1_dim, env.a1, env.a2, env.a3, 
config)) #agent3

for i in agents:
    state_dict = torch.load(os.path.join(path, 
'model_state_dict_{}'.format(agents.index(i))))
    state_dict2 = torch.load(os.path.join(path, 
'model2_state_dict_{}'.format(agents.index(i))))
    i.q.load_state_dict(state_dict)
    i.q.eval()
    i.q2.load_state_dict(state_dict2)
i.q2.eval()

My question is, in reinforcement learning, such as DQN, why is the test phase worse than the training phase?

Answer 1

The problem is your requirement. You should only require that the files and directory structure be exactly reconstructed after extraction. Not that the archive itself be exactly the same. Instead of running your MD5 on the archive, instead run it on the reconstructed files.

There is no way to assure the same zip result using different compressors, or different versions of the same compressor, or the same version of the same code with different settings. If you do not have complete control of the code creating and compressing the data, e.g., by virtue of having written it yourself and assuring portability across platforms, then you cannot guarantee that the archive files will be the same.

More importantly, there is no need to have that guarantee. If you want to assure the integrity of the transfer, check the result of extraction, not the intermediate archive file. Then your check is even better than checking the archive, since you are then also verifying that there were no bugs in the construction and extraction processes.