'Using Service Worker in Angular leading to 504 Error

I tried to implement Service Workers on an Angular Project lately, it seems to be caching all the files accordingly, the problem is, we kinda get a weird behaviour for going offline. When going Offline, the site can be used as normal and everything works fine, when trying to refresh the site, we get an 504 Error. On the other hand, if we check the "update on reload" checkbox on chrome (which should only create a new instance for the service worker AFAIK), reload the site online, and then uncheck the "update on reload" and go offline again, i can refresh as often as i like, everything works just fine. We also tried to implement a logic to have the service worker update everytime the site is refreshed, but this doesn't seem to be working either.

Does anyone know what is causing this weird behaviour? I mean it seems to be working just fine, but just under certain circumstances.