403. That’s an error - Rate-limit exceeded That’s all we know - Image from YouTube Data API

Question

I've created a website that uses the YouTube data API to get YouTube data for a chosen user. This has always worked fine for me, but the amount of traffic is now at a level that embedding YouTube images is no longer a 100% solid.

I use the YouTube Data API V3 to get the image URL of YouTube banner images, which are then added as the src attribute for an existing <img> element.

Because the website has pretty high amounts of traffic (mostly 500-3000 realtime users) the YouTube image servers (like this one: https://yt3.ggpht.com) started returning a 403 error when my server sends to many (image) GET requests. The error looks like this:

---

403. That’s an error.

Your client does not have permission to get URL https://yt3.ggpht.com/Xs3rhrNjuGoNnd6aXPdZqdfiOIp5EbvN9-PYSaP8xTY1cx6Ced5No6sHNZupNrQmsidY2W-X=w2560-fcrop64=1,00005a57ffffa5a8-nd-c0xffffffff-rj-k-no from this server. (Client IP address: xx.xx.xxx.xxx)

Rate-limit exceeded That’s all we know.

---

From the moment this limit is exceeded, my website shows no images (from the YouTube image server) for a while and after x minutes when the exceeded limit is reset the images work again. I want to prevent this from happening, because the YouTube bannerimage is important for the UX of my website.

My first question: Does anyone know what the exact numbers are for this rate limit on YouTube images? How many requests can be done in X time?

My second question: Does anyone know a solution to prevent triggering this error and still serve YouTube images to every user?

I have some theories of my own but they would have a lot of impact on my servers:

• Since the rate limit is bound to an IP address, I could run my website on multiple IP's with some kind of load balancer.
• I could save the images to my server when it is requested for the first time as a caching mechanism. When the image would get requested again I could serve it from my own server, which prevents limits from being exceeded. To prevent updated YouTube images from not showing I would have to delete the images every X hours.

I'd rather not do this because both of these theories would have big impact on the server load.

Answer 1

Both your theories would work in practice but as you mentioned, it's a lot of work. You can add the referrerPolicy="no-referrer" to your img tag and it should make the request without a referrer header, making the request not return a 403 anymore.

My assumption is that you, like myself were running on a local server with a popular port and Google/Youtube's image server calculates rate limits based on the referrer header (Although a very poor method, this seems logical based on the HTTP interaction between the image request getting a 200 VS a 403 being decided based on the referrer header). This means everyone who is accessing YouTube channel logos using localhost:3000 would be sharing the same rate limit due to sharing the same referrer header value and whatever the rate limit is being exceeded. This is only my theory though as it's impossible to tell what YouTube is doing on their backend.

Answer 2

I performed some experiments to understand better the problem.

It seems like YouTube images servers perform a kind of matching on referer (and probably on IP too) to limit the requests from external environments

As far as I've understood, the problem lies in HTTP referer. Indeed, when you experience a 403 error you can try to modify the HTTP request, changing the referer, and it works like a charm.

From the technical point of view, I think it is not possibile doing that via Javascript, you could set up a proxy to process all the img requests and change the referer as you like. The drawback is that all requests pass through your proxy and come from the same IP.

From the YouTube policy point of view, I do not know if that solution could be fine. Probably not (otherwise why that limitations?).