Google Translation Api Id Blocking Ip Address For Too Many Requests

I'm setting up a Django Views that requests products data from an API, parse them with BeautifulSoup, apply the googletrans module and save the response into my Postgresql database

Solution 1:

I suggest you read this article from MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

if this is the response you get so try and look at the header Retry-After in the response object.

so adding a sleep or other delay method, with the value of that header might fix your problem.

Solution 2:

Try adding delays between consecutive queries(using sleep) and play with the numbers to see what works for you. 2s delay after every pair of translation and 15s after every 10 to works fine for me.

Solution 3:

I have been blocked too because of many concurrent requests. Usually always gets blocked after 500 concurrent requests. What I did was to put a timeout of 60 seconds after every 100 concurrent requests. It may seem long, but it works. You could also achieve that with a 45 seconds timeout, but I set it to 60 just to make sure.

Here's an example

classGoogleAPI():

    def__init__(self):
        self.limit_before_timeout = 100
        self.timeout = 60deftranslate(self, source):
        translation = translator.translate(source, dest="ar")
        translation = translation.__dict__['text']
        if translation != ""and translation isnotNone:
            return translation

    defprocess(self):
        i = 0print("initiation")
        for t in list_of_data:
            if i < self.limit_before_timeout:
                i += 1
                self.translate(t)
            else:
                i = 0print("100 words added")
                time.sleep(self.timeout)
        print("All done")

Solution 4:

My IP is blocked after ~450 concurrent connections. I am using php for loop to translate my text array.

So, I changed my IP Address and and changed my code for waiting after every x seconds.

My Codes in For loop ($i is value from for loop):

if ($i % 100 == 0 && $i!=0) {
    //wait 60 seconds every 100usleep(60000000);   // 60 seconds
    echo str_pad("XX--> WAITING 60 SECONDS<br>",4096);
}               
else 
if ($i % 10 == 0  && $i!=0) {
    //wait 15 seconds every 10usleep(15000000); // 15 seconds
     echo str_pad("XX--> WAITING 15 SECONDS<br>",4096);
}
else    
if ($i % 2 == 0  && $i!=0) {
    //wait 2 seconds every 2usleep(2000000); // 2 seconds
     echo str_pad("XX--> WAITING 2 SECONDS<br>",4096);
}