Change the connection pool size for Python’s “requests” module when in Threading

by

(edit: Perhaps I am wrong in what this error means. Is this indicating that the connection pool at my CLIENT is full? or a connection pool at the SERVER is full and this is the error my client is being given?)

I am attempting to make a large number of http requests concurrently using the python threading and requests module. I am seeing this error in logs: 

WARNING:requests.packages.urllib3.connectionpool:HttpConnectionPool is full, discarding connection:

What can I do to increase the size of the connection pool for requests?

Answer

This should do the trick:

import requests.adapters

session = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100)
session.mount('http://', adapter)
response = session.get("/mypage")

Attribution
Source : Link , Question Author : Skip Huffman , Answer Author : Ekevoo

Leave a Comment