How to read total number of issues in Gitlab and store it in pandas/python?

Question

I need to extract all the issues out of Gitlab through url, but it can only give 100 records per page. Is there any way to read all the records from all the pages? For ex- Gitlab may have 5 pages, with total 550 records. But I can only get 1st page, or 2nd page, or 3rd page so on... with 100 records only. I want all 550 issues to be read with that single url.

url = "https://gitlab.com/api/v4/projects/00000000/issues?page=1&per_page=100&labels=xyz"

payload={}
headers = {'Authorization': 'Bearer XXXX-XXXXXXXXX'}

response = requests.request("GET", url, headers=headers, data=payload)
df = pd.read_json(io.StringIO(response.text))

Could anyone please help me out?

Answer 1

You need to make multiple requests, but it's easily done with a while loop. It works by using the link provided in the previous response header to make a new request before appending it to the original (and any previous) response content.

According to the API docs, the Issues are contained in a list, but it might not be the case, so you probably need need some extra logic and checks in there, but this should work as is:

url = "https://gitlab.com/api/v4/projects/4339844/issues"
payload= {}
headers = {}

content = []
link = url

while link:
    r = requests.get(link, headers=headers, data=payload)
    if r.status_code == 200:
        content += r.json()
    try:
        link = r.links['next']['url']
        print(link)
    except:
        link = ''

df = pd.DataFrame(content)
print(df)