How to read the header content of a webpage in Django?

I created a search engine in Django and bs4 that scrapes search results from the Ask.com search engine. I would like when Django fetches search results from Ask, it checks the value of the X-Frame-Options header or if CSP_FRAME_ANCESTORS has the values ​​"'self'", '*' in case the header exists in order to give a value to my notAccept boolean depending on the result of the condition.

I took inspiration from this page of the Django documentation and also from this other page to write the following code:

for page_num in range(1, max_pages_to_scrap+1):
            url = "https://www.ask.com/web?q=" + search + "&qo=pagination&page=" + str(page_num)
            res = requests.get(url)
            soup = bs(res.text, 'lxml')
            result_listings = soup.find_all('div', {'class': 'PartialSearchResults-item'})

            for result in result_listings:
                result_title = result.find(class_='PartialSearchResults-item-title').text
                result_url = result.find('a').get('href')
                result_desc = result.find(class_='PartialSearchResults-item-abstract').text
                final_result.append((result_title, result_url, result_desc))

            for header in final_result[1]:
                if(header.CONTENT_TYPE.X-Frame-Options == "DENY" | header.CONTENT_TYPE.X-Frame-Options == "SAMEORIGIN" | !(CSP_FRAME_ANCESTORS == ("'self'", '*'))): #The error is generated on this line
                     head = True
                     notAccept = bool(head)
                else:
                    notAccept = bool(False)

However, this gives me the following errors in the command line of my IDE:

"(" was not closed
Expected expression
Expected ":"

and the following warnings:

"Frame" is not defined
"Options" is not defined
"CSP_FRAME_ANCESTORS" is not defined

As if Django doesn't consider them as headers. All I want is to be able to read the contents of the header of each returned webpage. So I hope to count on the help of the community, thank you!

This is more of a Python Requests issue than Django.. from my limited knowledge I don't believe you can get header information of pages just by looking at the Links. You need to actually send a GET request because it's in the Header of the request:

for l in links:
    response = requests.get(l)
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        head = True
        notAccept = bool(head)
    else:
        notAccept = bool(False)

I was unable to find anything about CSP_FRAME_ANCESTORS tho..

Hopefully you can find some of this useful.. at the very least now you know to search python requests {x} on the topic

Back to Top