How to read the header content of a webpage in Django?
I created a search engine in Django and bs4 that scrapes search results from the Ask.com search engine. I would like when Django fetches search results from Ask, it checks the value of the X-Frame-Options header or if CSP_FRAME_ANCESTORS
has the values "'self'", '*'
in case the header exists in order to give a value to my notAccept
boolean depending on the result of the condition.
I took inspiration from this page of the Django documentation and also from this other page to write the following code:
for page_num in range(1, max_pages_to_scrap+1):
url = "https://www.ask.com/web?q=" + search + "&qo=pagination&page=" + str(page_num)
res = requests.get(url)
soup = bs(res.text, 'lxml')
result_listings = soup.find_all('div', {'class': 'PartialSearchResults-item'})
for result in result_listings:
result_title = result.find(class_='PartialSearchResults-item-title').text
result_url = result.find('a').get('href')
result_desc = result.find(class_='PartialSearchResults-item-abstract').text
final_result.append((result_title, result_url, result_desc))
for header in final_result[1]:
if(header.CONTENT_TYPE.X-Frame-Options == "DENY" | header.CONTENT_TYPE.X-Frame-Options == "SAMEORIGIN" | !(CSP_FRAME_ANCESTORS == ("'self'", '*'))): #The error is generated on this line
head = True
notAccept = bool(head)
else:
notAccept = bool(False)
However, this gives me the following errors in the command line of my IDE:
"(" was not closed
Expected expression
Expected ":"
and the following warnings:
"Frame" is not defined
"Options" is not defined
"CSP_FRAME_ANCESTORS" is not defined
As if Django doesn't consider them as headers. All I want is to be able to read the contents of the header of each returned webpage. So I hope to count on the help of the community, thank you!
This is more of a Python Requests issue than Django.. from my limited knowledge I don't believe you can get header information of pages just by looking at the Links. You need to actually send a GET request because it's in the Header of the request:
for l in links:
response = requests.get(l)
if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
head = True
notAccept = bool(head)
else:
notAccept = bool(False)
I was unable to find anything about CSP_FRAME_ANCESTORS
tho..
Hopefully you can find some of this useful.. at the very least now you know to search python requests {x}
on the topic