I hope you are all doing well!
Here are some new updates I would like to share with you about the web scraper.
Based off of my TA @bhanutejap I migrated to using selenium with python
This is what he posted
python from selenium import webdriver driver = webdriver.Firefox(executable_path=r'C:\Users\bteja\Documents\geckodriver.exe') ##this is the path where my geckodriver exists hivestats_url = 'https://hivestats.io/@bhanutejap' driver.get(hivestats_url) vote_value_percentage = driver.find_elements_by_xpath('/html/body/div/div/div/div/div/section/div/div/div/span/span') print(vote_value_percentage.text)
When I first tested this, it worked wonderfully and showed that I had 100% voting power!
I got excited!
But once I voted and ran the code again it gave me the error below
print(vote_value_percentage.text) IndexError: list index out of range
I modified the code a little but the issue remains the same
python from selenium import webdriver from selenium.webdriver.firefox.service import Service from selenium.webdriver.firefox.options import Options as FirefoxOptions options = FirefoxOptions() options.add_argument("--headless") driver = webdriver.Firefox(options=options) #this is the path where my geckodriver exists def get_voting_power(username): hivestats_url = 'https://hivestats.io/@' + username driver.get(hivestats_url) vote_value_percentage = driver.find_elements_by_xpath('/html/body/div/div/div/div/div/section/div/div/div/span/span') #/html/body/div/div/div/div/div/section/div/div/div/span/span return vote_value_percentage.text username = "armoredbanana" response = get_voting_power(username) print("The user: ", username, " has: ", response, " voting power")
But if we use
It outputs an empty list as the answer.
This means that our code is not picking up the xpath, but when it is 100% it does!
python output 100.00%
It gets even more interesting
I reran @bhanutejap 's code and found to give me 100% as well
Furthermore to investigate I grabbed the xpath of the time that it will take for my voting power to re charge, which is 2 hours right now. But look at what I get as the output
output Never voted before.
Very strange. The web scraper is getting data that I have never voted before, or that I have 100% voting power.
I am not sure what is going on, but whatever it is, its very interesting! I will keep you guys updated on how it goes and if you have any suggestions please feel free to let me know!