Some time ago I found sensitive personal data on a public web page. I notified the system owner, but was curious to see if my report would have any impact. I decided to monitor the web page to see whether the information was removed.
Since I only wanted to know whether the web page was still online, monitoring changes in the HTTP status code (instead of changes in page content) was enough for this use case. One way of keeping up with the status of a webpage is hitting Refresh until you get a different HTTP status code… But I decided to let my computer do this for me, so I could use my time for more interesting activities.
I decided to write
roger, a Python tool for HTTP status monitoring.
roger tries to load a given url, saves the current HTTP status code, and then sends you an email whenever that status changes.
Roger follows a fairly straight-forward algorithm:
Load the page for the first time and save the resulting HTTP status/error code:
# first time connecting to target to get initstatus try: print("Connecting...") connection = urllib.request.urlopen( urllib.request.Request(testurl), timeout=60) initstatus = connection.getcode() print("Initial statuscode", initstatus) connection.close() except urllib.error.HTTPError as error: print(type(error), error) initstatus = error.code print("Initial statuscode", initstatus)
Then, we keep repeating this process every n minutes, until we get a different result:
while not status_changed: print("Entering test loop with timeout of", str(n), "minutes") time.sleep(n * 60) print("Testing...") try: connection = urllib.request.urlopen( urllib.request.Request(testurl), timeout=60) print("Statuscode", connection.getcode()) if connection.getcode() != initstatus: newstatus = connection.getcode() print("New statuscode detected", initstatus, "->", newstatus) status_changed = True connection.close() except urllib.error.HTTPError as error: print(type(error), error) if error.code != initstatus: newstatus = error.getcode() print("New statuscode detected", initstatus, "->", newstatus) status_changed = True return newstatus
If a status change occurs, Roger composes an email with the details and sends it to me via my local mailserver.
You can expand this script with different forms of notification. For example, you can combine
roger with my project
zazu, to make a Twitter-bot that automatically posts updates about the status of your website. Or you could send the output of
roger to an irc-bot, to give you notifications over irc. The possibilities are endless.
roger can be used for all kinds of OSINT activities. I’m considering expanding it later to include content monitoring. The Python code is available on Github.
Judith van Stegeren is a Dutch computer scientist. She is working as PhD candidate at the University of Twente, where she researches natural language generation for the video games industry. She occassionaly works as a consultant in data engineering for textual data.