Roger monitors changes in HTTP status codes

In: Projects
Published on 2017-10-03
Written from the perspective of a computer security analyst.

Some time ago I found sensitive personal data on a public web page. I notified the system owner, but was curious to see if my report would have any impact. I decided to monitor the web page to see whether the information was removed.

Since I only wanted to know whether the web page was still online, monitoring changes in the HTTP status code (instead of changes in page content) was enough for this use case. One way of keeping up with the status of a webpage is hitting Refresh until you get a different HTTP status code... But I decided to let my computer do this for me, so I could use my time for more interesting activities.

I decided to write roger, a Python tool for HTTP status monitoring. roger tries to load a given url, saves the current HTTP status code, and then sends you an email whenever that status changes.

How does it work?

Roger follows a fairly straight-forward algorithm:

Load the page for the first time and save the resulting HTTP status/error code:

# first time connecting to target to get initstatus
try:
	print("Connecting...")
	connection = urllib.request.urlopen(
	    urllib.request.Request(testurl), timeout=60)
	initstatus = connection.getcode()
	print("Initial statuscode", initstatus)
	connection.close()
except urllib.error.HTTPError as error:
    print(type(error), error)
    initstatus = error.code
    print("Initial statuscode", initstatus)

Then, we keep repeating this process every n minutes, until we get a different result:

while not status_changed:
    print("Entering test loop with timeout of", str(n), "minutes")
    time.sleep(n * 60)
    print("Testing...")
    try:
        connection = urllib.request.urlopen(
            urllib.request.Request(testurl), timeout=60)
        print("Statuscode", connection.getcode())
        if connection.getcode() != initstatus:
            newstatus = connection.getcode()
            print("New statuscode detected", initstatus, "->", newstatus)
            status_changed = True
        connection.close()
    except urllib.error.HTTPError as error:
        print(type(error), error)
        if error.code != initstatus:
            newstatus = error.getcode()
            print("New statuscode detected", initstatus, "->", newstatus)
            status_changed = True
return newstatus

If a status change occurs, Roger composes an email with the details and sends it to me via my local mailserver. You can expand this script with different forms of notification. For example, you can combine roger with my project zazu, to make a Twitter-bot that automatically posts updates about the status of your website. Or you could send the output of roger to an irc-bot, to give you notifications over irc. The possibilities are endless.

roger can be used for all kinds of OSINT activities. I'm considering expanding it later to include content monitoring. The Python code is available on Github.