How-To: Make a Bot to Track Newegg GPU Prices and Get Notifications

Graphics card pricing is ridiculous and it’s hard to find a good deal these days. Let’s try to fix that.

After spending way more time trying to get a new graphics card than I’m comfortable admitting, I finally gave up. There’s no beating the bots, and now that scalpers have discovered graphics cards and consoles, there’s no telling when prices will get back to normal. That said, the transistor shortage that started in the second half of 2020 may be showing signs of letting up. Graphics cards seem to be more readily available over the last month or so, but never anywhere near the mythical (and mostly made up) MSRP. At least it’s a small improvement.

Even though I’ve given up on my quest for the ultimate /r/pcmasterrace gaming experience, I figured I could at least use my skills and knowledge and possibly help someone else succeed in their goal of getting a new graphics card. In this project, I’ll use a library called Beautiful Soup to analyze a website (Newegg) and check for desired pricing. I’ll also set up desktop and email notifications so you’ll never miss a deal. Here’s a link to the finished project in case you want to clone the project files and get started tracking. I made specific trackers for the RTX 3060 Ti and RTX 3070, but there’s also a generic tracker that you can use to track any product on Newegg.

Before you continue, make sure you have Python on your system.

• If you’re on Windows, the easiest way to get Python is to download it from the Microsoft Store. Microsoft has done a lot of work towards making Windows more developer friendly over the last few years.
• If you’re on a Linux distribution, you probably already have it installed but here are guides for Arch, Debian (incl. Ubuntu & Pop!_OS), and Red Hat based distributions.
• If you’re on MacOS, things appear to be significantly more complicated (because Apple). Here’s a walkthrough on freeCodeCamp.org.

About Beautiful Soup

Beautiful Soup is a Python library for parsing HTML and XML documents like web pages. By using Beautiful Soup, we can automate and speed up the process of visiting a website and looking for information. It’s especially useful on websites that always use a similar page layout and HTML structure like online stores and other websites with a lot of static content. Beautiful Soup does run into some issues with websites that use a lot of JavaScript to render the page, because that code is executed in the web browser (Amazon and Reddit come to mind). For this project we’re sticking to Newegg. I haven’t run into any problems yet (fingers crossed).

Dependencies

Before getting started, make sure you have the required dependencies I’ve used in this project.

For Linux:

pip3 install lxml beautifulsoup4 notify-py playsound

For Windows

pip3 install lxml beautifulsoup4 notify-py playsound=1.2.2

Important: Make sure you use playsound version 1.2.2 on Windows, as newer versions have issues with Windows using \ instead of / as a path separator.

requests is how we will retrieve the HTML document from the website so that it can be parsed by Beautiful Soup. smtplib is used to send email notifications. notifypy is used to send desktop notifications on both Windows and Linux (not sure on MacOS). playsound does exactly what it sounds like it does: plays an audio file. This goes hand-in-hand with notifypy sending a visual notification. And of course BeautifulSoup for digging through HTML documents.

Setup

There are a few pre-coding steps to go through that involve creating/editing config files and creating a dummy email account for the bot, so be sure to follow those in the README on the project page.

Code Walkthrough

Now that we’ve got Python and all the project dependencies installed, let’s get to the tracker. I’m using newegg-3060-ti.py as the basis for all of the following example code snippets.

Before we do anything, there are a number of imports.

import sys, requests, smtplib, config, smtpConfig
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from notifypy import Notify
from playsound import playsound
from time import sleep
from bs4 import BeautifulSoup
from threading import Thread
from datetime import datetime

Next we parse command line arguments. For the pre-made trackers, they accept one argument: a price limit. For the generic tracker, it accepts a filtered Newegg search URL and a price limit.

if len(sys.argv) == 2:
    try:
        priceLimit = int(sys.argv[1])
    except:
        sys.exit(f"\n🛑  Something went wrong. Here's a descriptive error message.")
elif len(sys.argv) == 1:
    priceLimit = 600
else:
    sys.exit("\n🛑  Incorrect usage. Here's a descriptive error message.")

The first function that is called is getItems(). This function uses the requests library to fetch a web page, then uses Beautiful Soup to parse the returned document and find all items on the page. On Newegg’s search page, these all have a common CSS class: item-container. That makes them easy to find.

def getItems():
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "lxml")
    containers = soup.find_all("div", class_="item-container")
    return containers

The next function in the project is checkPrices(). This function loops through each item-container HTML tag returned by getItems() and manipulates the text in such a way that it can be compared to the specified price limit. If the product is under the price limit, information like price, name, and product page link is gathered about the product. The item is then added to a list to be returned.

As I was testing, I noticed that an exception was thrown randomly when checking prices. There were sometimes one or two item-container entities with different tags (seemed to be promoted products or ads), so that caused issues. I solved that problem by simply catching the exception when it was thrown.

def checkPrices(items):
    deals = []
    for item in items:
        try:
            stringPrice = item.find("li", class_="price-current").find("strong").text
            intPrice = stringPrice.replace(",", "")
            intPrice = int(intPrice)
            if(intPrice <= priceLimit):
                nameTag = item.find("div", class_="item-info").find("a", class_="item-title")
                itemName = nameTag.text
                itemLink = nameTag.get("href")
                dealItem = {
                    "price": "$" + stringPrice,
                    "name": itemName,
                    "href": itemLink
                }
                deals.append(dealItem)
        except:
            pass
    return deals

The third function in this project is notify(). It does exactly what it says it does: it sends a notification to whatever desktop environment you’re in (Windows example) (GNOME example) and plays an audio file so you’ll be sure to notice it. The Discovery.mp3 file in the project repository is what I used. It’s short enough and sounds nice.

Note: make sure you’ve followed the setup steps in the README on the GitHub repository, as that’s where the notification sound you want to use is defined.

def notify():
    notification = Notify()
    notification.title = "Found 3060 Ti in stock!"
    notification.message = "Check your terminal!"
    notification.send()
    playsound(config.notificationSoundFile)

The last function the bot uses is sendEmail(). Another pretty self-explanatory function name (which should always be the goal!). Again, make sure you’ve followed the setup steps in the GitHub repository as this function relies on a smtpConfig.py file that contains the bot’s email address and password as well as the email address you would like it to send email notifications to.

def sendEmail(subject, msg):
    message = MIMEMultipart("alternative")
    message["To"] = smtpConfig.recipientEmail
    message["From"] = smtpConfig.senderEmail
    message["Subject"] = subject

    htmlMsg = f' <html> <body> <p> {msg} </p> </body> </html> '

    message.attach(MIMEText(msg, 'plain'))
    message.attach(MIMEText(htmlMsg, 'html'))

    smtp = smtplib.SMTP("smtp.gmail.com", 587)
    smtp.starttls()
    smtp.login(smtpConfig.senderEmail, smtpConfig.senderPassword)
    smtp.sendmail(smtpConfig.senderEmail, smtpConfig.recipientEmail, message.as_string())
    smtp.quit()

Lastly, we have the main loop of the bot that checks prices and sends notifications and emails if necessary. The notify() and sendEmail() functions can be started in their own threads to keep the loop on a consistent timer (defined in the project setup). The main logic is as follows:

1. Get the products from the Newegg search page
2. Check their prices and add them to a list if they match our price limit
3. If at least one deal was found, loop through the list and add each of them to a message to be printed to the console as well as a message to be sent in an email.
4. Send the desktop notification and email.
5. Wait for a specified number of seconds before the next check. (Waiting longer between checks should reduce the likelihood of you getting IP banned either temporarily or permanently. Something to keep in mind. I’ve run this bot for a couple days non-stop with a 15 second sleep timer and it seems to be fine.)

For simplicity’s sake, I’ll give a cut-down example of the loop here. Feel free to check out the full code in the project repository.

while True:
    try:
        items = getItems()
        deals = checkPrices(items)

        if len(deals) > 0:
            date = datetime.now()
            timestamp = date.strftime("%d-%b-%Y (%H:%M:%S)")
            msg = f"<h1>Found something for you!</h1><h3>{timestamp}</h3>"
            for deal in deals:
                consoleMessage += f"{deal['price']}\n{deal['name'][:75]}\n{deal['href']}\n\n"
                msg += f"Price:</strong> {deal['price']}<br><strong>Product</strong>: {deal['name'][:75]}<br><strong>Link</strong>: {deal['href']}<br><br>"
            print(consoleMessage)
            notify()
            sendEmail("Found a deal!", msg)
        sleep(config.sleepDuration)
    except KeyboardInterrupt:
        sys.exit("\n🛑 Price checker killed.")

And that’s it! These code blocks are all you need to get started tracking prices on Newegg and get notified when products are in stock!

Future Improvements

There’s definitely room for improvement here. It’s a little annoying to have to mess with config files. One thing that could be done is to prompt for the necessary information the first time the script is started, then write that information to a file to be used later. Another improvement would be to implement OAuth authentication so that the “Less secure app access” setting in the Google account settings can be kept turned off.

We’ll see how much time I feel like spending on improving this bot. I may get distracted and move on to other projects. Until then, happy hunting and good luck!