{"id":288,"date":"2022-02-03T01:08:20","date_gmt":"2022-02-03T07:08:20","guid":{"rendered":"https:\/\/rpayne.dev\/blog\/?p=288"},"modified":"2022-02-03T01:09:46","modified_gmt":"2022-02-03T07:09:46","slug":"how-to-make-a-bot-to-track-newegg-gpu-prices-and-get-notifications","status":"publish","type":"post","link":"https:\/\/rpayne.dev\/blog\/how-to-make-a-bot-to-track-newegg-gpu-prices-and-get-notifications\/","title":{"rendered":"How-To: Make a Bot to Track Newegg GPU Prices and Get Notifications"},"content":{"rendered":"\n<figure class=\"wp-block-image alignwide size-full is-style-default\"><img loading=\"lazy\" decoding=\"async\" width=\"1003\" height=\"749\" src=\"https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo.png\" alt=\"\" class=\"wp-image-289\" srcset=\"https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo.png 1003w, https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo-300x224.png 300w, https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo-768x574.png 768w, https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo-825x616.png 825w, https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo-600x448.png 600w, https:\/\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/01\/tracker-demo-400x299.png 400w\" sizes=\"auto, (max-width: 1003px) 100vw, 1003px\" \/><\/figure>\n\n\n\n<h1 class=\"wp-block-heading\" id=\"how-to-make-a-bot-to-track-newegg-gpu-prices-and-get-notifications\">How-To: Make a Bot to Track Newegg GPU Prices and Get Notifications<\/h1>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"graphics-card-pricing-is-ridiculous-and-it-s-hard-to-find-a-good-deal-these-days-let-s-try-to-fix-that\">Graphics card pricing is ridiculous and it&#8217;s hard to find a good deal these days. Let&#8217;s try to fix that.<\/h5>\n\n\n\n<p>After spending way more time trying to get a new graphics card than I&#8217;m comfortable admitting, I finally gave up. There&#8217;s no beating the bots, and now that scalpers have discovered graphics cards and consoles, there&#8217;s no telling when prices will get back to normal. That said, the transistor shortage that started in the second half of 2020 may be showing signs of letting up. Graphics cards seem to be more readily available over the last month or so, but never anywhere near the mythical (and mostly made up) MSRP. At least it&#8217;s a small improvement.<\/p>\n\n\n\n<p>Even though I&#8217;ve given up on my quest for the ultimate <a href=\"http:\/\/reddit.com\/r\/pcmasterrace\" data-type=\"URL\" data-id=\"reddit.com\/r\/pcmasterrace\">\/r\/pcmasterrace<\/a> gaming experience, I figured I could at least use my skills and knowledge and possibly help someone else succeed in their goal of getting a new graphics card. In this project, I&#8217;ll use a library called Beautiful Soup to analyze a website (Newegg) and check for desired pricing. I&#8217;ll also set up desktop and email notifications so you&#8217;ll never miss a deal. <a href=\"https:\/\/github.com\/rosspayn3\/newegg-tracker\" data-type=\"URL\" data-id=\"https:\/\/github.com\/rosspayn3\/newegg-tracker\">Here&#8217;s a link to the finished project<\/a> in case you want to clone the project files and get started tracking. I made specific trackers for the RTX 3060 Ti and RTX 3070, but there&#8217;s also a generic tracker that you can use to track any product on Newegg.<\/p>\n\n\n\n<p>Before you continue, <strong>make sure you have Python on your system<\/strong>.<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>If you&#8217;re on Windows, the easiest way to get Python is to <a href=\"https:\/\/www.microsoft.com\/store\/productId\/9P7QFQMJRFP7\" data-type=\"URL\" data-id=\"https:\/\/www.microsoft.com\/store\/productId\/9P7QFQMJRFP7\">download it from the Microsoft Store<\/a>. Microsoft has done a lot of work towards making Windows more developer friendly over the last few years.<\/li><li>If you&#8217;re on a Linux distribution, you probably already have it installed but here are guides for <a href=\"https:\/\/wiki.archlinux.org\/title\/python\" data-type=\"URL\" data-id=\"https:\/\/wiki.archlinux.org\/title\/python\">Arch<\/a>, <a href=\"https:\/\/wiki.debian.org\/Python\" data-type=\"URL\" data-id=\"https:\/\/wiki.debian.org\/Python\">Debian (incl. Ubuntu &amp; Pop!_OS)<\/a>, and <a href=\"https:\/\/developers.redhat.com\/blog\/2018\/08\/13\/install-python3-rhel\" data-type=\"URL\" data-id=\"https:\/\/developers.redhat.com\/blog\/2018\/08\/13\/install-python3-rhel\">Red Hat<\/a> based distributions.<\/li><li>If you&#8217;re on MacOS, things appear to be significantly more complicated (because Apple). <a href=\"https:\/\/www.freecodecamp.org\/news\/python-version-on-mac-update\/\" data-type=\"URL\" data-id=\"https:\/\/www.freecodecamp.org\/news\/python-version-on-mac-update\/\">Here&#8217;s a walkthrough on freeCodeCamp.org<\/a>.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"about-beautiful-soup\">About Beautiful Soup<\/h2>\n\n\n\n<p><a href=\"https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/\" data-type=\"URL\" data-id=\"https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/\">Beautiful Soup<\/a> is a Python library for parsing HTML and XML documents like web pages. By using Beautiful Soup, we can automate and speed up the process of visiting a website and looking for information. It&#8217;s especially useful on websites that always use a similar page layout and HTML structure like online stores and other websites with a lot of static content. Beautiful Soup does run into some issues with websites that use a lot of JavaScript to render the page, because that code is executed in the web browser (Amazon and Reddit come to mind). For this project we&#8217;re sticking to Newegg. I haven&#8217;t run into any problems yet (fingers crossed).<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"dependencies\">Dependencies<\/h2>\n\n\n\n<p>Before getting started, make sure you have the required dependencies I&#8217;ve used in this project.<\/p>\n\n\n\n<p><strong>For Linux:<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><code>pip3 install lxml beautifulsoup4 notify-py playsound<\/code><\/code><\/pre>\n\n\n\n<p><strong>For Windows<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><code>pip3 install lxml beautifulsoup4 notify-py playsound=1.2.2<\/code><\/code><\/pre>\n\n\n\n<p id=\"important-make-sure-you-use-playsound-version-1-2-2-on-windows-as-newer-versions-have-issues-with-windows-using-instead-of-as-a-path-separator\"><strong><em>Important:<\/em><\/strong> Make sure you use <code>playsound <\/code>version 1.2.2 on Windows, as newer versions have issues with Windows using <code>\\<\/code> instead of <code>\/<\/code> as a path separator.<\/p>\n\n\n\n<p><code>requests<\/code> is how we will retrieve the HTML document from the website so that it can be parsed by Beautiful Soup. <code>smtplib<\/code> is used to send email notifications. <code>notifypy<\/code> is used to send desktop notifications on both Windows and Linux (not sure on MacOS). <code>playsound<\/code> does exactly what it sounds like it does: plays an audio file. This goes hand-in-hand with <code>notifypy<\/code> sending a visual notification. And of course <code>BeautifulSoup<\/code> for digging through HTML documents.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup\">Setup<\/h2>\n\n\n\n<p>There are a few pre-coding steps to go through that involve creating\/editing config files and creating a dummy email account for the bot, so be sure to follow those in the <a href=\"https:\/\/github.com\/rosspayn3\/newegg-tracker#setup\">README on the project page<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"code-walkthrough\">Code Walkthrough<\/h2>\n\n\n\n<p>Now that we&#8217;ve got Python and all the project dependencies installed, let&#8217;s get to the tracker. I&#8217;m using newegg-3060-ti.py as the basis for all of the following example code snippets.<\/p>\n\n\n\n<p>Before we do anything, there are a number of imports. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import sys, requests, smtplib, config, smtpConfig\nfrom email.mime.text import MIMEText\nfrom email.mime.multipart import MIMEMultipart\nfrom notifypy import Notify\nfrom playsound import playsound\nfrom time import sleep\nfrom bs4 import BeautifulSoup\nfrom threading import Thread\nfrom datetime import datetime\n<\/code><\/pre>\n\n\n\n<p>Next we parse command line arguments. For the pre-made trackers, they accept one argument: a price limit. For the generic tracker, it accepts a filtered Newegg search URL and a price limit.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>if len(sys.argv) == 2:\n    try:\n        priceLimit = int(sys.argv&#91;1])\n    except:\n        sys.exit(f\"\\n\ud83d\uded1  Something went wrong. Here's a descriptive error message.\")\nelif len(sys.argv) == 1:\n    priceLimit = 600\nelse:\n    sys.exit(\"\\n\ud83d\uded1  Incorrect usage. Here's a descriptive error message.\")<\/code><\/pre>\n\n\n\n<p>The first function that is called is <code>getItems()<\/code>. This function uses the <code>requests<\/code> library to fetch a web page, then uses Beautiful Soup to parse the returned document and find all items on the page. On Newegg&#8217;s search page, these all have a common CSS class: <code>item-container<\/code>. That makes them easy to find.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def getItems():\n    response = requests.get(url)\n    soup = BeautifulSoup(response.text, \"lxml\")\n    containers = soup.find_all(\"div\", class_=\"item-container\")\n    return containers<\/code><\/pre>\n\n\n\n<p>The next function in the project is <code>checkPrices()<\/code>. This function loops through each <code>item-container<\/code> HTML tag returned by <code>getItems()<\/code> and manipulates the text in such a way that it can be compared to the specified price limit. If the product is under the price limit, information like price, name, and product page link is gathered about the product. The item is then added to a list to be returned.<\/p>\n\n\n\n<p>As I was testing, I noticed that an exception was thrown randomly when checking prices. There were sometimes one or two <code>item-container<\/code> entities with different tags (seemed to be promoted products or ads), so that caused issues. I solved that problem by simply catching the exception when it was thrown.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def checkPrices(items):\n    deals = &#91;]\n    for item in items:\n        try:\n            stringPrice = item.find(\"li\", class_=\"price-current\").find(\"strong\").text\n            intPrice = stringPrice.replace(\",\", \"\")\n            intPrice = int(intPrice)\n            if(intPrice &lt;= priceLimit):\n                nameTag = item.find(\"div\", class_=\"item-info\").find(\"a\", class_=\"item-title\")\n                itemName = nameTag.text\n                itemLink = nameTag.get(\"href\")\n                dealItem = {\n                    \"price\": \"$\" + stringPrice,\n                    \"name\": itemName,\n                    \"href\": itemLink\n                }\n                deals.append(dealItem)\n        except:\n            pass\n    return deals<\/code><\/pre>\n\n\n\n<p>The third function in this project is <code>notify()<\/code>. It does exactly what it says it does: it sends a notification to whatever desktop environment you&#8217;re in (<a href=\"https:\/\/i.imgur.com\/5MAkPRl.jpg\" data-type=\"URL\" data-id=\"https:\/\/i.imgur.com\/5MAkPRl.jpg\">Windows example<\/a>) (<a href=\"https:\/\/i.imgur.com\/ys1wQ1b.jpg\" data-type=\"URL\" data-id=\"https:\/\/i.imgur.com\/ys1wQ1b.jpg\">GNOME example<\/a>) and plays an audio file so you&#8217;ll be sure to notice it. The <a href=\"https:\/\/github.com\/rosspayn3\/newegg-tracker\/blob\/main\/Discovery.mp3\">Discovery.mp3 file<\/a> in the project repository is what I used. It&#8217;s short enough and sounds nice. <\/p>\n\n\n\n<p><strong>Note: <\/strong>make sure you&#8217;ve followed the setup steps in the README on the GitHub repository, as that&#8217;s where the notification sound you want to use is defined.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def notify():\n    notification = Notify()\n    notification.title = \"Found 3060 Ti in stock!\"\n    notification.message = \"Check your terminal!\"\n    notification.send()\n    playsound(config.notificationSoundFile)<\/code><\/pre>\n\n\n\n<p>The last function the bot uses is <code>sendEmail()<\/code>. Another pretty self-explanatory function name (which should always be the goal!). Again, make sure you&#8217;ve followed the setup steps in the GitHub repository as this function relies on a <code>smtpConfig.py<\/code> file that contains the bot&#8217;s email address and password as well as the email address you would like it to send email notifications to.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def sendEmail(subject, msg):\n    message = MIMEMultipart(\"alternative\")\n    message&#91;\"To\"] = smtpConfig.recipientEmail\n    message&#91;\"From\"] = smtpConfig.senderEmail\n    message&#91;\"Subject\"] = subject\n\n    htmlMsg = f' &lt;html&gt; &lt;body&gt; &lt;p&gt; {msg} &lt;\/p&gt; &lt;\/body&gt; &lt;\/html&gt; '\n\n    message.attach(MIMEText(msg, 'plain'))\n    message.attach(MIMEText(htmlMsg, 'html'))\n\n    smtp = smtplib.SMTP(\"smtp.gmail.com\", 587)\n    smtp.starttls()\n    smtp.login(smtpConfig.senderEmail, smtpConfig.senderPassword)\n    smtp.sendmail(smtpConfig.senderEmail, smtpConfig.recipientEmail, message.as_string())\n    smtp.quit()<\/code><\/pre>\n\n\n\n<p>Lastly, we have the main loop of the bot that checks prices and sends notifications and emails if necessary. The <code>notify()<\/code> and <code>sendEmail()<\/code> functions can be started in their own threads to keep the loop on a consistent timer (defined in the project setup). The main logic is as follows:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Get the products from the Newegg search page<\/li><li>Check their prices and add them to a list if they match our price limit<\/li><li>If at least one deal was found, loop through the list and add each of them to a message to be printed to the console as well as a message to be sent in an email.<\/li><li>Send the desktop notification and email.<\/li><li>Wait for a specified number of seconds before the next check. (Waiting longer between checks should reduce the likelihood of you getting IP banned either temporarily or permanently. Something to keep in mind. I&#8217;ve run this bot for a couple days non-stop with a 15 second sleep timer and it seems to be fine.)<\/li><\/ol>\n\n\n\n<p>For simplicity&#8217;s sake, I&#8217;ll give a cut-down example of the loop here. Feel free to check out the full code in <a href=\"https:\/\/github.com\/rosspayn3\/newegg-tracker\" data-type=\"URL\" data-id=\"https:\/\/github.com\/rosspayn3\/newegg-tracker\">the project repository<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>while True:\n    try:\n        items = getItems()\n        deals = checkPrices(items)\n\n        if len(deals) &gt; 0:\n            date = datetime.now()\n            timestamp = date.strftime(\"%d-%b-%Y (%H:%M:%S)\")\n            msg = f\"&lt;h1&gt;Found something for you!&lt;\/h1&gt;&lt;h3&gt;{timestamp}&lt;\/h3&gt;\"\n            for deal in deals:\n                consoleMessage += f\"{deal&#91;'price']}\\n{deal&#91;'name']&#91;:75]}\\n{deal&#91;'href']}\\n\\n\"\n                msg += f\"Price:&lt;\/strong&gt; {deal&#91;'price']}&lt;br&gt;&lt;strong&gt;Product&lt;\/strong&gt;: {deal&#91;'name']&#91;:75]}&lt;br&gt;&lt;strong&gt;Link&lt;\/strong&gt;: {deal&#91;'href']}&lt;br&gt;&lt;br&gt;\"\n            print(consoleMessage)\n            notify()\n            sendEmail(\"Found a deal!\", msg)\n        sleep(config.sleepDuration)\n    except KeyboardInterrupt:\n        sys.exit(\"\\n\ud83d\uded1 Price checker killed.\")<\/code><\/pre>\n\n\n\n<p>And that&#8217;s it! These code blocks are all you need to get started tracking prices on Newegg and get notified when products are in stock!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"future-improvements\">Future Improvements<\/h2>\n\n\n\n<p>There&#8217;s definitely room for improvement here. It&#8217;s a little annoying to have to mess with config files. One thing that could be done is to prompt for the necessary information the first time the script is started, then write that information to a file to be used later. Another improvement would be to implement OAuth authentication so that the &#8220;Less secure app access&#8221; setting in the Google account settings can be kept turned off. <\/p>\n\n\n\n<p>We&#8217;ll see how much time I feel like spending on improving this bot. I may get distracted and move on to other projects. Until then, happy hunting and good luck!<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Graphics card pricing is ridiculous and it&#8217;s hard to find a good deal these days. Let&#8217;s try to fix that.<a href=\"https:\/\/rpayne.dev\/blog\/how-to-make-a-bot-to-track-newegg-gpu-prices-and-get-notifications\/\" class=\"more-link\"><span class=\"more-button\">Continue reading<span class=\"screen-reader-text\">How-To: Make a Bot to Track Newegg GPU Prices and Get Notifications<\/span><\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"A project that I finished recently. Hope this can help somebody!","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[10,26],"tags":[31,32,29,27,28,30,17],"class_list":["post-288","post","type-post","status-publish","format-standard","hentry","category-how-to","category-programming","tag-automation","tag-bot","tag-gpu","tag-python","tag-script","tag-tracker","tag-web"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":373,"url":"https:\/\/rpayne.dev\/blog\/updates-job-and-hobbies\/","url_meta":{"origin":288,"position":0},"title":"Updates: Job and hobbies","author":"Ross Payne","date":"October 11, 2022","format":false,"excerpt":"Life updates including a new job, new game, and new toy","rel":"","context":"In &quot;Career&quot;","block_context":{"text":"Career","link":"https:\/\/rpayne.dev\/blog\/category\/career\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/rpayne.dev\/blog\/wp-content\/uploads\/2022\/10\/social-sharing-image.png?resize=1400%2C800&ssl=1 4x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/posts\/288","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/comments?post=288"}],"version-history":[{"count":48,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/posts\/288\/revisions"}],"predecessor-version":[{"id":374,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/posts\/288\/revisions\/374"}],"wp:attachment":[{"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/media?parent=288"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/categories?post=288"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rpayne.dev\/blog\/wp-json\/wp\/v2\/tags?post=288"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}