Products not updated since a while are not supposed to stay in the database nor
exposed via the API. This commit automatically updates all detected products
with the current date and adds a "retention" flag (number of days) to remove old
products. This flag is disabled by default.
Signed-off-by: Julien Riou <julien@riou.xyz>
A shop map was created to group URLs by shops and process them in order. Now
that we have Amazon and each URL can be parsed independently, there is no need
to group them anymore. Moreover, shops were passed as an argument to the
handleProducts function. Shop name can be deduced by the parser itself. The
parser has a reference to the database. The parser now select or create the shop
before parsing products.
Signed-off-by: Julien Riou <julien@riou.xyz>
By default, when a product is available, a notification is sent. When that same
product is not available, a reply is sent to the original message. With tons of
notifications, replies might be seen as flooding. This commit adds an option to
explicitly enable replies on Twitter and Telegram notifiers. By default, reply
messages are disabled.
Signed-off-by: Julien Riou <julien@riou.xyz>
This commit introduces the Amazon support with calls to the Product Advertising
API (PA API). For now, I was only able to use the "www.amazon.fr" marketplace.
I will add more marketplaces when my Amazon Associate accounts will be
validated.
Signed-off-by: Julien Riou <julien@riou.xyz>
Add `-api` mode to start the HTTP API with the following routes:
- /health
- /shops
- /shops/:id
- /products
- /products/:id
Signed-off-by: Julien Riou <julien@riou.xyz>
- Rename "Parser" to "URLParser"
- Make "Parse" function generic
- Rename "crawlShop" function to "handleProducts"
- Reduce "handleProducts" footprint a little bit
Signed-off-by: Julien Riou <julien@riou.xyz>
- Add newegg support (#14)
- Truncate too long product names for Twitter
- Configure headless browser address
- Bugfix include and exclude regexes
Signed-off-by: Julien Riou <julien@riou.xyz>
Add `browser_address` configuration setting to define where is the headless
browser instead of relying on the default value.
Signed-off-by: Julien Riou <julien@riou.xyz>
Features:
- Add Micro Center support (#2)
- Replace strings with regexp for Twitter hashtags
Bugfixes:
- Use array of key/value for Twitter hashtags (#1)
Cleanup:
- Remove useless compileRegex function
Signed-off-by: Julien Riou <julien@riou.xyz>
Local stores are set with an "storeid" param in the query string of the URL
and by a "storeSelected" cookie to avoid garbage in the query strings for
further requests. As product URL is a unique key in the database, Micro Center
is able to handle an URL with the store ID for every product. We can add this
storeid in the list of URLs to parse and job done. Every single Micro Center
local store are parsable.
Signed-off-by: Julien Riou <julien@riou.xyz>
As a good start, only the "shippable items" are parsed. Next enhancement would
be to configure local shops.
Signed-off-by: Julien Riou <julien@riou.xyz>
With Go, maps are not ordered the way they are declared. Keys can be read at any
order. When a pattern is too large ("rtx 3060"), when placed first, it can match
a name of another product ("rtx 3060 ti"). When placed second, the good hashtag
is chosen. This commit uses an array of maps because arrays are ordered.
Signed-off-by: Julien Riou <julien@riou.xyz>
- new language: go
- new shops: cybertek.fr, mediamarkt.ch
- deprecated shops: alternate.be, minershop.eu
- improved database transaction management
- better web parsing library (ferret, requires headless chrome browser)
- include or exclude products by applying regex on their names
- check for PID file to avoid running the bot twice
- hastags are now configurable
Signed-off-by: Julien Riou <julien@riou.xyz>
