- Only the first products are parsed
- Notifications not sent because only available products are listed and
the very first discovery doesn't send a notification to avoid false
positives
- Products stay available forever unless there is a retention period
configured because unavailable products are not listed
Signed-off-by: Julien Riou <julien@riou.xyz>
Send notifications only if the state duration is not instant. There are
a lot of replies with 0 seconds duration spamming channels.
Signed-off-by: Julien Riou <julien@riou.xyz>
* add price range filter (#26)
* auto-remove stale products (#30)
* create new thread when product is available again (#28)
Signed-off-by: Julien Riou <julien@riou.xyz>
To avoid scalpers' price, the bot now understand filters on prices
using a minimum and maximum value, in a currency and a pattern to
detect the model.
Example:
```
"price_ranges": [
{"model": "3090", "max": 3000, "currency": "EUR"},
{"model": "3080", "max": 1600, "currency": "EUR"},
{"model": "3070", "max": 1200, "currency": "EUR"}
],
```
More details in README.md.
Signed-off-by: Julien Riou <julien@riou.xyz>
Filters are now separate structures to include a product or not based
on their own set of properties. For now, include and exclude filters
are supported. They take a regex as an argument and include a product
if the regex matches (or doesn't match) the product name. This commit
will allow us to create new filters on product like on a price range.
Signed-off-by: Julien Riou <julien@riou.xyz>
Instead of spamming a twitter thread with generic replies confusing the
community because the original message was posted long time ago, the bot now
creates a new thread with all product information and an incrementing counter
for uniqueness.
This commit adds a hash attribute to help identify duplicate messages. Tweets
have a TweetID attribute for the initial thread identifier and a LastTweetID
attribute to keep track of the last reply to eventually continue the thread if a
duplicate is detected.
Signed-off-by: Julien Riou <julien@riou.xyz>
When workers threshold was reached, the current parser was skipped instead of
being processed later. Adding a for loop to retry instead.
Signed-off-by: Julien Riou <julien@riou.xyz>
- Add PostgreSQL support (#23)
- Keep track and remove stale products
- Select or create shop before parsing
- Remove auto increment from primary key
- Enable replies on Twitter and Telegram
Signed-off-by: Julien Riou <julien@riou.xyz>
Products not updated since a while are not supposed to stay in the database nor
exposed via the API. This commit automatically updates all detected products
with the current date and adds a "retention" flag (number of days) to remove old
products. This flag is disabled by default.
Signed-off-by: Julien Riou <julien@riou.xyz>
A shop map was created to group URLs by shops and process them in order. Now
that we have Amazon and each URL can be parsed independently, there is no need
to group them anymore. Moreover, shops were passed as an argument to the
handleProducts function. Shop name can be deduced by the parser itself. The
parser has a reference to the database. The parser now select or create the shop
before parsing products.
Signed-off-by: Julien Riou <julien@riou.xyz>
By default, when a product is available, a notification is sent. When that same
product is not available, a reply is sent to the original message. With tons of
notifications, replies might be seen as flooding. This commit adds an option to
explicitly enable replies on Twitter and Telegram notifiers. By default, reply
messages are disabled.
Signed-off-by: Julien Riou <julien@riou.xyz>
This commit introduces the Amazon support with calls to the Product Advertising
API (PA API). For now, I was only able to use the "www.amazon.fr" marketplace.
I will add more marketplaces when my Amazon Associate accounts will be
validated.
Signed-off-by: Julien Riou <julien@riou.xyz>
Add `-api` mode to start the HTTP API with the following routes:
- /health
- /shops
- /shops/:id
- /products
- /products/:id
Signed-off-by: Julien Riou <julien@riou.xyz>
- Rename "Parser" to "URLParser"
- Make "Parse" function generic
- Rename "crawlShop" function to "handleProducts"
- Reduce "handleProducts" footprint a little bit
Signed-off-by: Julien Riou <julien@riou.xyz>
