302 lines
11 KiB
Markdown
302 lines
11 KiB
Markdown
# END OF LIFE NOTICE
|
|
|
|
Due to lack of time to do the maintenance, this repository is now archived.
|
|
Don't hesitate to fork it!
|
|
|
|
# RestockBot
|
|
|
|
Year 2020 has been quite hard for hardware supply. Graphics cards are out of stock everywhere. Nobody can grab the new generation (AMD RX 6000 series, NVIDIA GeForce RTX 3000 series). Even older generations are hard to find. `RestockBot` is a bot that crawl retailers websites and notify when a product is available.
|
|
|
|
## Requirements
|
|
|
|
### Headless browser
|
|
|
|
Use Docker:
|
|
|
|
```
|
|
docker run --name chromium --rm -d -p 9222:9222 montferret/chromium
|
|
```
|
|
|
|
Or get inspired by the [source code](https://github.com/MontFerret/chromium) to run it on your own.
|
|
|
|
### Amazon (optional)
|
|
|
|
To access the [Product Advertising API](https://webservices.amazon.com/paapi5/documentation/) and start to notify for Amazon products, you will need to have a valid [Amazon Associates](https://affiliate-program.amazon.com) account in the [Marketplace](https://github.com/spiegel-im-spiegel/pa-api/blob/v0.9.0/marketplace.go#L36) of your choice. You will then be able to retreive your **partner tag**, and the **Marketplace name** obviously.
|
|
|
|
Once your account has been validated, you can request access to the Product Advertising API (PA API) to retreive your **access key** and your **secret key**.
|
|
|
|
Ensure you follow the **terms of services** before subscribing to the Amazon Associates program and use the PA API.
|
|
|
|
### Twitter (optional)
|
|
|
|
Follow [this procedure](https://github.com/jouir/twitter-login) to generate all the required settings:
|
|
* `consumer_key`
|
|
* `consumer_secret`
|
|
* `access_token`
|
|
* `access_token_secret`
|
|
|
|
### Telegram (optional)
|
|
|
|
Follow [this procedure](https://core.telegram.org/bots#3-how-do-i-create-a-bot) to create a bot `token`.
|
|
|
|
Then you have two possible destinations to send messages:
|
|
* channel using a `channel_name` (string)
|
|
* chat using a `chat_id` (integer)
|
|
|
|
For testing purpose, you should store the token in a variable for next sections:
|
|
```
|
|
read -s TOKEN
|
|
```
|
|
|
|
#### Chat
|
|
|
|
To get the chat identifier, you can send a message to your bot then read messages using the API:
|
|
|
|
```
|
|
curl -s -XGET "https://api.telegram.org/bot${TOKEN}/getUpdates" | jq -r ".result[].message.chat.id"
|
|
```
|
|
|
|
You can test to send messages to a chat with:
|
|
|
|
```
|
|
read CHAT_ID
|
|
curl -s -XGET "https://api.telegram.org/bot${TOKEN}/sendMessage?chat_id=${CHAT_ID}&text=hello" | jq
|
|
```
|
|
|
|
#### Channel
|
|
|
|
Public channel names can be used (example: `@mychannel`). For private channels, you should use a `chat_id` instead.
|
|
|
|
You can test to send messages to a channel with:
|
|
|
|
```
|
|
read CHANNEL_NAME
|
|
curl -s -XGET "https://api.telegram.org/bot${TOKEN}/sendMessage?chat_id=${CHANNEL_NAME}&text=hello" | jq
|
|
```
|
|
|
|
Don't forget to prefix the channel name with an `@`.
|
|
|
|
### Database
|
|
|
|
|
|
#### SQLite (default)
|
|
|
|
Default database driver is SQLite using the `restockbot.db` file.
|
|
|
|
#### PostgreSQL
|
|
|
|
To configure a PostgreSQL database, you can use Docker:
|
|
|
|
```
|
|
cp -p docker/postgresql.env.example docker/postgresql.env
|
|
docker-compose -f docker-compose-postgresql.yml up -d
|
|
```
|
|
|
|
Then see the configuration section to define the database configuration.
|
|
|
|
#### MySQL
|
|
|
|
To configure a MySQL database, you can use Docker:
|
|
|
|
```
|
|
cp -p docker/mysql.env.example docker/mysql.env
|
|
docker-compose -f docker-compose-mysql.yml up -d
|
|
```
|
|
|
|
Then see the configuration section to define the database configuration.
|
|
|
|
## Compilation
|
|
|
|
### With pre-built binaries
|
|
|
|
Download the latest [release](https://github.com/jouir/restockbot/releases).
|
|
|
|
Ensure checksums are identical.
|
|
|
|
### With make
|
|
|
|
Clone the repository:
|
|
```
|
|
git clone https://github.com/jouir/restockbot.git
|
|
```
|
|
|
|
Build the `restockbot` binary:
|
|
```
|
|
make build
|
|
ls -l bin/restockbot
|
|
```
|
|
|
|
Build with the architecture in the binary name:
|
|
|
|
```
|
|
make release
|
|
```
|
|
|
|
Eventually remove produced binaries with:
|
|
|
|
```
|
|
make clean
|
|
```
|
|
|
|
### With Docker
|
|
|
|
```
|
|
docker image build -t restockbot:$(cat VERSION) .
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Default file is `restockbot.json` in the current directory. The file name can be passed with the `-config` argument.
|
|
|
|
Options:
|
|
|
|
* `database` (optional)
|
|
* `type`: driver to use (`sqlite`, `postgres`, `mysql`)
|
|
* `dsn`: data source name (see [documentation](https://gorm.io/docs/connecting_to_the_database.html))
|
|
* `urls` (optional): list of retailers web pages
|
|
* `amazon` (optional)
|
|
* `searches`: list of keywords to search for (ex: `["nvidia rtx", "amd rx"]`)
|
|
* `access_key`: access key to access the [Product Advertising API](https://webservices.amazon.com/paapi5/documentation/)
|
|
* `secret_key`: secret key to access the [Product Advertising API](https://webservices.amazon.com/paapi5/documentation/)
|
|
* `marketplaces`: list of documents containing a Marketplace `name` and a `partner_tag` (ex: `{"marketplaces":[{"name": "www.amazon.com", "partner_tag": "mytag-01"}]}`)
|
|
* `amazon_fulfilled`: include only products packaged by Amazon
|
|
* `amazon_merchant`: include only products sold by Amazon
|
|
* `affiliate_links`: generate affiliate links with the partner tag
|
|
* `nvidia_fe` (optional)
|
|
* `locations`: list of NVIDIA stores (ex `["es", "fr", "it"]`)
|
|
* `gpus`: list of models (ex: `["RTX 3060 Ti", "RTX 3070"]`)
|
|
* `user_agent`: user agent to simulate a real web browser (ex: `Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0`)
|
|
* `timeout`: maximum time before closing the request (optional)
|
|
* `twitter` (optional):
|
|
* `consumer_key`: API key of your Twitter application
|
|
* `consumer_secret`: API secret of your Twitter application
|
|
* `access_token`: authentication token generated for your Twitter account
|
|
* `access_token_secret`: authentication token secret generated for your Twitter account
|
|
* `hashtags`: list of key/value used to append hashtags to each tweet. Key is the pattern to match in the product name, value is the string to append to the tweet. For example, `{"twitter": {"hashtags": [{"rtx 3090": "#nvidia #rtx3090"}]}}` will detect `rtx 3090` to append `#nvidia #rtx3090` at the end of the tweet.
|
|
* `enable_replies`: reply to original message when product is not available anymore
|
|
* `retention`: number of days to keep tweet references in the database (not deleted by default)
|
|
* `telegram` (optional):
|
|
* `channel_name`: send message to a channel (ex: `@channel`)
|
|
* `chat_id`: send message to a chat (ex: `1234`)
|
|
* `token`: key returned by BotFather
|
|
* `enable_replies`: reply to original message when product is not available anymore
|
|
* `include_regex` (optional): include products with a name matching this regexp
|
|
* `exclude_regex` (optional): exclude products with a name matching this regexp
|
|
* `price_ranges` (optional): define price ranges for products based on the model. List of rules containing `model` (regex to apply to the product name, string), `min` (minimum expected price, float), `max` (maximum expected price, float), `currency` (price currency used by the filter, string). For example `{"price_ranges":[{"model": "3090", "min": 0, "max": 3000, "currency": "EUR"}]}`
|
|
* `browser_address` (optional): set headless browser address (ex: `http://127.0.0.1:9222`)
|
|
* `api` (optional):
|
|
* `address`: listen address for the REST API (ex: `127.0.0.1:8000`)
|
|
* `cert_file` (optional): use SSL and use this certificate file
|
|
* `key_file` (optional): use SSL and use this key file
|
|
|
|
## Usage
|
|
|
|
### With binary
|
|
|
|
```
|
|
restockbot -help
|
|
```
|
|
|
|
### With Docker
|
|
|
|
```
|
|
docker run -it --name restockbot --rm --link chromium:chromium -v $(pwd):/root/ restockbot:$(cat VERSION) restockbot -help
|
|
```
|
|
|
|
## Execution modes
|
|
|
|
There are two modes:
|
|
* **default**: without special argument, the bot parses websites and manage its own database
|
|
* **API**: using the `-api` argument, the bot starts the HTTP API to expose data from the database
|
|
* **monitor**: using the `-monitor` (optionaly with `-monitor-warning-timeout` and `-monitor-critical-timeout` arguments), the bot checks for last execution times per shop to return a Nagios compatible output
|
|
|
|
## How to contribute
|
|
|
|
Lint the code with pre-commit:
|
|
|
|
```
|
|
docker run -it -v $(pwd):/mnt/ --rm golang:latest bash
|
|
go get -u golang.org/x/lint/golint
|
|
apt-get update && apt-get upgrade -y && apt-get install -y git python3-pip
|
|
pip3 install pre-commit
|
|
cd /mnt
|
|
pre-commit run --all-files
|
|
```
|
|
|
|
## How to parse a shop
|
|
|
|
### Create the Ferret query
|
|
|
|
`RestockBot` uses [Ferret](https://github.com/MontFerret/ferret) and its FQL (Ferret Query Language) to parse websites. The full documentation is available [here](https://www.montferret.dev/docs/introduction/). Once installed, this library can be used as a CLI command or embedded in the application. To create the query, we can use the CLI for fast iterations, then we'll integrate the query in `RestockBot` later.
|
|
|
|
```
|
|
vim shop.fql
|
|
ferret --cdp http://127.0.0.1:9222 -time shop.fql
|
|
```
|
|
|
|
The query must return a list of products in JSON format with the following elements:
|
|
* `name`: string
|
|
* `url`: string
|
|
* `price`: float
|
|
* `price_currency`: string
|
|
* `available`: boolean
|
|
|
|
Example:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"available": false,
|
|
"name": "Zotac GeForce RTX 3070 AMP Holo",
|
|
"price": 799.99,
|
|
"price_currency": "EUR",
|
|
"url": "https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_wgfx_pcie_puis_ref_est_in20007322.html"
|
|
},
|
|
{
|
|
"available": false,
|
|
"name": "Asus GeForce RTX 3070 DUAL 8G",
|
|
"price": 739.99,
|
|
"price_currency": "EUR",
|
|
"url": "https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_wgfx_pcie_puis_ref_est_in20005540.html"
|
|
},
|
|
{
|
|
"available": false,
|
|
"name": "Palit GeForce RTX 3070 GamingPro OC",
|
|
"price": 819.99,
|
|
"price_currency": "EUR",
|
|
"url": "https://www.topachat.com/pages/detail2_cat_est_micro_puis_rubrique_est_wgfx_pcie_puis_ref_est_in20005819.html"
|
|
}
|
|
]
|
|
```
|
|
|
|
`RestockBot` will convert this JSON to a list of `Product`.
|
|
|
|
### Embed the query
|
|
|
|
Shops are configured as a list of URLs:
|
|
|
|
```json
|
|
{
|
|
"urls": [
|
|
"https://www.topachat.com/pages/produits_cat_est_micro_puis_rubrique_est_wgfx_pcie_puis_f_est_58-11447,11445,11446,11559,11558.html",
|
|
"https://www.ldlc.com/informatique/pieces-informatique/carte-graphique-interne/c4684/+fv121-19183,19184,19185,19339,19340.html",
|
|
"https://www.materiel.net/carte-graphique/l426/+fv121-19183,19184,19185,19339,19340/"
|
|
]
|
|
}
|
|
```
|
|
|
|
The `Parse` function ([parser.go](parser.go)) will be called. In this example, the following **shop names** will be deduced: `topachat.com`, `ldlc.com` and `materiel.net`.
|
|
|
|
Each shop should implement a function to create a ferret query based on an URL:
|
|
* `func createQueryForLDLC(url string) string`
|
|
* `func createQueryForMaterielNet(url string) string`
|
|
* `func createQueryForTopachat(url string) string`
|
|
* ...
|
|
|
|
This function should be added to the switch of the `createQuery` function ([parser.go](parser.go)).
|
|
|
|
Products will then be parsed.
|
|
|
|
## Disclaimer
|
|
|
|
Crawling a website should be used with caution. Please check with retailers if the bot respects the terms of use for their websites. Authors of the bot are not responsible of the bot usage.
|