Commit Graph

28 Commits

Author SHA1 Message Date
alex f323dc2f41 processing: rewritten processing pipeline 2021-01-16 22:09:06 +03:00
alex 65947896b8 docker: Added main docker-compose config 2021-01-16 22:08:33 +03:00
alex 908996d863 settings: Add path for last_id file 2021-01-16 22:08:17 +03:00
alex 7d8df2761d scrape: Fix for saving last id 2021-01-16 22:07:44 +03:00
alex 3b15ef41f4 scrape: Use RabbitMQ for scraping results 2021-01-16 21:58:08 +03:00
alex 0647efe18f rabbit: Improve task sending 2021-01-16 21:53:04 +03:00
alex ba5706e5e3 settings: Add queue for repositories 2021-01-16 21:52:22 +03:00
alex 97fb017e7e feat: Added progress for jsl importing 2021-01-16 21:51:14 +03:00
alex d0741baf44 feat: Add remover for old tasks 2021-01-07 22:39:44 +03:00
alex 21ee304071 feat: Add import of scraped github data 2021-01-07 22:39:10 +03:00
alex 33def73159 feat: Add github scraper 2021-01-07 22:38:35 +03:00
alex 143f0a566f dev: pre-commit config 2021-01-07 22:34:27 +03:00
alex 164400efec dev: Simple contributing guide 2021-01-07 22:34:08 +03:00
alex dc58136c34 db: Add new index 2021-01-07 22:33:00 +03:00
alex 885e17754b auto: black run 2021-01-07 22:32:36 +03:00
alex af06978cc9 feat: multiprocessing for RabbitMQ worker
Helps with very long tasks
2021-01-04 18:22:35 +03:00
alex 4b035964d6 settings: Increased message TTL to 24 hours 2021-01-03 23:35:31 +03:00
alex 13d8911220 rabbit: Fixed auto_ack with generator consumer 2021-01-03 23:35:04 +03:00
alex 668e4b5044 docker: Improved context ignoring 2021-01-03 16:52:55 +03:00
alex d06167070d rabbit: small qos + acks for reliable messages 2021-01-03 16:32:19 +03:00
alex ca37d31e18 feat: initial versions of core components 2021-01-03 15:59:33 +03:00
alex 519374cc06 feat: optipng worker 2021-01-03 15:58:17 +03:00
alex 6ce3a307cc docker: compose for assets workers 2021-01-03 15:58:00 +03:00
alex 15f3a3c6c8 docker: Add libmagic and optipng to Dockerfile 2021-01-03 15:57:41 +03:00
alex 884bc0aa97 feat: draft version 2021-01-03 15:10:47 +03:00
alex f003ed21cd git: Improved ignores 2021-01-03 15:08:21 +03:00
alex 24b36e4ea3 git: Updated .gitignore 2021-01-03 15:06:47 +03:00
alex 7a0b207503 Initial commit 2021-01-03 13:42:12 +02:00