diff --git a/app/tag-data.json b/app/tag-data.json index 759df61..efaa5ce 100644 --- a/app/tag-data.json +++ b/app/tag-data.json @@ -1,17 +1 @@ -{ - "github": 1, - "guide": 4, - "next-js": 1, - "visual-studio-code": 1, - "tailwind-nextjs-starter-blog": 1, - "cygnus": 1, - "self-hosted": 1, - "server": 1, - "disney": 1, - "art": 1, - "store": 1, - "melis-sweetsimple": 1, - "python": 1, - "projects": 1, - "code": 1 -} +{"art":1,"store":1,"melis-sweetsimple":1,"visual-studio-code":1,"guide":4,"tailwind-nextjs-starter-blog":1,"disney":1,"github":1,"python":1,"projects":1,"code":1,"cygnus":1,"self-hosted":1,"server":1,"next-js":1} \ No newline at end of file diff --git a/data/blog/qbitmaid.mdx b/data/blog/qbitmaid.mdx index 2c8b469..cec2b80 100644 --- a/data/blog/qbitmaid.mdx +++ b/data/blog/qbitmaid.mdx @@ -1,8 +1,8 @@ --- title: Qbitmaid date: '2024-11-3' -lastmod: '2024-11-14' -tags: ['python', 'projects', 'code'] +lastmod: '2025-1-28' +tags: ['python', 'projects', 'code', 'docker', 'unraid'] draft: true summary: How I used python to keep my torrents in check layout: PostBanner @@ -10,13 +10,15 @@ images: ['https://s3.jonb.io/cdn/projects/qbitmaid.jpg'] --- ### qbit-maid + Development [^1] of qbitmaid was over the course of several months. At first, the project was called qbit-clean and didn't have all the features the project has now. The issue was mainly with my download cache in unraid being filled with torrents I no longer needed to seed[^2]. When I would get a notification from the server that the download cache was 95% full I would have to manually go to [qbittorrent](https://www.qbittorrent.org/), sort the torrents by age and remove the ones older than two weeks avoiding torrents I wanted to keep. -![qbittorrent](https://www.qbittorrent.org/img/qb_banner.webp) +[![qbittorrent](https://www.qbittorrent.org/img/qb_banner.webp)](https://www.qbittorrent.org/) -This was tedious. Very tedious. So I went off to do more work just to avoid a little. +This was tedious. Very tedious. So I went off to do more work just to avoid a little.`qbitmaid.py` is the main file that glues the project together. This was my first project where I heavily abstracted the design. More on this later. First, we'll connect to the API. I used an existing client [library](https://pypi.org/project/qbittorrent-api/2022.5.32/) that makes this process simpler than writing your own client. `qbitmaid.py`[^3] + ```python ... class Qbt: @@ -31,38 +33,157 @@ class Qbt: except qbittorrentapi.APIError as e: self.tl.exception(e) self.po.send_message(e, title="qbit-maid API ERROR") - # Pulling all torrent data - self.torrent_list = self.qbt_client.torrents_info() - #Main process block - if self.use_log: - list_qbit_api_info(self) - list_first_tor(self) - debug_torrent_list(self) - build_tor_list(self) - process_counts(self) - if self.use_log: - torrent_count(self) - tor_processor(self) - if self.use_log: - print_processor(self) - if self.delete_torrents: - tor_delete(self) - self.et = datetime.datetime.now() - get_script_runtime(self) - if self.use_pushover: - tor_notify_summary(self) - if self.use_apprise: - tor_notify_apprise(self, r, apprise_notify) - if self.use_healthcheck: - send_ping(self, r, self.healthcheck_url) -# Run -if __name__== "__main__": - Qbt() + ... ``` -This is the main file that glues the project together. This was my first project where I heavily +Then we use the api to make an list of the torrents: + +```python + # Pulling all torrent data + self.torrent_list = self.qbt_client.torrents_info() +``` + +Next, we "sift" out torrents to be deleted. This was created with a positive sieve meaning we specify positive scenarios. In other words, I know which torrents I want to keep as opposed to the torrents I don't want. Theres pros and cons to both scenarios however in the long term a positive sieve is less work. + +> `qlist.py` has functions at the bottom of the file that are referenced in the conditions. This very method of programming made it easy to write unit tests as I went. + +```python +if is_preme(torrent['seeding_time'], self.min_age): + continue +``` + +```python +def is_preme(seeding_time, minage): + if seeding_time <= minage: + return True +``` + +When it comes across an item that meets certain criteria it will skip it. For instance, the example above checks to see if it's too soon to remove a torrent. This is because some [trackers][private tracker] require a minimum seed time. If you were to remove a torrent sooner than they require, it could lead to getting kicked. + +`qlist.py` Has a couple jobs: + +- Tag torrents according to how they should be treated. +- Sort + +`qlist.py` + +```python +... +def build_tor_list(self): + while self.torrent_list: + ... + if is_tracker_blank(torrent['tracker']): + ... + continue + elif is_cat_ignored(torrent['category'], self.cat_whitelist.values()): + ... + continue + elif is_ignored_tag(self.ignored_tags.values(),torrent['tags']): + ... + continue + if is_tag_blank(torrent['tags']): + ... + if is_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()): + self.qbt_client.torrents_add_tags(self.tracker_protected_tag,torrent['hash']) + elif is_not_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()): + self.qbt_client.torrents_add_tags(self.tracker_non_protected_tag,torrent['hash']) + if is_preme(torrent['seeding_time'], self.min_age): + continue + elif is_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()): + if is_tag_blank(torrent['tags']): + self.qbt_client.torrents_add_tags(self.tracker_protected_tag,torrent['hash']) + ... + self.tracker_list.append(torrent) + elif is_not_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()): + if is_tag_blank(torrent['tags']): + self.qbt_client.torrents_add_tags(self.tracker_non_protected_tag,torrent['hash']) + ... + self.tracker_list.append(torrent) +``` + +In this case the items I want to keep stays. `qlist.py` passes the data over to `qprocess.py`. This is done through about 2 layers of abstraction. Unfortunatley, this type of programming makes it difficult to follow. + +`qprocess.py` has four jobs: + +- Collect telemetry +- Delete torrents if needed +- Enable debugging if needed + +`qprocess.py` + +```python +def tor_processor(self): + """Main logic to sort through both self.tracker_nonprotected_list and self.tracker_protected_list + If torrent meets criteria for deletion, its infohash_v1 will be appended to self.torrent_hash_delete_list + """ + for canidate in self.tracker_list: + if self.enable_telemetry: + header = ['state','ratio','tags','added','hash','name','tracker'] + row = [canidate['state'],canidate['ratio'],canidate["tags"],canidate['added_on'],canidate['infohash_v1'],canidate["name"][0:20],canidate['tracker']] + write_csv(self.cv,self.telemetry_outfile,header,row) + ... + elif is_protected_over_ratio(canidate['ratio'], 1.05, self.tracker_protected_tag, canidate["tags"]): + if self.use_log: + self.tl.debug(f'["{canidate["name"][0:20]}..."] is above a 1.05 ratio({canidate["ratio"]}).') + self.torrent_hash_delete_list.append(canidate['infohash_v1']) + ... + elif is_not_protected_tor(self.tracker_non_protected_tag, canidate["tags"]): + self.torrent_hash_delete_list.append(canidate['infohash_v1']) + ... + else: + if self.enable_dragnet: + header = ['state','ratio','tags','added','thash','tname','trname'] + row = [canidate['state'],canidate['ratio'],canidate["tags"],canidate['added_on'],canidate['infohash_v1'],canidate["name"][0:20],canidate['tracker']] + write_csv(self.cv,self.dragnet_outfile,header,row) + continue +``` + +I package this in a docker file: + +```Dockerfile +FROM python:alpine3.18 +WORKDIR / +COPY . opt +RUN apk add --no-cache supercronic +RUN pip install requests +RUN pip install qbittorrent-api +RUN chmod +x /opt/entrypoint.sh +CMD ["/opt/entrypoint.sh"] +``` + +Then use [Drone][drone] to package this into a container. This pushes the container to an [OCI repo][oci] in gitea. The application is configured through a toml file: + +```toml +[qbittorrent] +host = "192.168.x.x" +port = 8080 +username = "user" +password = "pass" + +... + +[healthcheck] +use_healthcheck = true +healthcheck_url = "https://example.com/ping/>" +``` + +Using unraid has honestly been a delight. I had some performance issues but that was due to how I was using the storage pool. + +![unraid](https://s3.jonb.io/cdn/blog/qbitmaid/unraid.png) + +Finally, this same container will run the test cases in `test_qbitmaid.py`. This is handled by drone. So eachtime I push new code to a development branch on gitea, it creates a container to test and tests the code. Once I see that it has passed, I can merge the code to the main branch. + +## Final Notes + +I have been using this for over 2 years. It was a huge learning experience and my coding practices have evolved over my newer projects. While I did make this for my use mainly, feel free to try it out! If you have any questions, you can open an issue [here](https://git.jonb.io/jblu/qbit-maid/issues/new). + [^1]: The Source Code can be found [here][source-code]. + [^2]: Private trackers require you to seed a torrent for a period of time. In my case, I have to seed for about 2 weeks or to a ratio of 1. + [^3]: Code has been removed for examples in this article. -[source-code]: https://git.jonb.io/jblu/qbit-maid \ No newline at end of file +[source-code]: https://git.jonb.io/jblu/qbit-maid +[private tracker]: https://en.wikipedia.org/wiki/BitTorrent_tracker#Private_trackers +[drone]: https://www.drone.io/ +[oci]: https://git.jonb.io/jblu/-/packages/container/qbit-maid/latest