jblu/jonbio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: jblu:6dcfad7f0daec1102c69f9461e72b2b3f3ae2843
Branches Tags
jblu:main
jblu:upstream
..
compare: jblu:79b1b2db403c95eaeafa2131dbd2d5c42e381656
Branches Tags
jblu:main
jblu:upstream

No commits in common. "6dcfad7f0daec1102c69f9461e72b2b3f3ae2843" and "79b1b2db403c95eaeafa2131dbd2d5c42e381656" have entirely different histories.
6dcfad7f0d ... 79b1b2db40

2 changed files with 49 additions and 154 deletions
Show Stats Expand all files Collapse all files

18
app/tag-data.json
View File

@ -1 +1,17 @@
{"art":1,"store":1,"melis-sweetsimple":1,"visual-studio-code":1,"guide":4,"tailwind-nextjs-starter-blog":1,"disney":1,"github":1,"python":1,"projects":1,"code":1,"cygnus":1,"self-hosted":1,"server":1,"next-js":1} {
"github": 1,
"guide": 4,
"next-js": 1,
"visual-studio-code": 1,
"tailwind-nextjs-starter-blog": 1,
"cygnus": 1,
"self-hosted": 1,
"server": 1,
"disney": 1,
"art": 1,
"store": 1,
"melis-sweetsimple": 1,
"python": 1,
"projects": 1,
"code": 1
}

183
data/blog/qbitmaid.mdx
View File

@ -1,24 +1,22 @@
--- ---
title: Qbitmaid title: Qbitmaid
date: '2024-11-3' date: '2024-11-3'
lastmod: '2025-1-28' lastmod: '2024-11-14'
tags: ['python', 'projects', 'code', 'docker', 'unraid'] tags: ['python', 'projects', 'code']
draft: false draft: true
summary: How I used python to keep my torrents in check summary: How I used python to keep my torrents in check
layout: PostBanner layout: PostBanner
images: ['https://s3.jonb.io/cdn/projects/qbitmaid.jpg'] images: ['https://s3.jonb.io/cdn/projects/qbitmaid.jpg']
--- ---
### qbit-maid ### qbit-maid
Development [^1] of qbitmaid was over the course of several months. At first, the project was called qbit-clean and didn't have all the features the project has now. The issue was mainly with my download cache in unraid being filled with torrents I no longer needed to seed[^2]. When I would get a notification from the server that the download cache was 95% full I would have to manually go to [qbittorrent](https://www.qbittorrent.org/), sort the torrents by age and remove the ones older than two weeks avoiding torrents I wanted to keep. Development [^1] of qbitmaid was over the course of several months. At first, the project was called qbit-clean and didn't have all the features the project has now. The issue was mainly with my download cache in unraid being filled with torrents I no longer needed to seed[^2]. When I would get a notification from the server that the download cache was 95% full I would have to manually go to [qbittorrent](https://www.qbittorrent.org/), sort the torrents by age and remove the ones older than two weeks avoiding torrents I wanted to keep.
[![qbittorrent](https://www.qbittorrent.org/img/qb_banner.webp)](https://www.qbittorrent.org/) ![qbittorrent](https://www.qbittorrent.org/img/qb_banner.webp)
This was tedious. Very tedious. So I went off to do more work just to avoid a little.`qbitmaid.py` is the main file that glues the project together. This was my first project where I heavily abstracted the design. More on this later. First, we'll connect to the API. I used an existing client [library](https://pypi.org/project/qbittorrent-api/2022.5.32/) that makes this process simpler than writing your own client. This was tedious. Very tedious. So I went off to do more work just to avoid a little.
`qbitmaid.py`[^3] `qbitmaid.py`[^3]
```python ```python
... ...
class Qbt: class Qbt:
@ -33,157 +31,38 @@ class Qbt:
except qbittorrentapi.APIError as e: except qbittorrentapi.APIError as e:
self.tl.exception(e) self.tl.exception(e)
self.po.send_message(e, title="qbit-maid API ERROR") self.po.send_message(e, title="qbit-maid API ERROR")
...
```
Then we use the api to make an list of the torrents:
```python
# Pulling all torrent data # Pulling all torrent data
self.torrent_list = self.qbt_client.torrents_info() self.torrent_list = self.qbt_client.torrents_info()
#Main process block
if self.use_log:
list_qbit_api_info(self)
list_first_tor(self)
debug_torrent_list(self)
build_tor_list(self)
process_counts(self)
if self.use_log:
torrent_count(self)
tor_processor(self)
if self.use_log:
print_processor(self)
if self.delete_torrents:
tor_delete(self)
self.et = datetime.datetime.now()
get_script_runtime(self)
if self.use_pushover:
tor_notify_summary(self)
if self.use_apprise:
tor_notify_apprise(self, r, apprise_notify)
if self.use_healthcheck:
send_ping(self, r, self.healthcheck_url)
# Run
if __name__== "__main__":
Qbt()
``` ```
Next, we "sift" out torrents to be deleted. This was created with a positive sieve meaning we specify positive scenarios. In other words, I know which torrents I want to keep as opposed to the torrents I don't want. Theres pros and cons to both scenarios however in the long term a positive sieve is less work. This is the main file that glues the project together. This was my first project where I heavily
> `qlist.py` has functions at the bottom of the file that are referenced in the conditions. This very method of programming made it easy to write unit tests as I went.
```python
if is_preme(torrent['seeding_time'], self.min_age):
continue
```
```python
def is_preme(seeding_time, minage):
if seeding_time <= minage:
return True
```
When it comes across an item that meets certain criteria it will skip it. For instance, the example above checks to see if it's too soon to remove a torrent. This is because some [trackers][private tracker] require a minimum seed time. If you were to remove a torrent sooner than they require, it could lead to getting kicked.
`qlist.py` Has a couple jobs:
- Tag torrents according to how they should be treated.
- Sort
`qlist.py`
```python
...
def build_tor_list(self):
while self.torrent_list:
...
if is_tracker_blank(torrent['tracker']):
...
continue
elif is_cat_ignored(torrent['category'], self.cat_whitelist.values()):
...
continue
elif is_ignored_tag(self.ignored_tags.values(),torrent['tags']):
...
continue
if is_tag_blank(torrent['tags']):
...
if is_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()):
self.qbt_client.torrents_add_tags(self.tracker_protected_tag,torrent['hash'])
elif is_not_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()):
self.qbt_client.torrents_add_tags(self.tracker_non_protected_tag,torrent['hash'])
if is_preme(torrent['seeding_time'], self.min_age):
continue
elif is_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()):
if is_tag_blank(torrent['tags']):
self.qbt_client.torrents_add_tags(self.tracker_protected_tag,torrent['hash'])
...
self.tracker_list.append(torrent)
elif is_not_protected_tracker(torrent['tracker'], self.tracker_whitelist.values()):
if is_tag_blank(torrent['tags']):
self.qbt_client.torrents_add_tags(self.tracker_non_protected_tag,torrent['hash'])
...
self.tracker_list.append(torrent)
```
In this case the items I want to keep stays. `qlist.py` passes the data over to `qprocess.py`. This is done through about 2 layers of abstraction. Unfortunatley, this type of programming makes it difficult to follow.
`qprocess.py` has four jobs:
- Collect telemetry
- Delete torrents if needed
- Enable debugging if needed
`qprocess.py`
```python
def tor_processor(self):
"""Main logic to sort through both self.tracker_nonprotected_list and self.tracker_protected_list
If torrent meets criteria for deletion, its infohash_v1 will be appended to self.torrent_hash_delete_list
"""
for canidate in self.tracker_list:
if self.enable_telemetry:
header = ['state','ratio','tags','added','hash','name','tracker']
row = [canidate['state'],canidate['ratio'],canidate["tags"],canidate['added_on'],canidate['infohash_v1'],canidate["name"][0:20],canidate['tracker']]
write_csv(self.cv,self.telemetry_outfile,header,row)
...
elif is_protected_over_ratio(canidate['ratio'], 1.05, self.tracker_protected_tag, canidate["tags"]):
if self.use_log:
self.tl.debug(f'["{canidate["name"][0:20]}..."] is above a 1.05 ratio({canidate["ratio"]}).')
self.torrent_hash_delete_list.append(canidate['infohash_v1'])
...
elif is_not_protected_tor(self.tracker_non_protected_tag, canidate["tags"]):
self.torrent_hash_delete_list.append(canidate['infohash_v1'])
...
else:
if self.enable_dragnet:
header = ['state','ratio','tags','added','thash','tname','trname']
row = [canidate['state'],canidate['ratio'],canidate["tags"],canidate['added_on'],canidate['infohash_v1'],canidate["name"][0:20],canidate['tracker']]
write_csv(self.cv,self.dragnet_outfile,header,row)
continue
```
I package this in a docker file:
```Dockerfile
FROM python:alpine3.18
WORKDIR /
COPY . opt
RUN apk add --no-cache supercronic
RUN pip install requests
RUN pip install qbittorrent-api
RUN chmod +x /opt/entrypoint.sh
CMD ["/opt/entrypoint.sh"]
```
Then use [Drone][drone] to package this into a container. This pushes the container to an [OCI repo][oci] in gitea. The application is configured through a toml file:
```toml
[qbittorrent]
host = "192.168.x.x"
port = 8080
username = "user"
password = "pass"
...
[healthcheck]
use_healthcheck = true
healthcheck_url = "https://example.com/ping/<uuid>>"
```
Using unraid has honestly been a delight. I had some performance issues but that was due to how I was using the storage pool.
![unraid](https://s3.jonb.io/cdn/blog/qbitmaid/unraid.png)
Finally, this same container will run the test cases in `test_qbitmaid.py`. This is handled by drone. So eachtime I push new code to a development branch on gitea, it creates a container to test and tests the code. Once I see that it has passed, I can merge the code to the main branch.
## Final Notes
I have been using this for over 2 years. It was a huge learning experience and my coding practices have evolved over my newer projects. While I did make this for my use mainly, feel free to try it out! If you have any questions, you can open an issue [here](https://git.jonb.io/jblu/qbit-maid/issues/new).
[^1]: The Source Code can be found [here][source-code]. [^1]: The Source Code can be found [here][source-code].
[^2]: Private trackers require you to seed a torrent for a period of time. In my case, I have to seed for about 2 weeks or to a ratio of 1. [^2]: Private trackers require you to seed a torrent for a period of time. In my case, I have to seed for about 2 weeks or to a ratio of 1.
[^3]: Code has been removed for examples in this article. [^3]: Code has been removed for examples in this article.
[source-code]: https://git.jonb.io/jblu/qbit-maid [source-code]: https://git.jonb.io/jblu/qbit-maid
[private tracker]: https://en.wikipedia.org/wiki/BitTorrent_tracker#Private_trackers
[drone]: https://www.drone.io/
[oci]: https://git.jonb.io/jblu/-/packages/container/qbit-maid/latest