Jump to content

Change order in queue for Seeding tasks


dlucre

Recommended Posts

Did some searching through the forum and didn't find anything specifically regarding Queue Order for Seeding tasks.

What I am looking for is the ability to prioritise when files start seeding.

In the same way that you can order 5 downloads 1-5, it would be nice to order 5 seeding tasks from -1 to -5 for example. At the moment, the seeding tasks seem to start up in whatever order they please, and the order is always

"*" whatever that means.

Anyway, hope I am making some sense and that you'll consider it.

Cheers

p.s. I'm new here. Just give me a slap if I've stepped out of line.

Link to comment
Share on other sites

I second dlucre's motion, and agree with firon's observation that the current queue scheduler certainly is not "very customizable". It has driven me nuts to such an extent that today I started looking around for other clients. Vuze certainly looks like it has a much better queueing algorithm but I was frankly shocked by the, excuse me, crap that was loaded on top of the original Azureus engine. I am back to utorrent and really would like the issue of long-term seeding to be addressed.

My complaint is this: I have removed any seeding target (ie my ratio=-1) because I don't understand why this is a relevant benchmark at all. What happens in practice is that heavily over-seeded torrents which are slow to reach the upload target (surprise, surpirse!) prevent underseeded torrents (which have long reached their seeding target with consummate ease) from being scheduled. If you download material regularly, but not even particularly heavily - say a stream every 4 days or so - then there is always likely to be enough new material below the seeding target to stop utorrent from ever serving up your back catalogue. This is not helpful to the BT community.

Certainly when the upload target ratio is set to (-1), I expect utorrent to work out a new strategy / offer me an alternative strategy to configue. A good start would be a simple prioritization by seed:peer ratio with the option to consider x number of peers = 1 seed. Right now, as I write this, most of my bandwidth is going to a torrent with seed:peer = 1412:612 - how insane in that?!? A swarm with 612 peers probably does not even need a seed...

At the same time, there are plenty of torrents in back catalogues languishing around in "queued seed" with seed:peer ratios of 8:4 and worse. These are the torrents that need my help! Yet short of going in and manually managing the queue, there is nothing sensible I can configure in utorrent.

There would also need to be some pre-emption. Right now it would seem to me that once a torrent is seeding and just so long as there are peers, no new torrent will ever be scheduled. In case of the popular torrent above, this means that it will never stop uploading.

PLEASE IMPROVE !!

And please, please - with sugar on top - don't go down the Vuze route and force me to look at some stupid promotional content... :-( Thank you!

Post Scriptum:

The above assumes that you do not want to make fundamental changes to utorrent's approach to seeding. However, I do not even understand why torrents are rotated in and out of active seeding. To my mind, utorrent should seed (that is, advertise as "available" to trackers) ALL torrents all of the time. And it should then simply assign bandwidth to incoming requests for pieces based on some seed:peer considerations (as before). At least in this fashion a swarm in urgent need of help will always find to me.

As the implementation stands, most of my torrents are never available to the community.

Link to comment
Share on other sites

?????

I just noticed the 1.8.3 Beta and thought I take a peek at the ChangeLog, in case I was missing out on some earth-shattering developments. Granted, there are some critical fixes for crashes but I am still trying to decide on my favourite between

- Change: support some older skins

and

- Change: lengthen "About" dialog.

It's a tough one. No, the reasons the scheduler's pushed back is because it ... works, doesn't it? Sort of, -ish...

Allow me to take the other side of this argument: the "crash when parsing some magnet uri components without values" affects the ONE dude who happened to have stumbled upon such a URI which he will be aware of immediately and will remove. The crash, however inconvenient to him and embarrassing to you, is of negligable significance to the wider p2p community.

The dismal performance of the queue scheduler on the other hand, while seemingly of little significance to the leecher in us, is the boomerang that ultimately impacts on all as it effectively starves the torrent pool of seeds.

The least that you could do (for the p2p community) is to sit down and write an honest manual page that tries to address the issue of "best seeding practice" given the limitations of the current implementation. In so doing, you will likely also stumble upon a better algorithm for future implementation.

I was in a bitch-fight the other day with a private tracker who was trying to get one over the public community by pointing to excessive leeching. Yet "excessive leeching" can hardly be the fault of users when half the client software is incapable of seeding effectively...

Link to comment
Share on other sites

... 1.9 changelog

1.8.3 version is, as you well should know, improvements to the 1.8 line of code.

Make docs.. sure many helpers here make them and refer to them, but the fact others need to be told where to find them shows what general user desire is to learn on their own. Why make docs when noone reads them. Did you know uT came with a manual for instance?

Link to comment
Share on other sites

Yes, I did. Read it cover to cover. Thrice. Good introduction to p2p technology for a (then) newbie like myself. As I said though, a little light on optimal seeding strategy. But then there may not be one. After toying with different configurations, I have now given up on utorrent's scheduler and am manually pausing torrents when I notice that I am providing bandwidth to swarms > 100 peers...

Link to comment
Share on other sites

I am still trying to decide on my favourite between

- Change: support some older skins

and

- Change: lengthen "About" dialog.

It's a tough one.

Oh, so picking insignificant changes and comparing them with feature requests gets you brownie points and strengthens your argument... how? uTP is arguably more important than queue ordering, as it can improve bandwidth utilization to maximize uploads, and it's a huge change to the technology underlying µTorrent, on track to become an IETF standard. You make it seem as if the devs are simply sitting on their laurels, twiddling their thumbs.

Link to comment
Share on other sites

I can't speak to the actual motivations of the developers, but if s/he has 15 minutes before lunch, a simple change can be finished. Recoding seed scheduling could take an entire weekend (if not more) for example.

ovonrein - this is kind of silly, but I'm at a point where I'm considering running two instances of µTorrent. One for downloading torrents, and one for long-term seeding. Long-term seeding will have 500 global connections, 250 active torrents each getting two connections max. It just seems to me, you can't have optimal settings for downloading torrents and long-term seeding in the same client configuration. If µTorrent did what you wanted, I wouldn't have to come up with such schemes.

Link to comment
Share on other sites

Thanks - I can see we (well, at least I am) getting somewhere.

Yes, jewelisheaven, I did actually read http://forum.utorrent.com/viewtopic.php?id=34259 and used it as a baseline for setting up my utorrent. I did not see the rant in http://forum.utorrent.com/viewtopic.php?id=57196 but it seems educational. The undercurrent in both documents is that the user of utorrent is technically minded. And from that follows, certainly in the piece by Rigmar Radio, that those of us running sub-optimal p2p configurations are somehow malicious. A similar accusation was levvied at me the other day by some private tracker. Chill, and consider Hanlon's Razor...

I am actually fairly technically minded, hence my willingness to read all these docs, fiddle with my config and engage in this forum. I also very much understand (and endorse) the basic premise of p2p technology: that to enjoy downloads, one must show willing to upload.

I think that my situation is fairly typical. I am on a (slow-ish) ADSL line with 8Mb down/0.5Mb up. That is, I am a natural born leecher: downloads are just swell; uploads are my problem. Particularly since my ISP also throttles p2p, given half a chance. Leave utorrent to do its thing - it has a preference for seeding the same torrents over extended periods of time - and bandwidth utilization falls as the ISP gets on top of my game. Rotate torrents regularly, by contrast, and my upload speeds will run at full potential.

This paragraph should explain, hermanm, why two or any higher number of instances of utorrent won't make a bit of difference: the queueing algorithm on the uploading instance will not improve and my ISP will do the rest.

I would like to recap, briefly, my grievances about the current scheduler:

(P1) The number of active torrents has a global cap. Marry this to the scheduler preference to stick with active torrents for as long as peers remain connected and my ISP's desire to throttle my traffic by observing my active connections and you get to a situation where after a short while I fall below my potential overall bandwidth for uploading.

(P2) Using target seed ratios as a primary criterion for seed selection is perverse. Virtually by definition, highly seeded material will be slow to seed up to target. And while not at target, it prevents poorly seeded material (which by definition has easily attained its target) from uploading. Yet it is precisely the latter material that needs the community's help.

(P3) Withholding most of the torrents from the p2p community most of the time (as in "queued seed") makes the "mating" process harder on poorly seeded material. Poorly seeded material very rarely gets scheduled anyway because of (P2), yet to meet a peer, he needs to look for a download at precisely the rare moment that I offer to upload.

I see the following remedies:

(R1a) Continue to activate torrents for as long as total bandwidth utilization is below configured potential. Instead of a maxTorrent#, allow me to configure a max bandwidth allocation per torrent. Since I should not be running more than 4 concurrent torrents, I would configure maxTorrentBW=maxBW/4.

(R1b) There should be a minTorrentBW below which the torrent is pre-empted (dropped, queued, whatever).

(R2) Provide an alternative selection algorithm based on peer:seed ratios. This should be activated when people set target upload ratio=-1 (disabled). I'd argue, in fact, that this should become utorrent's default config.

(R3) Consider seeding all torrents all of the time. Or consider seeding torrents with a seed:peer ratio worse than x all of the time. Rigmar takes a dim view on seeding in disproportion to bandwidth ("seeds-that-do-nothing"). I cannot understand the argument. However, assuming for a moment that his argument does hold some currency, then let the scheduler at least assure to rotate everything on a time-share basis.

(R1a+B) will see to it that my ISP is defeated by means of a constant rotation of active torrents. (R2) ensures that the treasure of available material in the p2p community is maximized/perpetuated. Finally, (R3) should allow peers to find a seed. It can then be left to the scheduler to determine which torrents to assign bandwidth to, in accordance with (R2). And, surely, the downloading client will be smart enough to skew requests for pieces towards the fastest uploaders, no?

Ultima, I was being polemic. As in politics, so in IT development: there's always the "little tweak" that one can easily make that gets the public's attention, and then there are the bigger building sites that, even when fixed, don't really stir the public's imagaination. To anyone non-technical - the majority of users, I suspect - it's the slickness of the UI that counts. However, as Rigmar alluded, the penetration of the p2p community by bittorrent/utorrent bestows a heavy responsibility on your development team to consider the impact of your product on the overall community.

Put it this way: when I see public AXXO torrents with seed:peer ratios of 17560:567, then this (1) clearly confutes the accusation (from private trackers) that the public is not seeding and (2) is a clear manifestation of (P2).

My two pennies worth...

Link to comment
Share on other sites

ovonrein - even if the seed/peer ratio is for an aXXo torrent, shouldn't you contribute what you downloaded at least? Then, once you've met your obligation, you can seed the other unpopular torrents?

> the queueing algorithm on the uploading instance will not improve and my ISP will do the rest.

The ISPs we use differ, so I can't really comment on your ISP's bandwidth management tactics. "Constant rotation" may help your specific situation, but its not clear to me if it would help or harm µTorrent users on other ISPs.

~~~

I am under the impression queue.prio_no_seeds and queue.use_seed_peer_ratio will handle the seeding queue. It doesn't work for you?

queue.prio_no_seeds: Enabling this option gives torrent jobs without seeds higher priority when seeding than other torrent jobs.

queue.use_seed_peer_ratio: When this option is enabled, µTorrent will determine the seeding queue order based on the ratio of the number of seeds to the number of peers connected in the swarm. The lower the seed:peer ratio is for a torrent job, the higher priority it will be given in the seeding queue. If a torrent job has 0 peers and queue.dont_count_slow_ul is disabled, it will be given the lowest priority. Otherwise, if the aforementioned option is enabled, the torrent job is treated as if there is 1 peer in the swarm.

Link to comment
Share on other sites

I realized after posting that my language could be misinterpreted, hermanm. Constant rotation would result from (R1b) in my case since my ISP appears to throttle down torrents after some time. The pre-emption would lead to rotation. If your ISP does not throttle, your torrents should never fall below my suggested minTorrentBW (and would thus not be preempted).

As regards AXXO, this is my whole point: I am a good p2p citizen not because I upload everything back to ratio 1 but because I am willing to seed that which you cannot get otherwise. What's the point of my becoming the 18,000st seeder for a torrent? Or even getting involved in a swarm of 600 peers? Better to focus my (paltry) bandwidth on swarms of only 6. I will seed the AXXO torrent again once seeds fall below 100 - my new (manually implemented) policy :D

---.---

Could it be, Switeck, that you meant your comment in reply to my post on http://forum.utorrent.com/viewtopic.php?id=57196? I cannot make sense of it in the context of this thread. My "Bandwidth" config is

global max# connect = 60

max# peer/torrent = 35

upload slot# = 4

If I read my screen right then it is rare for there to be more than 8 peers to be connected to any one torrent at any one time. Which I suppose makes the 35 pretty academic...

---.---

PS: hermanm, you will find that all the queue parameters that you identified are turned on by default, which is fine. And it makes no difference, regrettably.

Link to comment
Share on other sites

A Seeding Torrent Max connections limit would seem to affect the queue little yes...however my hidden point was faster seeding would decrease the need for complex seeding policies.

Any policy that continues to start new torrents until max download speed is reached is a quick disaster. Upload speed is too much of a limiting factor, so anyone downloading much faster than they're uploading for hours or days straight becomes increasingly unlikely to return the favor to others. This serves only to prematurely kill torrent swarms.

Likewise, only downloading 1 or 2 torrents at a time and getting stuck on 1-2 very slow downloading torrents is a potential disaster -- you may not get a chance to finish downloading the last torrent in your queue and your own upload is not "speeding up" the process much.

Your max upload speed may not be reached either if there's also a too-strict limit on seeding torrents at once.

Always making available the most needy torrent for seeding would be the ideal goal but determining which is "most needy" could require a bit of bandwidth if you had 100+ torrents spread between many trackers. Private torrents makes this even worse, because there's no peer exchange or DHT...forcing peers/seeds to find each other only through tracker updates. And the tracker may hand out only 1-50 ips of potential peers/seeds, some may be firewalled, disconnected, at max connections, or be peers/seeds you're already connected to. Since this typically only happens once every 30 minutes or hour, this can be a painfully long wait on torrents with lots of ip churn.

Link to comment
Share on other sites

Thank you again, Switek, for sharing your thoughts. I have replied to your observation on downloads slots in http://forum.utorrent.com/viewtopic.php?pid=408805#p408805. I would like to import your comments re upload slots from there into this thread:

"The problem is...you cannot currently set your max connections per torrent to only 10 without it affecting ALL your active downloading torrents too! They could end up connecting to 10 seeds, thus leaving you nobody to share with...or 10 very slow peers or seeds which give you little no matter how much you give them."

This reads to me as backing hermanm's suggestion that I should run more than one instance of utorrent. In fact, I could easily use Opera to do all my downloading (which it is very good at) and simply run utorrent to do the seeding (which Opera is awful at)?

I read somewhere how to install utorrent as a service. With the GUI thus unavailable, this implies that there must be a way to configure utorrent to automatically queue torrents. I could therefore configure Opera for all my users such that it will never upload and have it store all their downloads in a folder where utorrent will automatically upload from. Then I lower the max number of connections per torrent in utorrent so as to be optimal for uploading.

Does this strike you as a sensible strategy?

"Always making available the most needy torrent for seeding would be the ideal goal but determining which is "most needy" could require a bit of bandwidth if you had 100+ torrents spread between many trackers."

Do you refer to the 'scrapes'? Do they really require more than negligible bandwidth? If so, let the scheduler not be totally dumb about this: seed:peer ratios do not change much over short periods of time - in either direction. In other words, a torrent that currently runs at 17560:567 is unlikely to collapse to 10:3 within 1hr, and vise versa. That is, the scheduler can limit its scraping activity to 1 scrape every 5 secs for example (to be configured?). Even with 100+ torrents this is unlikely to signifcantly distort activity on the queue.

I have also rethought a bit my

"(R3) Consider seeding all torrents all of the time. Or consider seeding torrents with a seed:peer ratio worse than x all of the time."

I think that, if a short seeding queue is desirable (*), a better strategy is

(R3') Continuously seed all torrents with seed:peer ratios worse than the best ratio of the currently uploading seeds. The reason for modifying my suggested remedy in such a manner is that, if (R2) is implemented properly, then any incoming request for seeding on one of the idle active seeds with the poorest ratios will immediately prompt the scheduler to stop seeding the active torrent with the (relatively) best ratio, thereby shortening the queue of active seeds.

In fact, it seems that this behaviour results even in 1.8.2 from the setting of queue.slow_ul_threshold. Given that I set target_ratio=(-1), the seeding order is now determined by seed:peer ratios and idle active seeds with poor ratios fall below the threshold so that more seeds keep being added. And, I suppose, were one of those idle seeds to become active beyond the threshold, then utorrent should drop a better ratio'ed torrent...

(*) Does everybody here favour a short seeding queue because of some aspect of the p2p protocol? What is the reason for not seeding everything and being selective in who you respond to?

Link to comment
Share on other sites

True, scrapes don't take much bandwidth. But a torrent with 20 peers and 1 seed might be 20 peers all stuck on 18% and 1 very slow seed and availability of 118%...or 20 peers varying from 1% to 99% done with 1 very fast seed and availability of 900%. The first needs lots of help, the second is running just fine. This cannot be determined by just scrapes.

To find out more about the seeds/peers, you'd have to do tracker updates then connecting to them.

"Does everybody here favour a short seeding queue because of some aspect of the p2p protocol? What is the reason for not seeding everything and being selective in who you respond to?"

Long queues are simply bad. They represent a false illusion of availability to others just because you HAVE those torrents. Triage and/or Round-robin strategies to try to help as many people as possible when overloaded just means ever-increasing waits until the point where connections that are not always on (dial-up and some ADSL lines) might as well not participate.

Long queues are also trying to fit a general purpose "share everything" p2p model into a BitTorrent (download accelerator of specific files) framework. BitTorrent is sharing one file/torrent-to-many people topology. A long queue can be viewed as sharing many files/torrents to many people topology. You create a contention ratio between your files/torrents just like ISPs have contention ratios...the effective speed of each torrent gets forced lower and lower because they're all fighting over a limited resource (your upload.) Often this results in popular stuff crowding out rare.

Link to comment
Share on other sites

Reminds me of LimeWire. You queue files to download but they never start because you're in the back of the line. This is why I think a very low max peers per torrent is good. If your global max is 1,000 and you set your max peers to 5, you can actively seed 200 torrents.

Link to comment
Share on other sites

Forget Limewire, think Emule/Edonkey with file queues 1000's long!

"Your file has been queued for download, please wait a week for it to begin downloading!"

"If your global max is 1,000 and you set your max peers to 5, you can actively seed 200 torrents."

You seem to be on ComCast, so you neither have an upload fast enough to effectively seed 20 reasonably active torrents at once nor download fast enough to take advantage of downloading from 1000 peers/seeds at once. The extra bandwidth overheads for having that many connections won't help you or the other end. :(

Link to comment
Share on other sites

The server I would use for seeding would likely be from somewhere else and have much a higher upstream (e.g. 100 Mbps or 1,000 Mbps). If the sever had 100 Mbps upstream, then 200 active torrents with 5 peers max would be a reasonable setup?

Link to comment
Share on other sites

Yes, if you had 100 mbit/sec upload then 200 ACTIVE torrents at once are possible...but even that might be a little slow each if you had all 1000 connections at once. :P

It's ok to have more than 200 started torrents in that case so long as queueing only allows about 200 to be active at once. Due to various bottlenecks, keeping even a 10 mbit/sec upload line busy can be hard with 10 semi-active torrents.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...