Active versus Queued Triggering / Signalling

[CrazyDiamond]

Most times I have up to 25 torrents in my 'queuing' window. However, of course, at any given time only a very small number (4/5) are active downloading / seeding, and the rest are in the "queued" or "queued seed" state.

Where can I read about what triggers a torrent to go from one state to another? I.e:

- Go from 'queued' to 'download', and vice-versa.

- Go from 'queued seed' to 'seeding', and vice-versa.

Is it some form of signalling within the BT protocol to the tracker(s) and/or peers, or is it completely internal to the client program?

Again, where can I find some information on it?

[Firon]

Your preferences dictate how many active torrents are started. Preferences -> Queueing.

You can read the manual if you wanna know more about how ut works. It's quite thorough.

[CrazyDiamond]

Where in the manual does it specifically talk about the triggers for queuing state change?

[Ultima]

The queue is not something spec-defined (that is, it's internal to the application). How the client decides to move a torrent from queued to started depends on various factors (exactly what criteria I'm not completely certain). Some of the things that affect queue states are the Preferences > Advanced > queue.* options (described in the manual).

[CrazyDiamond]

Okay, so there is no active messaging / signalling between client program and tracker. Queue state changes are strickly determine by client program logic and various 'metrics' so to say.

Maybe if torrent tracker did signalling with client programs, better swarm 'management' could occur. For example at a given moment in time the tracker could tell my client to take this download and place it in the queue state, and take another torrent download and switch it to download (active) state, because the swarm for that cloud now can provide better resources thus resulting in better downloads. As my client downloads more and more chunks, tracker signalling would also do the same thing, and switch my sessions to provide the best resources to the swarms that need it more.

Such capabilities would require the server to become much bigger and robust to track all those PCs, but it could be possible. A Cisco CallManager (now called Unified Commuincations Manager) state-machine cluster can track 30,000 VoIP phones, and state-machine clusters can be intra-clusters for even greater scalability.

Just a thought.

[Ultima]

Each torrent swarm is a completely separate/disjoint network onto itself. Each torrent can have different trackers, or even multiple trackers. There are clients that don't even have concepts of torrent queues, like BitTornado, so it wouldn't make sense for a tracker to impose queues on clients (even hints). And honestly, queues shouldn't be controlled from a centralized tracker to begin with; the entire idea just doesn't sit right with me.

[CrazyDiamond]

"And honestly, queues shouldn't be controlled from a centralized tracker to begin with"

I'm not sure I was thinking a centralized tracker would (totally) control a client and its queues. The past couple days I have been trying something. I want my downloads and those stuck in queue state. I pick one stuck in "queued" state and move it up to a higher order. Sometimes I get a great result.....awesome download speed. So when compared to my bandwidth download capabilities I am getting crappy performance for the active torrents, yet there was a queued torrent that could of gotten great performance but the opportunity was missed cause it was stuck in queue state. The same is true for my queued seeds. I move one so that it becomes active and it starts uploading at a rate significantly higher than the other seeds combined. Had I not done that manual move, some people out there would not have gotten those downloads from me.

Some form of signaling might help 'match-up' clients within swarms for a win-win benefit..i.e seeders and leechers. My other thought is that certains swarms would die quicker because they were 'unhealthy' to start with.

Does that make sense?

[Ultima]

That's luck-of-the-draw; you just happened to connect to a fast peer (or several fast peers). It's not guaranteed to happen every single time, and there's no way for anyone (including the tracker) to determine any possible chances that you'll connect to a fast peer. For all you know, the active torrents might have as likely a chance for you to connect to a fast peer, but you just happened to miss it. No tracker hints would help in picking out fast torrents. Judging based on seed/peer counts (the most common metrics by which users incorrectly extrapolate speed projections) don't work either.

If there is no centralized tracker to compare your torrents to see which ones "might" give you good speeds, how would you expect the hints to be of any use? They certainly wouldn't be relative priorities if the tracker doesn't have information on all your torrents, and because (as I mentioned earlier) each torrent is disjoint (use different trackers), there is no way to get relative hints anyway. If you don't have relative hints, then every single time a tracker says "here, start this torrent," it might knock a possibly-faster torrent off the active list and into queued.

[Switeck]

Even fast peers might have max connections and refuse you...so even IF the tracker knew who would be fast peers to you, that wouldn't be enough information!

[CrazyDiamond]

I'm reading back and I guess I didn't explain good enough. Through signalling (robust signalling obviously), the tracker and the clients would exchange information on the current health of the swarm. The health of the swarm is obviously the collective health of each peer in the swarm. Each peer would provide signalling to the tracker (and maybe to peers afterwards) regarding its current metrics. So as a downloader my peer would learn which peers to avoid because their connections are maxed out. Other peers who have bandwidth remaining would become more favorable to my client. As an uploader the same would be true. If I have bandwidth available to provide better uploads then my peer becomes more favorable to other downloaders. Once my bandwidth is utilized then I become less favorable to other peers. As peers disconnect from me and my bandwidth becomes 'freed up', my peer become more more favorable in the swarm again. From the point of multiple torrents, my peer would be able to use its intelligence through signalling to make decisions as to which torrents to use and which to avoid at given moments in time. If torrent/swarm "A" is healthier while as the same time torrent/swarm "B" is a piece of crap, then my client will establish (more) connections with peers in swarm "A" and avoid (or reduce) connections to peers in swarm "B". If the torrent/swarm health is bad enough, my client state switches to (or stays in) the queued state. If the torrent/swarm health is good enough, my client torrent state switches to (or stays in) the active state. Maybe the concept of torrent states being active or queued would be come obsoleted since the client can now make decisions based on the health of each peer in a torrent/swarm.

In a nutshell, I was trying to say that through robust signaling peers would intelligently learn about the availability of resources of other peers and thus 'luck of the draw' type logic would be greatly reduced or even eliminated. The tracker itself may not make decisions, it just becomes a signalling relayer. Maybe it does make some decisions, I don't know, but that would be a deeper dive protocol discussion...for guys like Bram.

[Ultima]

Client-tracker connections aren't persistent.

Are you saying that that needs to change so that trackers can continuously pump signals out to peers? That's a ton of overhead, and trackers probably wouldn't be able to handle having that many connections open simultaneously.

If you're not saying that connections should be persistent, then the only other way would be for peers to continuously poll the tracker to emulate real-time updates on a peer's available bandwidth, because stale data is pretty useless for what you seem to want. That'd generate an unbelievable amount of overhead as well. Yet, this would possibly still fall into the luck-of-the-draw situation -- just because you know a peer told the tracker that it has bandwidth or free slots doesn't mean someone else didn't get to the peer first in between the time the peer reported to the tracker and the time you received the information from the tracker.

Even periodic polls (every announce or whatever) would cause a lot of overhead, considering the sheer number of peers potentially on each swarm (and thus, the amount of data that would need to be sent each poll could become rather large).

Edit: Note that I'm not necessarily fighting the idea itself, but the drawbacks associated with it, and perhaps its limited efficacy.

[CrazyDiamond]

Regarding whether connections are persistent or not, how the actual mechanics would be or not, ....I am not saying or suggesting anything. I'll let the protocol engineers think that out. What I am saying is two things:

(1) Signalling is (assuming the protocol engineers do a great job of it) a very efficient way of communicating lots of information using very little actual information to Tx / Rx (transmit / receive). It can also be very quick and rapid.

(2) There are in production today computer and logic systems that are signalling and/or state-machine based. The overhead of the signalling technology is actually minor, and for the 'server at the center of it' state-machine technology today can also be very efficient.

You're right that by the time my peers learns that a peer has available resources and my peers goes to connect to it, that said peer could already be 'in use'. I had already thought of that, which is partially why I said "....would provide signalling....(and maybe to peers afterwards) regarding its current metrics."

Also I am not saying I want the BT protocol, and associated torrent clients to evolve to work this way. I am saying based on my observations, what I have read, and feedback via this forum, that I wonder if BT could evolve into a more robust protocol using signaling technology.

Edit: I note your edit.

[Switeck]

Typical tracker updates are once every 30 minutes or an hour. The torrent swarm structure would look NOTHING like it did in that amount of time.

Trackers are already overloading from the current levels of signaling. Adding more they have to collate and communicate...probably won't help matters.

Peer Exchange is more suited to this task, but even that is only sent from what I understand once every 5 minutes.

Many peers/seeds have upload speeds of 10 KiloBYTES/second or less. Quite possibly, this is either all their connection is capable of...or due to other activities, all their spare bandwidth.

When dealing with monster torrents that have 50,000+ peers/seeds, passing all relevant information between peers can become a nightmare. What you're proposing sounds like it should help such torrents the most...yet I fear it would be those that are hurt the most.

[CrazyDiamond]

When the discussion is about the evolution of something then the "typical" can be often (but not necessarily always) scrapped to make way for a new way of doing something.

Regarding trackers being already overloaded with their current levels if signalling.....and I'm not saying this next comment is applicable, but it could be. H.323 protocol is a brutally poor protocol in how it sets up sessions and how much resources and time is required. It works, buts its an inefficient clunker. SIP on the other hand is a very well defined protocol, and has a very well defined message set, and as such it requires very little resources to maintain messaging between peers and/or proxy. (Note, while all forms of signalling is messaging/communications, not all forms of messaging/communications is signalling.) I know of VoIP clusters that maintain 30,000 VoIP phones. Guess what the signalling bandwidth is for each phone? 800bps (that's bits (not bytes) per second). Guess how many setup and teardowns there are for sessions in a cluster of this size.....lots n lots n lots. Yet the 'tracker' that does all the processing runs on Linux and runs on a 'lower end' HP appliance server.

Regarding the 10KBps upload for many peers....you're absolutely right. The signalling overhead would have to be very efficient so that torrent users get the most 'bang for their bit'. Losing most of your bandwidth to overhead would be defeating.