Few things

[mjr]

In compact mode, the string is a series of 6 bytes, ip and port. This is fine for now, but IPv6 is an eventuality. How should the protocol be upgraded to account for the larger IP size? compact6 instead of compact?

Also, how about an ability for a client to communicate to a tracker how much it downloaded/uploaded to others based on peer id? Could be something as simple as <peerid>=amount. If the program downloads from a peer, amount would be negative. Otherwise, positive. A tracker could send a response indicating that it supports the feature during the start event so that the client doesn't needlessly record the stats, nor send them.

Lastly, there *really* needs to be a protocol change for a client program to report to the tracker that a peer is sending bogus data.

[DreadWingKnight]

http://bittorrent.org/ipv6_tracker_extension.html

Also, how about an ability for a client to communicate to a tracker how much it downloaded/uploaded to others based on peer id?

Massive needless bloat would be added to tracker requests.

Lastly, there *really* needs to be a protocol change for a client program to report to the tracker that a peer is sending bogus data.

Why?

[jewelisheaven]

Trackers don't care about peer behaviour... they track peers. Nowai? Peer counts, and connection information. Why give them more overhead when most trackers are already overloaded.

[mjr]

First, an initial thanks for the heads up on the IPv6 info. Now, on to my responses.

Why give them more overhead when all they need is an IP and port? Surely, they don't need upload, download, useragent, client version (hidden in peer id), unique ids (key field), peer id, warning message, or failure reason.

Massive needless bloat, as it has been called, is already in tracker requests. An average compact reply is half the size of the request.

To the 'why' on reporting bad peers: Perhaps, maybe, just maybe trackers would rather not have people messing with data. If you look at it another way, bad peers take up space in tracker database, and eat up bw with announces. A minimum valid announce request contains (not exclusively) an infohash, a peer id, an ip, and a port, which are 77 bytes minimum, each time the bad peer announces. A tracker will probably store all of this information. An announce reply will contain 6 bytes for the peer if they're using compact, 36 bytes if not. That wasted bandwidth and database space, for a peer that is contributing essentially nothing, adds up over time.

Telling the tracker how much is transferred per peer will be an issue for some smaller sites, thus it should be optional. The start event would cause the tracker to indicate if the feature is supported. Such a feature will help trackers to order peers based on transfer speed, so that users spend less time downloading.

Our tracker, which gets over 3000 queries per second, is more than capable of handling such features. We'd much rather look to the future than have our head in the sand.

[The8472]

That wasted bandwidth and database space, for a peer that is contributing essentially nothing, adds up over time.

and you want that fixed by having peers send MORE data? interesting line of reasoning.

Surely, they don't need upload, download, useragent, client version (hidden in peer id), unique ids (key field), peer id, warning message, or failure reason.

yes, that's why the opentracker guys advocate the use of UDP-announces for example

Telling the tracker how much is transferred per peer will be an issue for some smaller sites

Only for site administrators who do not understand how bittorrent works. Trackers have nothing to say how things work. They are not meant to be in control as the more control they have the more peers would depend on them... which results in a single point of failure among other things.

[mjr]

"Oh crap, peer -AZ2053xyz is sending me bad data" a few times by a few peers is nothing compared to weeks or months or even years of a peer constantly announcing and constantly sending bad data. Think for the long run, not the short term payoff.

Bittorrent was not designed to be completely decentralized. Otherwise, DHT/PEX would not have been an afterthought. Hell, the private bit (block DHT), which I believe was popularized by uTorrent, essentially forces the tracker to be a single point of failure.

[The8472]

haha, you're funny... if "a few times by a few peers" is enough to get a tracker ban clients then malicious clients could also notify the tracker and ban honest clients... you haven't even thought out your threat model.

And btw, peer IDs are far from reliable, since you're using Azureus in your example... Azureus allows you to use a different peer ID for tracker-client and peer-peer communication for example.

Rule #1 of distributed protocol design: Do not ever trust data reported by a client (neither as tracker nor as peer), only trust data you can measure yourself.

You (and also private tracker admins) blatantly ignoring that. And btw... either BitComet and Azureus were the first popular clients to adopt the private flags, µT was still in its infancy back then.

If you want to ban spamming clients, do it like the opentracker guys... and btw, bad piece sending clients are a very small minority, i don't see how this would be a bandwidth concern. Inefficiencies in honest clients are far more taxing.

[mjr]

Ah, the attack on the person instead of a counter. Great stuff.

Trust isn't really an issue when 30% or more of the peers are reporting one particular peer as being a problem. I'd like to know how we're ignoring data integrity. That's pretty much the entire reason for the initial post.

Stating that those clients were the first to use the private flag helps support my statement on how bittorrent isn't truly decentralized. I thank you for that.

The bandwidth use wasn't the reason I suggested the bad peer feature. I just used the bandwidth math to counter the ill conceived and oft used "that'll use too much bandwidth" argument.

I beg of you to stop bastardizing the meaning of "spam." When people equate "spam" with "crap I don't wanna see," we end up with spam filters having quality similar to Yahoo and Hotmail. Even AOL has gotten to the point of essentially asking senders if the mail is spam or not because users don't know the meaning of the word. However, I digress.

The problem with open tracker is that it's too open. There's no controls. There's no ability to automatically stop abuse. So, in essence, doing it like the opentracker guys would be to just ignore it and hope it eventually goes away. Again, it's not a bandwidth concern for the tracker. It's a bandwidth and general annoyance concern for the users. You are almost guaranteed to get bad pieces daily, with the frequency increasing based on the openness of the site, the amount of peers, and the legitimacy of the data. Believe it or not, there are a lot of ISPs that charge for the transfer, not the speed, and some users might not like paying for useless transfer.

[jewelisheaven]

... the problem with open tracker is that it's too open.

OpenTracker is being used by TPB for the same reason uT is used by many people who know what they're doing with and what they want from a bittorrent client. Small footprint, protocol adherance, full featureset, need I go on?

Hashfailed data is blocked in < 1 piece for all pieces larger than 128 KiB in uT 1.8. Additionally uT's rangeblock feature ensures clients which DO send bad data when aggregated are removed from the swarm. It would behoove anyone who experiences the hashfails on a consistent basis to invest in ipfilter.dat. This would especially apply to your theoretical case with a person on a quota wanting to minimize useless transfer. I hope you're now aware of the ipfilter feature of uT, if you weren't before. I'd like to reiterate your defensive arguments aren't very thought out. I get bad pieces in 1/10 swarms for copyrighted content. In all non-copyrighted content the last hashfail I got was in 2007. If you're going to use scenarios please at least use numbers which represent some metric. Due to the inconsistent nature of language and personal connotative meanings your babble gets lost in translation, as it were.

[The8472]

i agree with jewelisheaven, if all this is about bad pieces and not tracker traffic then we can forget the whole issue. µT 1.8 will implement Azureus' bad piece detection code, which has already proven itself to be very efficient and the data lost due to hash fails is usually in the sub-percent region, even in heavily polluted swarms (given a suitable choice of piece sizes relative to the torrent size).

So this entire topic is moot imo.

To reiterate my former statement: Trackers do not have to be "in control" all fairness-related problems can be solved on the client end. And another argument: Trackers dropping peers would do no good, they still could join via PEX/DHT

Client side solutions are more efficient and more robust and preserve openness, which is an important factor if you don't want a single point of failure.

[mjr]

It's great that uTorrent has IP filtering. I'm very well aware of that, as are probably most of its users. But this isn't a client subforum. It's a subforum specific to the protocol. Most programs don't support filtering. The reason I mentioned opentracker's openness is because if I, or anyone else, wanted to run a completely open tracker, one already exists. Its footprint is small because there are no controls at all. This is no good if you want to run a tracker for game updates, or legitimate distribution of copyrighted material. The protocol as it stands allows for controls, just not reporting bad data or transfer.

Here's the two goals: Control bad data, and control or gain stats on distribution. Let's discuss the merits of that instead of implementation.

[The8472]

Control bad data

Bad data only affects the client that receives it, if it's an issue for the user he can do something about it, no need to change the protocol

and control [...] distribution

Encrypt your content, only provide keys to authorized users. No need to change anything about the protocol

gain stats on distribution

Join a swarm with a client, use the data that the tracker collects anyway... those two options should cover most realistic needs for statistics, nothing that would have to be implemented in the protocol.

The baseline is that whatever you want can already be done... it's just that it does not match how you imagined to do it. Again, bittorrent is a more-or-less open system, with swarming behavior and distributed decision-making. Instead of trying to adapt it into centralized, control-freak-friendly system you could adapt to the existing system instead which already provides most things to some extent or another.

And btw, opentracker DOES ban spamming peers such as mediasentry, they came up with their own neat trick to identify that w/o even knowing anything about the .torrents (since they just have the infohashes) and their ban-lists match manually compiled lists (such as those used by TPB) quite well. And opentracker + tpb (which also runs on opentracker software) together stem about 13 million peers and yet you're talking about it as if it were a "failure" not be capable of solving some big problems that supposedly exist.

So please tell me a scenario that would be important to millions (or at least hundred thousands) of users that would warrant a change to the protocol itself instead of the user(s) coming up with a solution that's achievable with the current system.

[mjr]

The solutions currently available require much more work by a lot more people. Instead of a large chunk of users discovering a problem and then causing it to disappear, every last user has to discover it, and deal with it, while the problem persists. The stats could be obtained from the swarm, but that means the tracker has to connect to each swarm to obtain it. That's a lot of overhead. You're right in stating it's not a failure. It's a problem, though, that could be mitigated. Every problem should be nipped at the bud before it becomes a failure. Compact mode is a perfect example of that.

Forgive my ignorance, but I've gone through the opentracker code and couldn't find anything that resembles banning. Could you tell me which function handles it? If I'm wrong in that regard, I'd be (unhappily) willing to admit it.

Sure, ways to get the data already exist. But, it doesn't make much sense to go from LA to NY via Irkutsk.

[The8472]

every last user has to discover it

Not really... ever heard of "default settings"?

[mjr]

Default settings don't mean jack if they still have to download to discover a seeder is sending bogus data. It's still wasted time and wasted bandwidth.