Why multiple simultaneous downloads?

[WildeGeek]

A few times I've noticed in my logs that the same IP address has downloaded the same audio file as many as 20 times at the same time. If it was spread out over the day I might think, "firewall proxy" or "NAT", but these are all within about 2 seconds. The logs show that these are completed downloads (all bytes of the file were transferred).

I found this phenomenon slightly irritating but didn't think much of it until this morning, when someone did this to a big chunk of my back catalog overnight. One IP address generated 300+ downloads of only 9 files in the space of about 6 minutes.

This looks like the behavior of a BOT, but the user-agent says Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322). Which looks like a browser client.

1) Does anybody know why this is happening?
2) Does anybody know how to stop this bandwidth abuse?

[swingercast]

Perhaps its BITS?

Background Intelligent Transfer Service (BITS)

If so, it's not abuse but it screws your stats all up.

John

[WildeGeek]

Perhaps its BITS?

Background Intelligent Transfer Service (BITS)
I Googled this, and the first half-a-dozen articles said nothing about this protocol downloading the same file multiple times. It made it sound like it is just some background process that's aware of available bandwidth.

If so, it's not abuse but it screws your stats all up.
If they're downloading the same file, in its entirety, 1 to 2 dozen times, then I would call it abuse. That is what the logs indicate is happening.

[swingercast]

Here is what I understand about how it works. Remember, I'm a swinger and a podcaster (in that order so don't take my word for it.) :D

Say you have a 20MB file, if someone wants to download it and their browser uses BITS then the download gets split up into different requests. For example, the computer might determine that the 20MB file can be downloaded most efficiently if it is split into 10 2MB requests.

Hope this info is helpful.

Download the BITS.doc on the page linked here for a detailed explanation:

http://www.microsoft.com/windowsserver2003/techinfo/overview/bits.mspx

John

[Seuss]

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

Sounds like Instant Media. http://www.im.com Their old client used BITS and had generated with many podcasters concerns you have raised so their new client is BITS free but perhaps you have hits from users still running that old one.

[AGuyandaGirl]

I have been learning a bit more about this mysterious BITS myself, and here is the conversation that I recently had with a programmer at Microsoft.

Subject: RE: (Microsoft Team RSS Blog) : Background Intelligent Transfer Service and measuring downloads
Date: November 1, 2006 11:11:56 AM PST

Hi Joe,

BITS uses standard HTTP RANGE requests, which means in the server logs you will find multiple "hits" for the same URL from the same users, however you will notice that each request has a http range header, which allows you correlate the multiple hits into a single one for tracking purposes. In other words, this issue is not specific to BITS, but any client that uses HTTP RANGE requests.

I hope this helps,
-Walter

-----Original Message-----
Sent: Saturday, September 23, 2006 4:09 PM
To: Team RSS (External)
Subject: (Microsoft Team RSS Blog) : Background Intelligent Transfer Service and measuring downloads

"...For podcasters, the only way we can accurately measure our audience is via downloads. My understanding of BITS technology is a little bit like a torrent, in otherwords BITS helps conserve BW, but is somewhat meddlesome as all the tiny hits are counted as downloads.

My question is this: Is there a way that I can (or have my service provider) account for all these BITS and determine an accurate number of completed downloads? ..."

-Joe

I hope that helps pull back the curtain just a little.

Joe

[WildeGeek]

Subject: RE: (Microsoft Team RSS Blog) : Background Intelligent Transfer Service and measuring downloads
Date: November 1, 2006 11:11:56 AM PST

Hi Joe,

BITS uses standard HTTP RANGE requests, which means in the server logs you will find multiple "hits" for the same URL from the same users, however you will notice that each request has a http range header, which allows you correlate the multiple hits into a single one for tracking purposes. In other words, this issue is not specific to BITS, but any client that uses HTTP RANGE requests.

I hope this helps,
-Walter

So now we need to figure out how the get APACHE to report in its logs that there have been RANGE requests.

The strange thing is, that the logs report the number of bytes transferred in the transactions to be the entire length of the file. So if someone uses BITS and initiates 12 connections to get a file, and the file is 1MB, the APACHE logs say they transferred 12MB. Your ISP is probably using those same logs to determine the bandwidth you're using.

[AGuyandaGirl]

Vista is going to use BITS from what I understand, so perhaps we will see more traction on this soon. On another note, I have unlimited bandwidth, so the ISP can measure all they want, I don't care (as much). : b

Joe