Enhanced P2P Method ?

[GhostShip]

It seems some researchers have decided to release an idea to help speed up file transfers using a new system.

http://arstechnica.com/news.ars/post/20070410-accelerated-p2p-by-similarity-searches.html

P2P file sharing offers the possibility of dramatically increased download speeds by avoiding the problems associated with a single (overloaded) server.  But even P2P downloads can bog down when files are first seeded or if few active users have a copy of the file.  A research team with members at Carnegie Mellon, Purdue, and Intel thinks they've found a way around some of these limitations using what they call Similarity-Enhanced Transfer (SET), a technique they claim can speed up P2P downloads by anywhere from 30 to 70 percent.  They'll be presenting their technique at the 4th Symposium on Networked Systems Design and Implementation tomorrow. 

The "Similarity" portion of SET comes from the realization that many of the files being shared contain pieces of identical data. Examples include music files that differ only in terms of tags, movies or movie trailers that are dubbed in different languages, and updated versions of software. Like other P2P systems, SET divides large files into small segments. Once that process is complete, however, the SET software searches for similar files using a method called "handprinting," which is similar to the pattern matching techniques used to cluster search results or filter spam. Once similar files are identified, they are scanned for any individual chunks that are identical to pieces of the file being downloaded.

The presentation will come with actual implementation code, and the team hopes to see others put it to use. "This is a technique that I would like people to steal," said David Andersen of Carnegie Mellon, "Developers should just take the idea and use it in their own systems

This method while sounding useful looks to me like being a lemon as it requires the person looking for the file to have an expanded file of extra information to decided what does or does not match, currently a file is located by searching for a text string or hash match in most p2p systems.
Any network using it would need most likely need to change the protocol to accomodate this new method, not an easy option in the case of most existing networks. 

[Scyre]

Wasn't this the idea behind the OFF file sharing system about a year or two ago?

[GhostShip]

Now you mention it it does sound similar but its going to be hard to see this coming to fruit as while the concept sounds exciting on paper its imposing a wealth of extra load on the downloader, although on the positive it could open up the possibility of literally generating files from no where 

[Scyre]

Yeah, as I recall the OFFS (Owner Free Filing System) had the idea that many forms of data contained the same small pieces...

For instance, a small block of data in a picture may be identical to a block of data located within a video or maybe an archive. Since the blocks are the same in both, then they can be shared once for both files.

They furthered this idea by saying that the same block of data can represent ANY piece of data, depending on how it is XORed with another bloack of data (the decoder), so that a block can be used for any file whatsoever... It just has to have the right calculations done to it.

The decoder blocks would be stored on numerous different systems and no one would ever know exactly what decoder blocks would be on their system, as it would be part of a whole 'world collection' that noone actually owned, but everyone could access.

Each decoder block would also contain the pointer to the next decoder block somewhere else, perhaps halfway around the world.

Anyway, I will stop rambling lol, just thought that some would find that interesting.

Best Regards, and Be Good!
Scyre

[White Stripes]

just thought that some would find that interesting.

RIAA: "you infringed on our copyright!"

music fan: *"plays" XOR file, horrid static eminates from speakers* "uh.. no i didnt.. its just random data"

RIAA: *huddles to try to find another way to get blood out of turnip*

yeah.. i know thats not exactly a real-life situation but thats the 'image' i got in my head