Designing a new file transfer method for MX

[sIrMoOn]

As anyone who's developed for it knows, the basic set of protocols used by MX-compatible clients is somewhat outdated.  As time goes on, we will be forced to upgrade some of these protocols if the network is to remain relevant.  As far as I can tell, the most common cry of alarm around here is "oh no IPv6" - an issue we will have to address at some point, but one I want to put off for now.  No, what I want to talk about is file transfer.

I have identified some "nice to have" features for a new file transfer protocol that I plan to add to my work-in-progress network client.  As these are matters that affect everyone, I figured I would solicit some input from every knowledgeable person who might wish to offer it.

My list to date:

* 64-bit file sizes - this would increase the maximum file size from the current 2 gigabytes to something like 9 petabytes
* support for out-of-order transfers (at downloader's discretion)
* support for alternate hashing algorithms (SHA, etc.; the normal MX algorithm is vulnerable to file spoofing attacks)
* support for transfers between firewalled peers (current plan is a UDP conversation initiated by both sides simultaneously)
* encrypt or at least obfuscate the conversation in a way that would make it difficult for qos-type devices to identify (currently MX file transfers aren't encrypted or obfuscated in any way, and are easily identified for throttling purposes)
* some means of negotiating such a transfer between clients using today's wpn (so not every primary has to be updated before secondary clients can begin using it)

For the last item, my current plan is to add a special prefix to the filename in request/confirm packets, utilizing characters not valid in true filenames.

Any thoughts?

[']['affy]

anyway to see how much of a file someone has or is getting sirmoon?

[Lagerlout666]

personally i dont like the getting the file in blocks, one of winmx's greatest perks to me over torrents was the facility to see a movies beginings with VNC before gettin a huge chunk of the file first like you do with torrents, if you have ever suffered a badly named file you will understand what i mean.

The way to beat qos in my eyes is to have the handshake always a random key. this could be called from the initial handshake and on the end of thechunk being sent will be the new expected key, always changing and pretty hard to block.

Ill be honest im not up on the encryption side of things but you could look to waste and such programmes for a little enlightenment.

[GhostShip]

I cant agree that most folks are concerned with IPv6 , they are like myself concerned with how we can avoid being throttled to death, so in this respect your discussion thread is very relevant Sir Moon 

I have one or two workable ideas to utilise the existing networks capabilties without affecting any of the current clients and I,ll make some diagrams of the exact method required so you can look them over, although with the best will in the world it may be better off folks deciding on something completely new and have "old style compatibility mode" to ensure there is a smooth transition to any new standard, of course getting agreement on any standard will not be easy 

I am a little concerned that you have not given the readers here any idea of how things work at the moment to base any ideas on, I suspect this being so you have unintentionaly limited the amount of viable new ideas and discussion partners to those already holding that knowledge, any chance of giving the folks here a brief run down on the current method and why its believed necessary to update it ?

 

[sIrMoOn]

personally i dont like the getting the file in blocks, one of winmx's greatest perks to me over torrents was the facility to see a movies beginings with VNC before gettin a huge chunk of the file first like you do with torrents, if you have ever suffered a badly named file you will understand what i mean.

One thing about out-of-order transfers is, code which supports it almost inevitably supports in-order transfers as well.  Whether in-order, completely-out-of-order, or beginning-first-rest-out-of-order is the best way to download files is up to debate and experimentation, and perhaps a configurable setting in the client, but I do think that the MX network tends to have many more copies of beginnings than are needed, and comparatively fewer copies of ends, and that this is not an optimal situation.

For the record however my own in-development client currently saves files to disk in pretty much the same way WinMX does, which does not provide any means of keeping track of what sections are downloaded or not and thus pretty much forces in-order transfers.  Being able to actually download files out-of-order is not currently on my list of features for the first version of MoonMX; however, I do think it is a good idea to support out-of-order upload for other clients that may wish it, and to support (but not require) out-of-order download in a later version.

The way to beat qos in my eyes is to have the handshake always a random key. this could be called from the initial handshake and on the end of thechunk being sent will be the new expected key, always changing and pretty hard to block.

Ill be honest im not up on the encryption side of things but you could look to waste and such programmes for a little enlightenment.

I agree that the content should ideally look completely random to whoever is sniffing it = )

As for Waste, its security model is completely connection-oriented, and so would not apply well to UDP protocols, which to my knowledge is the best way to accomplish goal #5 from my list, transfer between firewalled users.  Waste accomplishes firewall-independence by routing all traffic, including file transfer, through their equivalent of a primary node.  Obviously routing files through the primaries would greatly increase the bandwidth requirements for primary nodes, and so is something I wish to avoid.  All that said, Waste's security model is worth considering for secondary and primary links, if (when) it comes time to upgrade those.

anyway to see how much of a file someone has or is getting sirmoon?

I'm not sure I understand the question here - you already get progress bars for uploads, do you not?

I cant agree that most folks are concerned with IPv6 , they are like myself concerned with how we can avoid being throttled to death, so in this respect your discussion thread is very relevant Sir Moon 

Well, most folks who have posted anything protocol-related.  I suppose the people concerned with throttling are the silent majority : )

I have one or two workable ideas to utilise the existing networks capabilties without affecting any of the current clients and I,ll make some diagrams of the exact method required so you can look them over, although with the best will in the world it may be better off folks deciding on something completely new and have "old style compatibility mode" to ensure there is a smooth transition to any new standard, of course getting agreement on any standard will not be easy 

There's an old joke about this - "Standards are wonderful; there are so many to choose from."

I am a little concerned that you have not given the readers here any idea of how things work at the moment to base any ideas on, I suspect this being so you have unintentionaly limited the amount of viable new ideas and discussion partners to those already holding that knowledge, any chance of giving the folks here a brief run down on the current method and why its believed necessary to update it ?

Well I had listed my goals for a new protocol, so the "why its believed necessary to update it" comes down to "it doesn't do any of these things".  That said I've gone back and expanded on a couple points for users who may not know why they're important/useful.

[']['affy]

what i'm trying to say is is there a way to see how much of a file someone has b4 u start them.

[sIrMoOn]

I think I see what you're getting at - so you could start someone who maybe doesn't have much to go and won't take very long.

It could be done - and your client could even be made to enforce it by not sending them more than they said they needed - but since you would need the information before the file transfer started a new file transfer protocol by itself couldn't do this.  We'd have to make some enhancements to how files are queued.

It's a good idea, so thank you = )  That said introducing something like that without requiring new primaries might be hard to do.  I will definitely look into it.

[GhostShip]

It seems to have gone all quiet here so I suppose its time I said some more on this interesting subject 

I just read up at the list of possible and potential enhancements you made and think its best they are commented on in order so you have some idea what I'm rambling on about Sir Moon 

* 64-bit file sizes - this would increase the maximum file size from the current 2 gigabytes to something like 9 petabytes

For those that are not aware of why Winmx has this limit its because without it we lose windows 98 compatibility, so although a useful plan it will mean losing some potential users of a new client, my suggestion is to gather data to see how many 98 using folks are currently on the network have before implementing this.

* support for out-of-order transfers (at downloader's discretion)

This is a sensible idea for files that many people have the start of but not the end section, although this could be abused by the Cartel in the same sort fake block disruption tricks we see them using on torrents, torrents at least have the back-up system of being able to compare each chunk to a hash.
All that aside I do think this as an option would be a great idea as it would speed up multiple client downloads no end.

 

* support for alternate hashing algorithms (SHA, etc.; the normal MX algorithm is vulnerable to file spoofing attacks)

I have long thought this would be a great idea but the current hash size is a fixed field size in both primary and secondary traffic and any size increase will mean mean a rewrite of all protocols utilising this data field, not impossible of course but long winded and of course the length field will need to be worked in their somewhere, also we mustn't forget why the hashing system is so weak, once again its to ensure the hashing time is speedy and not a dead crawl to win98 users, its a trade off system.
One other important factor being forgotten here is that having spoofed hashes on the network means its impossible for the cartel to prove definitively that you have a particular file as its well known they can be the same for different files, and when facing a legal case this could be an important factor in proving your innocence.

* support for transfers between firewalled peers (current plan is a UDP conversation initiated by both sides simultaneously)

This will of course likely require you to add your own protocol extensions and as that seems the rage theses days I don't see any problems, it would help of course to publicise the changes so others could implement them at some stage.

* encrypt or at least obfuscate the conversation in a way that would make it difficult for qos-type devices to identify (currently MX file transfers aren't encrypted or obfuscated in any way, and are easily identified for throttling purposes)

I think you know my views on this one Sir Moon, its something we needed yesterday, the only real issue here is how to implement the system you decide to use to utilise the current primary network to Carry your modified traffic, lots of lateral thinking required 

* some means of negotiating such a transfer between clients using today's wpn (so not every primary has to be updated before secondary clients can begin using it)

This is often the topic of some of our recent discussions and I'm sure you have some ideas that will do the job and at the least in the new clients only you can use some of the valid but "invalid " features of the system to obtain a free ride, supplemented by a new type or two if necessary.

Anyone else have any opinions on this matter is welcome to speak up, Sir Moon is asking you what you want him to code into his client so nows a good time to speak up 

[Ace]

for fake file checking can use         test 2 files for equality
   
 by  calculate the hash code of each file using the  System.Security.Cryptography.HashAlgorithm     class

lot of code to wright  for here  but the baisc c#     version is

           //calculate  the hash for files.
               
         byte[] hashByesA = hashAlgCumputeHash(fsA);
         byte[] hashByesB = hashAlgCumputeHash(fsB);
 
          //Compare the hash.
         
         if(BitConverter.ToString(hashBytesA) ==
            BitConverter.ToString(hashBytesB))
         {
         
        a hash code  checks each file by bytes    then compare s the bytes again the file 
        any data tampering

      this just a basic form code as a viewing example  theres lot more to the code   
      which can google      as no your coding with diff c# idle then wot i use  and the
       System.Security.Cryptography.HashAlgorithm     may be diff to wot i use      but gives you idea wot i mean
           totaly agree wot QS said regarding 64b  file size would kill lot off plp that use older windows versions
     allso to use the partial   string     it done in blocks off 8,192bytes  which is the max buffer < in memory>   size untill eof     < end of file >