Author Topic: My personal ideas for a “safe” NEW WinMx file transfer system (Read 33053 times)

Hans-Linux · « **on:** August 22, 2011, 01:53:54 pm »

Hi,

I am for many years a WinMx user and p...off at the inability to search for files. Therefore most of my downloads are done using Bit-torrent. This works very well if I can find the appropriate torrent file and is very fast for downloading disk images.

I would like everyone to have a look at “GNUnet” being developed at the Technical University of Munich:
https://gnunet.org
http://en.wikipedia.org/wiki/GNUnet

Everone is welcome to pull apart and improve my personal ideas for a“safe” NEW WinMx file transfer system:

1.) Utilise or create a reliable and easy to use open source transfer client and protocol for speed and security, such as “Transmission” (www.transmissionbt.com).
2.) Implement a rock solid system hosted on each client for uploading the data of the the files located in the client's Upload (Library) directory.
3.) Implement a distributed, redundant databases hosted by “Primaries”. The databases contain the file name, SHA, file length and time stamps for creation, upload, last completed download, number of commenced downloads, number of completed downloads. The database is searchable for full or partial file names and SHA. Similar to those used by “Pirate Bay” and others, but without web interface.
4.) The “Primaries” work like a DNS server tree with automatic root servers and backup and load balancing servers and “Trackers” and/or dicovery via DHT.
5.) Traffic between clients and “Primaries” and data stored on “Primaries” are encrypted.

Operation:
1.) Primaries connect and update their databases with “rsync”.
2.) Clients connect to a “Primary” upload their Library information consisting of file names, SHA, file length and file creation date.
3.) Primary verifies and enter the received data into the database. If the SHA exists, links the file name to the existing SHA and updates the duplicate field.
4.) Client sends search parameter (keyword, full file name or SHA) to a “Primary”
5.) “Primary” searches the database. If match is found and client request download, “Primary” generates and returns the Torrent File to the client.
6.) Client commences download and confirms completed download to a “Primary”.
7.) Primary updates it database.

Protection:
1.) All traffic between Clients and Primaries is encrypted.
2.) Fake file protection is automatically handled by the Primaries. This is done by analysing the balance between the number of started and completed downloads. Clients providing fake files are blocked. Corrupted files are blocked
3.) Directory flooding is avoided by implementing a limit to the number of new files that can be added per client per day.

Unknowns:
1.) Total number of Clients
2.) Total number of Primaries
3.) Total number of different files
4.) Total number of different file names
The size of distributed database and indexes for 1,000,000 files is estimated to be 512 MB

Catrooms:
Incorporate a suitable open source chatroom client and server as a option into each client.

Hans

GhostShip · « **Reply #1 on:** August 22, 2011, 06:15:27 pm »

My thanks for your suggestions Hans, I and I'm sure others will take some time to look in detail at what you have suggested and I will post back here when I have done so and consulted around the community to get an idea of feeling regarding whether the system you outline is one thats felt a good "fit" for the communities requirements

Once again my thanks for your input in our hour of need

White Stripes · « **Reply #2 on:** August 23, 2011, 12:57:54 am »

http://www.filetopia.org/

i know im going to get chided for this... but filetopia uses public key encryption with random key generated each time the app is started (and you can even pick the cypher and bitdepth) supports 'trading' but keeps the traders and sharers seperate..... ....and is where most of the ex-winmx users are migrating to... so most likely a few folks here already know a few that are there....

bad part is its closed source.... but that may be a good thing too... depends...

silicon_toad2000 · « **Reply #3 on:** August 23, 2011, 01:19:38 am »

We're not here to advertise other p2p networks, but it's well worth looking at the other communities to see how and why they do things.

White Stripes · « **Reply #4 on:** August 23, 2011, 01:34:59 am »

you said in another post that winmx 'the app' and winmx 'the community' arent the same thing... and since a lot of winmx the community has already made this move it seemed logical.....

or put another way... its less an advertisement for another network than it is info on where a large chunk of the 'community' has moved....

* Silver Stripes shrugs?

GhostShip · « **Reply #5 on:** August 23, 2011, 03:03:17 am »

I have used the filetopia network myself and thus know quite a bit about it.

Its pretty obvious to anyone who has been used Filetopia aka Tesla that whilst the transactions are very well encrypted the amount of files is pretty drastic compared to winmx and the speed delivered is really nothing to write home about, however the worst thing about it all is the centralised nature of the main server, that's right folks its not even decentralised and that my friends is the reason why I may suggest its temporary usage but in the long term its a liability waiting to happen.

Now can we get back on topic please Stripes, its getting rather unfair for other posters to have you hijacking their threads with your unhelpful anti-winmx suggestions at every turn, what about taking yourself a break and allowing me to use this wasted post time elsewhere.

achilles · « **Reply #6 on:** August 23, 2011, 03:46:31 am »

Stripes, Filetopia is a great program! It just needs more users. I would like to see the network made decentralized though, and it's development has not been continued in quite a while. It's probably the best alternative to WinMx. In fact, I would like to see the WinMx community, and Filetopia community unite to create a new ultra secure P2P network. How about it? Should we approach them in working together to achieve a common goal? We could bring the talent of both communities together. Everyone would win with such a deal!

White Stripes · « **Reply #7 on:** August 23, 2011, 05:25:53 am »

Quote from: GhostShip on August 23, 2011, 03:03:17 am

Now can we get back on topic please Stripes, its getting rather unfair for other posters to have you hijacking their threads with your unhelpful anti-winmx suggestions at every turn, what about taking yourself a break and allowing me to use this wasted post time elsewhere.

anti winmx the application.... not anti winmx the community.... same as i wouldnt recommend anyone use windows 95 directly connected to the internet....

and i didnt even know there was a 'filetopia' till i heard it through the winmx grapevine.... ...the barstools may be different but if its the same set of friends sitting next to each other i see no problem....

and i hijacked nothing... folks are going to leave with... or without you....

and as far as decentralized verus centralized... well... you need at least some amount of babysitting, and software thats still supported (or is fully open source), in a 'closed loop' type system else you end up with... well... the problem winmx.exe has now...

White Stripes · « **Reply #8 on:** August 23, 2011, 01:48:09 pm »

btw, filetopia and tesla are not the same thing.... tesla is defunct....

http://www.slyck.com/news.php?story=454

v4 didnt make it... obviously (that article was penned in 2004)... but v3 isnt 'centralised'.... no more than winmx anyway.... (node/supernode comparable to winmx secondary/primary)

...i think achelles is the only one in this thread (aside from the OP) with a level head .... and may be onto something there.... MX with randomly generated public key encryption? .... course that would kill 3rd party apps... namely chatservers.... but it would stop the 'war'* (for lack of a better something to call it)....

*kinda sucks that the 'three musketeers'** of mx (the original hostsfile circa 2005) became warring entities... mx v riaa (and to a lesser extent mpaa due to filesize limit) makes sense .... wmw v pie not so much .... then its wmw & pie vs ... well.. a different pie... then we get down to the wire and its gs vs km ........

......taking a step back and looking at the history pages it seems..... silly..... a civil war (never understood what part of war is 'civil' ... dontcha just love the english language) fought from various angles by various 'armies' since frontcode dropped off the map in 2k5......

**for those curious that would be km, vladd, and gs... (if im remembering correctly)

Lord Acton was right.....

PatrickShartz · « **Reply #9 on:** August 23, 2011, 05:50:09 pm »

Quote from: GhostShip on August 23, 2011, 03:03:17 am

Now can we get back on topic please Stripes, its getting rather unfair for other posters to have you hijacking their threads with your unhelpful anti-winmx suggestions at every turn, what about taking yourself a break and allowing me to use this wasted post time elsewhere.

I see next to your name you're supposed to be retired? Why do you have to play baby sitter if someone else has apparently taken over your job? We can have an intelligent and grown up conversation without you having to intervene...

The only thing restricting talk of other P2P networks and P2P technology is doing is making this website even less relevant. The network and this website are already tiny fractions of what they once were, it is simply not wise to shut even more people out when they are the few that actually post on this forum. If they were not here to post meaningful and interesting discussions (if i do say so myself) this place would be a ghost town as no one ever comes on here except the occasional person asking where this so called patch is that has so far been promised for the past 6 months but has not yet seen the light of day. Nor have any meaningful updates on the status of this so-called patch been made on the very generic and lacking any kind of useful information "attack information" page.

It makes absolutely no sense why you would shun the mere mention of bittorrent when public and private torrents sites are where the users of WinMX actually get their content, and then in turn bring it to share on WinMX. There are no scene release groups on this network ripping and encoding their own movies... the media comes from elsewhere and trying to simply ignore it is shooting yourself in the foot.

And when you say Silver Stripes has hijacked the thread, the original poster is the one who asked for these kinds of replies. He is interested in actually evolving WinMX and switching to a more modern file transfer method which has been suggested multiple times by multiple people on this forum, only to be shot down by yourself. To me is seems like you want WinMX to live, but only if you're the one to actually think up the way of how to fix it and you're the one who implements it. This is supposed to be a community website yet you're the one making all of the decisions, you're the one running the show, and no matter how many times you tell us otherwise it is still going to be true. You ask for help, and when help arrives you don't take it. Then you complain no one helps. You're way to hypocritical for my liking and it's getting old really fast.

I defended you when birdman/whoever made that blog about you, but now after seeing it through my own eyes I'm going to have to redact my defenses. Because honestly, those blog posts are 100% accurate, and you are just a controlling old man who is afraid of change. And simply because people such as Bluey who never contribute anything of meaning to this forum except snide and rude comments, and all of the other "supervisors" have no clue what the hell is going on right in front of their faces does not mean that everyone else has fallen under your cloak of stupidity. as well. Hell even your technical people are not up to par. "RebelMX" whoever that is had no idea what the actual primary udp protocol was when I explained how vulnerable the people on the WinMX network are in reference to search results. If your technical coders are not even familiar enough with the network, how is the patch ever going to get done?

RebelMX · « **Reply #10 on:** August 23, 2011, 06:06:18 pm »

To clear up, as a developer, I disagree with the entire idea of producing a new network. If that's what you guys want then go make it happen. However that IMO is not the future for winmx nor the "WPN".

As far as the patch is concerned, the attacks begun very early in May, which makes the situation less than 4 months old (lets not over-react and round everything up). I offered a theorectical solution back at that time, however I was not part of, or trusted much by WMW or GhostShip. In the past 2 weeks I have begun working on the practical implication of my theory, aside from any other WMW projects that are going on, with another developer assisting me.

The truth of the matter is that I have just begun a new college course. My rl is completely seperate from my online hobby, and thus I am no professional programmer. I believe my theory will work, and is well into the development stage. It could take weeks or maybe even a couple of months due to my recent additional time constraints. I am happy to explain this technique if you so require, but please note this is NOT the official WMW patch, so I cannot speak for that.

As for GhostShip's retirement... who knows, and atm, who cares either. Lets forget about "new clients" and "new features" and just get this god damn network back up and working. Then you lot can continue your name calling after. I'm not a huge fan of GhostShip before people try and claim I'm "ass kissing", he knows this, but we have a productive agreement atm. Which is more than most people who disagree on almost everything ;P

Patrick: The search packet has no real threat WHEN the network is operating normally. The searches that are used to data mine who is sharing what, can easily be picked up by tools already out there and the ip's added to the blocklist. The same blocklist already working in the patch you are no doubt using. So my lack of knowledge was truely a lack of care, on a particular packet that has no real effect on the overall network when its working normally. Due to the vast amount of data pouring through the network currently it disguishes these ip's and therefore yes it creates a partical worry, which will revert back again once a fix is released.

PatrickShartz · « **Reply #11 on:** August 23, 2011, 06:36:14 pm »

Quote from: RebelMX on August 23, 2011, 06:06:18 pm

Patrick: The search packet has no real threat WHEN the network is operating normally. The searches that are used to data mine who is sharing what, can easily be picked up by tools already out there and the ip's added to the blocklist. The same blocklist already working in the patch you are no doubt using. So my lack of knowledge was truely a lack of care, on a particular packet that has no real effect on the overall network when its working normally. Due to the vast amount of data pouring through the network currently it disguishes these ip's and therefore yes it creates a partical worry, which will revert back again once a fix is released.

Again you show no actual understanding of what I'm actually talking about. The search bots connect via secondary, then the primaries that they connect to do the searching. As there is no way to ask a primary what secondaries are connected to him/her, so there is no way to actually block the search bots that are collecting the information. Again you are confusing the udp search result packets that I am speaking of that contain the actual ip address of the person sharing the file with the internal tcp search packets that trigger the very udp packets that I am speaking of.

The blocklist you are talking about is what is filtering fake files, that has nothing to do with the passive method they are using to collect information on users of the winmx network and what files each ip address is sharing. As they are simply searching and receiving search results.

RebelMX · « **Reply #12 on:** August 23, 2011, 06:58:52 pm »

Actually, there you are wrong. The blocklist prevents any ip on that list from connecting to a primary. That therefore prevents them searching through a patched primary.
The current searches are made by either a) an ip not on the blocklist as it has not been added yet (see my post above for the reason) or b) due to the fact they are searching the network through an unpatched (read dll patch) primary (and therefore are using Sabre's method of connection) which offers no filtering or blocking.

So once again, I state, the searches are actually no worries for the network once it is back to working correctly. Perhaps checking the patch src and the tcp socket functions will help you see this. TCP packets are already linked to the blocklist, all TCP primary and secondary.

PatrickShartz · « **Reply #13 on:** August 23, 2011, 07:10:52 pm »

Quote from: RebelMX on August 23, 2011, 06:58:52 pm

Actually, there you are wrong. The blocklist prevents any ip on that list from connecting to a primary. That therefore prevents them searching through a patched primary.
The current searches are made by either a) an ip not on the blocklist as it has not been added yet (see my post above for the reason) or b) due to the fact they are searching the network through an unpatched (read dll patch) primary (and therefore are using Sabre's method of connection) which offers no filtering or blocking.

So once again, I state, the searches are actually no worries for the network once it is back to working correctly. Perhaps checking the patch src and the tcp socket functions will help you see this. TCP packets are already linked to the blocklist, all TCP primary and secondary.

No, you are not understanding what this conversation is about.

The people doing the fake file flooding are not the same as the people doing the passive data recording, therefor the blocklist has nothing to do with what I am actually talking about. As of this moment I can see with my own custom monitoring tools, the search bots are still running actively just as they have been for the past 5 years, currently around the k's as they go in alphabetical order down a keyword list.

We do not know the ip addresses of the search bots, as there is no way to ask a primary what secondaries are connected to them. They are not on the blocklist.

RebelMX · « **Reply #14 on:** August 23, 2011, 07:15:25 pm »

And there you truely are wrong. There are other tools used to gain the information of the ip's of those connected to the network. Primative and basic they do work. Just because you dont know of them, and havent heard them mentioned doesnt mean they are about. I havent seen them but have spoken to a reliable source who helped develop and run them...

My point still remains, ONCE then network has quietened down a little these tools can be utilised in order to control the blocklist a little better and therefore eliminate the search bots from the network (where possible). The blocklist is designed for fake file flooding, yes, but it actually would work for other things (as has been proven before) and therefore the ip's of the search bots, added to the blocklist would sort this out. Don't worry so much about small, minor, temporary issues!

PatrickShartz · « **Reply #15 on:** August 23, 2011, 07:21:48 pm »

Sorry, but the only way to actually discover the ip addresses of the search bots that connect via secondary to a primary is to be the primary they connect to. There is no way to discover the ip address of someone who is not sharing any files, and connected via secondary behind a primary unless you are that primary.

And since you yourself said they are using the pie patch cache servers to gain connection to the network, you have very little chance of actually having one of those users connect to you as they obviously do not use the same internal node exchange mechanisms that the official WinMX client uses.

I'm not going to take "oh yea someone said a tool does this" as an excuse for your poor understanding of how the network works. I know what methods there are of obtaining an ip address of a WinMX user, and these search bots that do not share any files at all, and are behind these primaries are not in any position to have their ip address discovered except in the single circumstance that I mentioned above.

RebelMX · « **Reply #16 on:** August 23, 2011, 07:46:58 pm »

I never said they used a pie patch. I said they COULD use.
They have to use a cache to access the network, so therefore they have to use the normal exchange method to get primary ip's.

I'm not continuing this conversation in the open as I feel I've already said plenty to explain the methods used. If you don't like or agree with those methods thats fine, I'm not going to change your mind. However if you seriously think I dont understand the primary protocol, when I'm the only person who has created anywhere near a primary client, then continue with that opinion. I don't care what you or anyone else thinks of me, I know, as do many others, exactly what knowledge and abilities I bring to the table, openly and honestly.

PatrickShartz · « **Reply #17 on:** August 23, 2011, 09:05:15 pm »

Quote from: RebelMX on August 23, 2011, 07:46:58 pm

I never said they used a pie patch. I said they COULD use.
They have to use a cache to access the network, so therefore they have to use the normal exchange method to get primary ip's.

I'm not continuing this conversation in the open as I feel I've already said plenty to explain the methods used. If you don't like or agree with those methods thats fine, I'm not going to change your mind. However if you seriously think I dont understand the primary protocol, when I'm the only person who has created anywhere near a primary client, then continue with that opinion. I don't care what you or anyone else thinks of me, I know, as do many others, exactly what knowledge and abilities I bring to the table, openly and honestly.

Sadly all I see from you is a lot of talk and nothing to show for it.

cuttingedge · « **Reply #18 on:** August 23, 2011, 11:30:02 pm »

You 2 are both right on the edge of curing this thing. I do think there is A way to find the IPs in the UDP packet. I can get IPs in the TCP. I think using the block list will still be a problem. A new block list would have to be maintained daily by the Primaries which wont happen for various reasons. Then you will only have half fixed it.
Your search has to be refiltered. For secondaries, no problem. For Primaries the file filtering program I have yet to conceive. Ya gotta plug 3 holes and I may not be able to come up with a fitting program for such an animal. But im going to try.
I think new patch needs the block list and file filters.

GhostShip · « **Reply #19 on:** August 24, 2011, 12:31:44 am »

I can confirm that there is a tool that detects these data miners and while we haven't publicised its creation or operation it been doing its job for 2 years and allowing us to block such mining, it uses a method overlooked by many technical folks, for instance KM even when he was asking why IP's where on the blocklist as he was unable to detect them using "traditional" tools

I think this is a case of bounded thoughts leading the conversation here and still not meeting the target , wmw have always been happy to work with the best of the community programmers and the fact that we have been able to deliver time and again shows its not just hot air.

Btw simply because you post via a proxy like the attackers do Patrick there is no excuse to be rude to me or anyone else here who disagrees with you, this is not a bit torrent support site, I had hoped that was clear enough at this stage, also I notice you bring nothing to the table like most critics, perhaps you can prove me wrong ?

If you are really interested in why I'm still here and not resting somewhere else enjoying the network we have collectively helped to save from death many times then simply ask most MX users who else was well known and thus in the best position to organise the communities response.
I was in reality asked by the wmw staff to help out during this emergency firstly to negotiate with KM as I know him well and then once his public deal breaking and cache attacks started to continue to work towards stabilising the community given the serious shortage of coding talent on hand, I will take my leave when its over so feel free to actually put something into the pot yourself if you feel serious about aiding me to retire and your able to do, if not then can I respectfully ask you step out of my way as I'm not happy to stop progress on any front to waste time on mindless drama or those who propagate it.

Talk is cheap, and as I hope I have demonstrated manners cost nothing.