IIRC, and I am not sure that I do, but isn't there some bug in the windows TCP/IP stack that you can't have too many "open" udp "connections" at once?
All of the communication is done through a single UDP socket. DTCP is a multiplexing transport protocol which operates over a single UDP connection.
You are correct about the number of open UDP sockets though. On any UNIX or NT variant the limit is usually 1024 to 2048 per process, and 64k per IP address (the PORT value in UDP or TCP is only 2 bytes)
This is why native UDP or TCP cannot support the required number of connections to perform direct queries to each peer in a large network.
If the users are chained together through ids one hop at a time, then you would have to route and re-ruote a query for their ids before you even do anything!
No, you missed a major point; there is no routing and no forwarding.
This is what makes it so simple, and linear. You directly communicate with all the peers you want to query. Everyone directly communicates with each other. The only thing this implies is a transport service which can support a large number of concurrent connections efficiently. This is what DTCP does.
A simple query with an 8 character query string would be 56 bytes. The string above might be as much as 160 bytes.
I did some monitoring of the gnutella network for a few months, and the size of an average query is about 8-16 bytes at most. Many many queries where even less.
You've still got to query each peer, and a linear search like that isn't acceptable when you've got a lot of people hitting a big database
There is no big database. There are lots and lots of little databases.
And the network adapts to load. I go into this somewhat on the site.
The important thing I want to point out is that this network is used mainly for locating content. Once you have found it, it may reside within Freenet, it may reside within OpenCOLA, or MojoNation, etc. And then you will benifit from their architecire for the actual delivery of the data.
The broadcasts are used solely for discovery of resources, with the delivery being a whole other scenario. UDP is a horrible bulk transfer protocol.
Thats discussed on the site I mentioned, but essentially you each pick an ID to associate a given peer with. Its that simple.
2. Let's say 10% of those 10K people are doing searches. That saturates a 56K modem, assuming you can really get your packets down to 56 bytes
It would only saturate your link if all 10,000 searched at once. If they all searched within a 3 minute time period, or no more than 70 in one second, your link will not saturate. And the packet is 56 bytes for an 8 character query. For a 16 character query, it would be 64 bytes. etc.
What happens when you try to have 100K people? One million? How about the 10 million+ of Napster? Your scheme would not scale.
That depends on how good of a peer you are. If you dont repsond much, and have a very low link, then you will be at the bottom of those 100,000 hosts query lists, and will get queried infrequently. I cover this on the site, but this is not a problem. The only thing that is limiting your use of the network is how much memory you have (you would need a hundred meg or so for a million connections) and your bandwidth.
Actually, UDP is almost universally used by game programmers because it is the only way to get around NAT (without central servers, which dont work well for gaming)
Also, no firewalls and proxies do not filter UDP by default (none that I have encountered), although you can configure them to do so.
No, because you control exactly how often or how much response you provide.
If you are getting swamped, you will respond to less and less queries, and then your quality in the eyes of those peers will drop, thus, you will receive less and less queries.
This is actually a balanced type of configuration, which handles load in an efficient manner.
Also note that over a DSL line, you could receive in excess of 10,000 queries a second.
The more relevent question is whether you can have a peer-to-peer network without central servers that *can* scale. And the answer is "no".
Not so fast. Right now, the biggest problem with decentralized networks is that they all have some form of routing/forwarding. If you got rid of routing/forwarding, then they could scale.
For instance, lets say you have a napster style peer group, 10,000 peers. What if, to query these peers, you sent a small UDP packet to each of them directly? No routing, no forwarding. How long would this take?
Modem: 2.5 minutes
DSL: 13 seconds
I would say that this is an acceptable period of time. And the bandwidth used was all your own, nobody elses, except for the 56bytes each peer received for that single packet they got from you.
I am working on such a network, its called The ALPINE Network and has all the features mentioned.
So, if you get rid of the forwarding/routing you can have a decentralized network that scales linearly.
It scales linearly, and provides a query mechanism that rivals the performance of a centralized directory. (Although the bandwidth is more than a centralized query, but at least you have direct control over how much bandwidth you use and how).
At any rate, I could use development assistance a great deal. Let me know if anyone is interested.
I am currently working on a fully decentralized searching network. You can read more about it here.
The key aspects of this network will be:
- No forwarding. This is currently eating gnutella alive. A UDP based multiplexed transport protocol is used to maintain hundreds of thousands of direct connections to all the peers you want to communicate with. You can also tailor your peering groups precisely to what you desire, as far as quality, reliability, etc.
- Low Communication Overhead. All queries that are broadcast are performed with minimal overhead within UDP packets. A typical napster breadth query (10,000 peers) would take a few minutes on a modem, and seconds on a DSL line.
- Adaptive Configuration. Peers that have better or more responsive content will gravitate towards the top of your query list, thus, over time you will have a large collection of high quality peers which will greatly increase the chance of you finding what you need.
There are a number of other features, however too much to detail here.
Also, this is under heavy development, and not operational. I am going solo on this at the moment, and so progress is slow. However, once completed, it *should* be a scalable alternative to completely decentralized searching / location.
1- If you are maintaining code, you should not be refactoring code. There should be 'specific' bugs that you are fixing. Dont do more than this. If you do more than this, the additional changes must be tested, documented, etc. This is not the purpose of maintenance.
2- If the code is that bad, then propose refactoring to management and let them consider and delegate. It is highly unlikely that the developers doing maintenance will be the ones who design the refactored implementation.
Management hates refactoring because they have to justify and fund a project to provide the same functionality which currently exists (although in a much less glorius form)
Development entails not just coding, but testing, integration, documentation, etc, etc. The code part of the project is the smaller of the whole, which is why management may be relunctant, even if you say you can code the changes in a few months.
Does anyone know if this book is someone more geared towards a laymon, maybe a guide rather then a reference?
This book is definately more of a guide, however, it is detailed enough to serve as a general reference as well. (Detailed reference == see the code;)
If you mean TCP/IP illustrated V.2 concerning the BSD stack implementation, then yes, this book might be slightly better from a technical prerequisite standpoint.
You will want to know some basic CPU architecture information to get the most of this book. I.e. how CPU caches work, virtual/physical address mapping, etc.
I recently purchased this book (about a month and a half ago) and am delighted in its breadth and clarity.
The authors layout the information they wish to present clearly, and every chapter is a refinement of these main areas of functionality.
The book is also sprinkled with a lot of code, so that you can see the concepts in action. There are also plenty of diagrams, which gives the book a feel similar to something R. Stevens would write. (like TCP/IP illustrated / Unix Network Programmin)
If this is the kind of thing your interested in, definately spin for a copy. It can be a bit hard of a read given its size and density, however worth the effort.
I want to know what your view on the copyright wars of late might be.
Specifically, the FCC passed a law prohibiting the recording of HDTV digital content. Do you feel that this is a violation of the home audio recording act/betamax decision?
Does the FCC have any interest in large media corporations planning systems to prohibit time shifting of broadcast context? (unrecordable music / tv / etc)
At one time companies were loyal to their employees. They valued their contributions, saw them as more them numbers and assets, and life was good. Employees were loyal to their employers in return. It was a mututally benificial relationship
Things have changed, and loyalty was lost in the cold cogs of business and profit.
Today, the only one you should be loyal to is yourself. Businesses that cannot create an environment conducive to employee needs are not fit to be in business.
Get the hell out of there and let a functional businesses which will value and utilize your skills benifit. Your on a sinking ship. Get out while the getting is good.
Re:Isn't this what Reiser FS is for?
on
MySQL FS
·
· Score: 3
goober:$ cd/mnt/sqldb
goober:$ ls
USER_ID FIRST_NAME LAST_NAME TIMESTAMP
goober:$ mkdir AGE
goober:$ echo "Oh crap, there goes my schema!"
"Oh crap, there goes my schema!"
goober:$ cd USER_ID
goober:$ ls
11023 11025 11044 11055 11092
goober:$ rm 11023
goober:$ echo "Wow! I hope that wasnt relational!"
"Wow! I hope that wasnt relational!"
goober:$ exit
Seriously, what type of integrity checking will be enforced in this filesystem?
I am betting that you either have robust integrity, which would give a completely counterintuitive file system, or lax integrity which would open the doors for all sorts of mischevious errors and data corruption.
When you die, everyone will say you did nothing but good things at your funeral. They will forgive you, and wish you a peacefull slumber in the dirt through eternity.
You may wish to consider that some people have, unlike yourself, not been born to always make the correct choices, but have sometimes had to make serious mistakes first. From these mistakes they must have the opportunity to make their lives right.
Sure, but that does not mean that we should all forget that anything happened. Which is exactly what you want apparently.
Sorry, but it doesnt work that way.
If my past mistakes will continue to penalize me, then I have no incentive to make the diffcult choice to right my path.
Yes, you do. Because if you continue making bad choices, you will incur further consequences, and make yourself even more miserable.
Doing things the 'right' way, allows you time to rebuild your credit, or history. It will never be a clean slate, because you DID FUCK UP, but it can always be better.
If I am arrested for a DWI, why should I make the decision to correct my behavior, when I will be unable to find employment 10 years later regardless?
Because the next time your in jail for a lot longer. The third or fourth, you may actually kill some little girl crossing the street. Then you get many many years. Oops.
If I am a foolish tennager who undertakes a bad debt, why should I even bother applying to college, knowing the debt will deny me the possibility of financial aid?
Then study your ass of and get a scholarship. Work your ass off and save and pay for it yourself. No one MUST HAVE credit. It is nice to have, but certainly not a requirement. Get a grip.
That is not the CRA's fault you idiot. That is VISA telling you to PROVE that the charges on your CARD were fraudulent.
That is a risk in carrying a credit card. If VISA says that you did NOT make the purchases, all you have to do is tell the CRA that you dispute the item. They will then verify this with visa, and it will be gone.
The problem is when there are legitimate or fradulent accounts with lenders. If you are a fraud victim, and this does happen often, i.e. identity theft, you have to get the authorities involved, lock away the fucker, and give the details to the creditors. Cleaning up your CRA info is the easy part.
People aren't complaining about the consequenses of their actions. They're complaining about being saddled with the consequences of OTHER people's actions
You didn't read katz post. The teacher WAS a drunk driver, the spray painter WAS a vandal, and the college kid DID screw over the music company.
The bigger question which katz should be posing is wether the consequences of their actions are just.
Should a drunk driver never be allowed to teach?
Does a spary painting have anything to do with your qualifications for a security clearance?
Slashdot Mirror
IIRC, and I am not sure that I do, but isn't there some bug in the windows TCP/IP stack that you can't have too many "open" udp "connections" at once?
All of the communication is done through a single UDP socket. DTCP is a multiplexing transport protocol which operates over a single UDP connection.
You are correct about the number of open UDP sockets though. On any UNIX or NT variant the limit is usually 1024 to 2048 per process, and 64k per IP address (the PORT value in UDP or TCP is only 2 bytes)
This is why native UDP or TCP cannot support the required number of connections to perform direct queries to each peer in a large network.
If the users are chained together through ids one hop at a time, then you would have to route and re-ruote a query for their ids before you even do anything!
No, you missed a major point; there is no routing and no forwarding.
This is what makes it so simple, and linear. You directly communicate with all the peers you want to query. Everyone directly communicates with each other. The only thing this implies is a transport service which can support a large number of concurrent connections efficiently. This is what DTCP does.
A simple query with an 8 character query string would be 56 bytes. The string above might be as much as 160 bytes.
I did some monitoring of the gnutella network for a few months, and the size of an average query is about 8-16 bytes at most. Many many queries where even less.
You've still got to query each peer, and a linear search like that isn't acceptable when you've got a lot of people hitting a big database
There is no big database. There are lots and lots of little databases.
And the network adapts to load. I go into this somewhat on the site.
The important thing I want to point out is that this network is used mainly for locating content. Once you have found it, it may reside within Freenet, it may reside within OpenCOLA, or MojoNation, etc. And then you will benifit from their architecire for the actual delivery of the data.
The broadcasts are used solely for discovery of resources, with the delivery being a whole other scenario. UDP is a horrible bulk transfer protocol.
No, FidoNet requires a 'devlivery' or 'bulk transfer' protocol.
The protocol used by ALPINE is for messaging. The types of broadcasts are very small packets. Usually 50-60 bytes. This makes a huge difference.
1. How do you identify all the peers?
Thats discussed on the site I mentioned, but essentially you each pick an ID to associate a given peer with. Its that simple.
2. Let's say 10% of those 10K people are doing searches. That saturates a 56K modem, assuming you can really get your packets down to 56 bytes
It would only saturate your link if all 10,000 searched at once. If they all searched within a 3 minute time period, or no more than 70 in one second, your link will not saturate. And the packet is 56 bytes for an 8 character query. For a 16 character query, it would be 64 bytes. etc.
What happens when you try to have 100K people? One million? How about the 10 million+ of Napster? Your scheme would not scale.
That depends on how good of a peer you are. If you dont repsond much, and have a very low link, then you will be at the bottom of those 100,000 hosts query lists, and will get queried infrequently. I cover this on the site, but this is not a problem. The only thing that is limiting your use of the network is how much memory you have (you would need a hundred meg or so for a million connections) and your bandwidth.
Actually, UDP is almost universally used by game programmers because it is the only way to get around NAT (without central servers, which dont work well for gaming)
Also, no firewalls and proxies do not filter UDP by default (none that I have encountered), although you can configure them to do so.
No, because you control exactly how often or how much response you provide.
If you are getting swamped, you will respond to less and less queries, and then your quality in the eyes of those peers will drop, thus, you will receive less and less queries.
This is actually a balanced type of configuration, which handles load in an efficient manner.
Also note that over a DSL line, you could receive in excess of 10,000 queries a second.
The more relevent question is whether you can have a peer-to-peer network without central servers that *can* scale. And the answer is "no".
Not so fast. Right now, the biggest problem with decentralized networks is that they all have some form of routing/forwarding. If you got rid of routing/forwarding, then they could scale.
For instance, lets say you have a napster style peer group, 10,000 peers. What if, to query these peers, you sent a small UDP packet to each of them directly? No routing, no forwarding. How long would this take?
Modem: 2.5 minutes
DSL: 13 seconds
I would say that this is an acceptable period of time. And the bandwidth used was all your own, nobody elses, except for the 56bytes each peer received for that single packet they got from you.
I am working on such a network, its called The ALPINE Network and has all the features mentioned.
So, if you get rid of the forwarding/routing you can have a decentralized network that scales linearly.
I am nearing completion of a network that satisfies a, b, c, e.
I havent started on d and f, but they could be added.
This project is called The ALPINE Network
It scales linearly, and provides a query mechanism that rivals the performance of a centralized directory. (Although the bandwidth is more than a centralized query, but at least you have direct control over how much bandwidth you use and how).
At any rate, I could use development assistance a great deal. Let me know if anyone is interested.
Regards...
This was a plea for development assistance ;)
I could very much use some additional C++ development talent to help with this project. Anyone who is interested please let me know.
Thanks...
I am currently working on a fully decentralized searching network. You can read more about it here.
The key aspects of this network will be:
- No forwarding. This is currently eating gnutella alive. A UDP based multiplexed transport protocol is used to maintain hundreds of thousands of direct connections to all the peers you want to communicate with. You can also tailor your peering groups precisely to what you desire, as far as quality, reliability, etc.
- Low Communication Overhead. All queries that are broadcast are performed with minimal overhead within UDP packets. A typical napster breadth query (10,000 peers) would take a few minutes on a modem, and seconds on a DSL line.
- Adaptive Configuration. Peers that have better or more responsive content will gravitate towards the top of your query list, thus, over time you will have a large collection of high quality peers which will greatly increase the chance of you finding what you need.
There are a number of other features, however too much to detail here.
Also, this is under heavy development, and not operational. I am going solo on this at the moment, and so progress is slow. However, once completed, it *should* be a scalable alternative to completely decentralized searching / location.
A few things I would like to point out:
1- If you are maintaining code, you should not be refactoring code. There should be 'specific' bugs that you are fixing. Dont do more than this. If you do more than this, the additional changes must be tested, documented, etc. This is not the purpose of maintenance.
2- If the code is that bad, then propose refactoring to management and let them consider and delegate. It is highly unlikely that the developers doing maintenance will be the ones who design the refactored implementation.
Management hates refactoring because they have to justify and fund a project to provide the same functionality which currently exists (although in a much less glorius form)
Development entails not just coding, but testing, integration, documentation, etc, etc. The code part of the project is the smaller of the whole, which is why management may be relunctant, even if you say you can code the changes in a few months.
Does anyone know if this book is someone more geared towards a laymon, maybe a guide rather then a reference?
;)
This book is definately more of a guide, however, it is detailed enough to serve as a general reference as well. (Detailed reference == see the code
If you mean TCP/IP illustrated V.2 concerning the BSD stack implementation, then yes, this book might be slightly better from a technical prerequisite standpoint.
You will want to know some basic CPU architecture information to get the most of this book. I.e. how CPU caches work, virtual/physical address mapping, etc.
I recently purchased this book (about a month and a half ago) and am delighted in its breadth and clarity.
The authors layout the information they wish to present clearly, and every chapter is a refinement of these main areas of functionality.
The book is also sprinkled with a lot of code, so that you can see the concepts in action. There are also plenty of diagrams, which gives the book a feel similar to something R. Stevens would write. (like TCP/IP illustrated / Unix Network Programmin)
If this is the kind of thing your interested in, definately spin for a copy. It can be a bit hard of a read given its size and density, however worth the effort.
I want to know what your view on the copyright wars of late might be.
Specifically, the FCC passed a law prohibiting the recording of HDTV digital content. Do you feel that this is a violation of the home audio recording act/betamax decision?
Does the FCC have any interest in large media corporations planning systems to prohibit time shifting of broadcast context? (unrecordable music / tv / etc)
Thanks...
At one time companies were loyal to their employees. They valued their contributions, saw them as more them numbers and assets, and life was good. Employees were loyal to their employers in return. It was a mututally benificial relationship
Things have changed, and loyalty was lost in the cold cogs of business and profit.
Today, the only one you should be loyal to is yourself. Businesses that cannot create an environment conducive to employee needs are not fit to be in business.
Get the hell out of there and let a functional businesses which will value and utilize your skills benifit. Your on a sinking ship. Get out while the getting is good.
Except there is no SQL interface to reiserFS ;)
Lets see...
/mnt/sqldb
goober:$ cd
goober:$ ls
USER_ID FIRST_NAME LAST_NAME TIMESTAMP
goober:$ mkdir AGE
goober:$ echo "Oh crap, there goes my schema!"
"Oh crap, there goes my schema!"
goober:$ cd USER_ID
goober:$ ls
11023 11025 11044 11055 11092
goober:$ rm 11023
goober:$ echo "Wow! I hope that wasnt relational!"
"Wow! I hope that wasnt relational!"
goober:$ exit
Seriously, what type of integrity checking will be enforced in this filesystem?
I am betting that you either have robust integrity, which would give a completely counterintuitive file system, or lax integrity which would open the doors for all sorts of mischevious errors and data corruption.
Does anyone else think the $50,000 application fee for a TLD to even be considered is enough for investigation?
What the hell is ICANN doing that requires 50 G's to process an application???
Perhaps their data entry personel are making $5,000,000 / hour...
Accountability isn't forever.
When you die, everyone will say you did nothing but good things at your funeral. They will forgive you, and wish you a peacefull slumber in the dirt through eternity.
Well, the doctor didn't lube my enema this morning, so I'm a bit cranky.
Nothing like tearing sphincter to start your day.
You may wish to consider that some people have, unlike yourself, not been born to always make the correct choices, but have sometimes had to make serious mistakes first. From these mistakes they must have the opportunity to make their lives right.
Sure, but that does not mean that we should all forget that anything happened. Which is exactly what you want apparently.
Sorry, but it doesnt work that way.
If my past mistakes will continue to penalize me, then I have no incentive to make the diffcult choice to right my path.
Yes, you do. Because if you continue making bad choices, you will incur further consequences, and make yourself even more miserable.
Doing things the 'right' way, allows you time to rebuild your credit, or history. It will never be a clean slate, because you DID FUCK UP, but it can always be better.
If I am arrested for a DWI, why should I make the decision to correct my behavior, when I will be unable to find employment 10 years later regardless?
Because the next time your in jail for a lot longer. The third or fourth, you may actually kill some little girl crossing the street. Then you get many many years. Oops.
If I am a foolish tennager who undertakes a bad debt, why should I even bother applying to college, knowing the debt will deny me the possibility of financial aid?
Then study your ass of and get a scholarship. Work your ass off and save and pay for it yourself. No one MUST HAVE credit. It is nice to have, but certainly not a requirement. Get a grip.
That is not the CRA's fault you idiot. That is VISA telling you to PROVE that the charges on your CARD were fraudulent.
That is a risk in carrying a credit card. If VISA says that you did NOT make the purchases, all you have to do is tell the CRA that you dispute the item. They will then verify this with visa, and it will be gone.
The problem is when there are legitimate or fradulent accounts with lenders. If you are a fraud victim, and this does happen often, i.e. identity theft, you have to get the authorities involved, lock away the fucker, and give the details to the creditors. Cleaning up your CRA info is the easy part.
People aren't complaining about the consequenses of their actions. They're complaining about being saddled with the consequences of OTHER people's actions
You didn't read katz post. The teacher WAS a drunk driver, the spray painter WAS a vandal, and the college kid DID screw over the music company.
The bigger question which katz should be posing is wether the consequences of their actions are just.
Should a drunk driver never be allowed to teach?
Does a spary painting have anything to do with your qualifications for a security clearance?