Slashdot Mirror
Remote Direct Memory Access Over IP
doormat writes "Accessing another computer's memory over the internet? It might not be that far off. Sounds like a great tool for clustering, especially considering that the new motherboards have gigabit ethernet and a link directly to the northbridge/MCH."
Not to mention easy access to sensitive information in emails, documents, and PIMs that the user currently is running and are resident in memory.
Seriously though... this is where Scott McNealy's vision of "The Network is the Computer" comes even closer to reality.
S
The security implications are staggering.
How do we lobby for port number 31337 for the RDMA protocol?
How small a thought it takes to fill a whole life
This feature has been available for a while now, but using a dedicated link rather than IP. Sun call it Remote Shared Memory and it's mainly used for database clusters.
I take it that error code 500 will be used when the DIMM or controller is fried?
Sharing memory is not necessary in distributed programming if the variables are kept mostly local and a single computer works mainly with what it has stored in its local memory. This is very applicable to renderfarms where the acceleration scheme itself works very well for distributed rendering because methods such as the grid subdivides into cells each of which can be stored on and evaluated on a single computer with its local memory. Only a central computer is needed to control these nodes and store the ouput which is of very limited size and without great computational needs.
Checking out my form of escapism.
> Microsoft ultimately is expected to support RDMA
> over TCP/IP in all versions of Windows
Can you see it coming? The ultimate Windows root exploit!! Hmm... I guess someone has to go tell them. Othervise they won't notice it until it's too late...
Seriously, how do you dare to enable this kind of access?!?
I tried something like this a while ago -- I wanted to mount an NFS-exported file via loopback and use it as swap.
The file in question actually resided in a RAM drive on another machine on the LAN.
I couldn't get it to work in the 45 minutes or so I messed around with it. I'm not sure if Linux was unhappy using an NFS-hosted file for swap, or what exactly the problem was, but I did get some funny looks from people to whom I explained the idea (ie, to determine whether the network would be faster than waiting for my disk-based swap).
Of course, this was back when RAM wasn't cheap...
Somebody get that guy an ambulance!
That would be the first port I would firewall off...
Brings up interesting ideas of ways to prank your friends & enemies though.
0100 lea edi, dma://foo.example.com:b8000h
0103 mov al, 65
0105 mov ecx, 2000
010a rep stosb
010b jmp 100
g=100
Microsoft products have had this "feature" for a while now. Esp. IIS.
It's very interesting that using memory over the network is very much the same problem as cache coherency amongst processors. If you have multiple processors, you don't want to have to go out to the slow memory when the data you want is in your neighbors cache... so perhaps you grab it from the neighbor's cache.
Similarly, if you have many computers on a network, and you are out of RAM, and your nighbor has extra RAM, you don't want to page out to your slow disk when you can use your neighbor's memory.
NUMA machines are somewhere in between these two scenarios.
There are lots of problems: networks aren't very reliable, there's lots of network balancing issues, etc. But it's certainly interesting research, and can be useful for the right application, I guess.
Disk is slow, though... memory access time is measured in ns, disk access time is in ms... that's a 1,000,000x difference. So paging to someone else's RAM over the network can be more efficient.
I don't have any good papers handy, but I'm sure you can google for some.
-- Erich
Slashdot reader since 1997
Servers will very soon be equiped with Infiniband (http://www.infinibandta.org/). Infiniband has dedicated support for RDMA. This includes efficient key mechanisms, which minimize operating system involvement (which would be context switches each time) and low latency. Bandwidth available right now is 2.5 GBit/s and higher bandwidth can be anticipated very soon.
Security does not mean 100% exploit-proof, it means it secures your information/services given certain desired lengths of protection and certain operating conditions.
While M$ is probably not going to get this one right, it doesn't mean that someone can't. This *is* a desirable feature for some applications, and it is possible to make a secure environment (where secure is defined for the application), and make it seamless as well. That is the whole goal of network security professionals.
If anything, the fact that people already know what kinds of "old" exploits this may be vulnerable to, it means that we are already headed in the right direction.
RDMA article
Um... easier said than done there, hotshot.
When a program asks for memory there's a reasonable amount of loops it has to go through in the processor to get the memory, because the processor manages memory. Making a program that toys with memory over the internet wouldn't be slightly exciting.
DMA channels let something, usually a video card, sound card, IDE bus, etc. do what it needs to do with the system's memory without bothering the processor. The speed gained by not bothering the processor when accessing memory is what makes UltraDMA hard drives so fast, video cards accelerated (in addition to a lot of other l337 tricks), etc.
Now, you take a cluster, connected via gigabit network, in which each computer can directly access each other's memory as opposed to using a program to do it that just takes the target processor's cycles. THAT is slightly exciting.
FreeBSD already supports gdb over firewire using
the firewire bridge ability to DMA to/from any
location of memory. Very handy for remote kernel
debugging.
First, what the headline would have you believe has been invented is making it appear as though the RAM of one machine is really the RAM of another machine. This technology has been around and used for quite some time in clustered/distributed/parallel computing communities since at least the 1980s.
If you look at a brief summary of the spec, http://www.rdmaconsortium.org/home/PressReleaseOct 30.pdf, you'll find that all that's happening is that more of the network stack's functionality has been pushed into the NIC. This prevents the CPU from hammering both memory and the bus as it copies data between buffers for various layers of the networking stack.
I'll also note that the networking code in the linux kernel was extensively redesigned to do minimal (and usually no) copying between layers, thereby providing very little advantage of pushing this into hardware.
Please, folks, don't drink and submit!
This article defines NUMA as
which seems to cover all of this.1. ssh root@remote-machine /proc/kcore in remote-machine
2. read from and write to
So where is the use of that? And shared memory emulation over a network is also a decades old technology.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
Allowing one to access the memory of a remote computer over an IP network. Several programs have presented this useful feature including BIND DNS server, Sendmail MTA and of course MS IIS web service. The technology is called "buffer overflow" and has been used by many individuals for "fun and profit"^H^H^H^H^H^H^H^H^H^H their computing needs. The ultimate guide to using this great feature has been seen here
Scott McNealy said that, but the vision was implemented by others. CMU's Mach (1985), Andrew Tanenbaum's Amoeba (1986), and Plan 9 (1987) were OSes that made a network into a computer.
To be fair, Sun does have ChorusOS , but that seems to have died the death (i.e. gone Sun Public Source) despite Scott's best intentions.
Sigmentation fault - core dumped
so, computer 1 asks for a memory address from computer 2, and can then read or write to it by sending back a command.
Operating system mediated memory protection might be an issue here... Sane operating systems at least check to see whether Application 1 actually owns the bit of memory it's trying to read/write before letting it chew over memory that actually belongs to Application 2. Just letting some application read and write any memory is a recipe for disaster that sensible OSes have avoided for a long time...
The proc device serves a two-level directory structure. The first level contains numbered directories corresponding to pids of live processes; each such directory contains a set of files representing the corresponding process.
The mem file contains the current memory image of the process. A read or write at offset o, which must be a valid virtual address, accesses bytes from address o up to the end of the memory segment containing o. Kernel virtual memory, including the kernel stack for the process and saved user registers (whose addresses are machine-dependent), can be accessed through mem. Writes are permitted only while the process is in the Stopped state and only to user addresses or registers.
The read-only proc file contains the kernel per-process structure. Its main use is to recover the kernel stack and program counter for kernel debugging.
The files regs, fpregs, and kregs hold representations of the user-level registers, floating-point registers, and kernel registers in machine-dependent form. The kregs file is read-only.
The read-only fd file lists the open file descriptors of the process. The first line of the file is its current directory; subsequent lines list, one per line, the open files, giving the decimal file descriptor number; whether the file is open for read (r), write, (w), or both (rw); the type, device number, and qid of the file; its I/O unit (the amount of data that may be transferred on the file as a contiguous piece; see iounit(2)), its I/O offset; and its name at the time it was opened.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
"Imagine a Beo-(clobber mangle clobber mangle)..$%@$%@$@%$!"
-jc
I work at InfiniCon Systems and we do a lot of InfiniBand related things. We have switching, adapters, and connections from IB to gigabit ethernet and IB to fibre channel. See http://www.infinicon.com/.
The real next steps for IB is 12X (30 Gb) and on mother board IB. 12 X is in development. Currently, IB adapters are limited by the PCI-X slot they sit in. PCI-X DDR and PCI Express should help, but just having it on the mother board and throwing PCI out would be interesting. Small form factor clusters would really benefit to just having memory, CPU, and an HCA.
-- soldack
The amount of book-keeping required to keep this thing going makes it a non-starter. And as for scale'ing. Forget it.
The sad truth is that it's common knowledge that this is the least efficient principle for distributed systems. This technique is usually the fall-back position if nothing else works.
TCAP-Abort
That would be possible, but do you think it's faster than GLX? It both goes over a network first... and I guess the security implications of such an attempt are more serious than the ones present in GLX implementations....
If a train station is a place where a train stops, what's a workstation?
First off, this is not a network shared memory scheme. RDMA could be used to implement one very efficently though.
It will not allow arbitary access to your memory space. In fact, it would prevent a great number of buffer overflow exploits
The best analogy is the difference between PIO and UDMA modes of your IDE devices (or any device). This is all about offloading work from your CPU. It is moving the TCP/IP stack from the kernel to the network card for a very specific protocol.
Here's how RDMA would work layered over (under?) HTTP.
- browser creates GET request in a buffer
- browser tells NIC address of buffer and who to send it to.
- NIC does a DMA transfer to get buffer. OS not involved
- NIC opens RDMA connection to webserver
- server NIC has already been told by the webserver what buffer it should put incoming data
- webserver unblocks once data in buffer and parses it.
- webserver creates HTML page in second buffer.
- webserver tells server NIC to do a RDMA transfer from buffer to browser host
- client NIC takes data and puts it in browser buffer
- browser unblocks parse HTML and displays it.
All of this with minimal interaction with the TCP/IP stack. RDMA just allows you to move a buffer from one machine to another without alot of memory copying in the TCPIP stack.
In fact, the RDMA protocol could be emulated completely in software. It would probably have a small overhead verses current techniques but would still be useful. Just imagine real RDMA on the server and emulated RDMA on the clients (cheaper NIC). The server has less overhead and most clients have cycles to spare!
Sounds like a great tool for clustering, especially considering that the new motherboards have gigabit ethernet and a link directly to the northbridge/MCH.
There's just one problem with that... ethernet (even GigE) is *not* a good connection for clustering. Sure, the bandwidth is semi-decent, but the *latency* is the killer. Instead of a processer waiting a number of nanoseconds for memory (as with local memory), it'll end up waiting as much as milliseconds. That may not sound like much, but from nano to micro you jump seven orders of mangitude!
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
..."See, we TOLD you it was a feature!" Microsoft will also sue the researchers working on this project, citing they Innovated this years ago.
CAn'T CompreHend SARcaSm?
We can also start wrapping processor instructions in XML and transmit them via SOAP, in order to create more interoperability between different machine architectures! Remember, we already have IP over XML :-)
That's what the whole thing sounds like to me...
Don't drink and su! antidisestablishmentariazationally
I have seen dell 2650s hit over 800 Megabytes (6.4 Gb) per second running MPI over InfiniBand using large buffer sizes. The limit is pretty much the PCI-X 133 Mhz interface we are on. I suspect that with PCI-X DDR and PCI Express, we will be able to get a lot closer to 10 Gbit.
-- soldack
Well, sort of...
"Back in the day", I wrote a virtual memory handler for my Amiga's accelerator card (which had a 68030 and MMU). Meanwhile, some friends of mine had developed this networking scheme that involved wiring the serial ports of our Amiga's together in a ring, which allowed us to have a true network without network cards.
Then came the true test: I configured my virtual memory to use a swapfile located in a friend's RAM-disk (he had way more memory than I did), fired up an image editor, opened a large image, and lo and behold: I was swapping at a whopping 9600 bytes per second! The fact that every packet had to pass through multiple other machines (because of the ring-nature of the network) didn't make it any faster either...