Slashdot Mirror
Remote Direct Memory Access Over IP
doormat writes "Accessing another computer's memory over the internet? It might not be that far off. Sounds like a great tool for clustering, especially considering that the new motherboards have gigabit ethernet and a link directly to the northbridge/MCH."
Not to mention easy access to sensitive information in emails, documents, and PIMs that the user currently is running and are resident in memory.
Seriously though... this is where Scott McNealy's vision of "The Network is the Computer" comes even closer to reality.
S
The security implications are staggering.
How do we lobby for port number 31337 for the RDMA protocol?
How small a thought it takes to fill a whole life
This feature has been available for a while now, but using a dedicated link rather than IP. Sun call it Remote Shared Memory and it's mainly used for database clusters.
> Microsoft ultimately is expected to support RDMA
> over TCP/IP in all versions of Windows
Can you see it coming? The ultimate Windows root exploit!! Hmm... I guess someone has to go tell them. Othervise they won't notice it until it's too late...
Seriously, how do you dare to enable this kind of access?!?
I tried something like this a while ago -- I wanted to mount an NFS-exported file via loopback and use it as swap.
The file in question actually resided in a RAM drive on another machine on the LAN.
I couldn't get it to work in the 45 minutes or so I messed around with it. I'm not sure if Linux was unhappy using an NFS-hosted file for swap, or what exactly the problem was, but I did get some funny looks from people to whom I explained the idea (ie, to determine whether the network would be faster than waiting for my disk-based swap).
Of course, this was back when RAM wasn't cheap...
Somebody get that guy an ambulance!
That would be the first port I would firewall off...
Brings up interesting ideas of ways to prank your friends & enemies though.
Microsoft products have had this "feature" for a while now. Esp. IIS.
It's very interesting that using memory over the network is very much the same problem as cache coherency amongst processors. If you have multiple processors, you don't want to have to go out to the slow memory when the data you want is in your neighbors cache... so perhaps you grab it from the neighbor's cache.
Similarly, if you have many computers on a network, and you are out of RAM, and your nighbor has extra RAM, you don't want to page out to your slow disk when you can use your neighbor's memory.
NUMA machines are somewhere in between these two scenarios.
There are lots of problems: networks aren't very reliable, there's lots of network balancing issues, etc. But it's certainly interesting research, and can be useful for the right application, I guess.
Disk is slow, though... memory access time is measured in ns, disk access time is in ms... that's a 1,000,000x difference. So paging to someone else's RAM over the network can be more efficient.
I don't have any good papers handy, but I'm sure you can google for some.
-- Erich
Slashdot reader since 1997
Servers will very soon be equiped with Infiniband (http://www.infinibandta.org/). Infiniband has dedicated support for RDMA. This includes efficient key mechanisms, which minimize operating system involvement (which would be context switches each time) and low latency. Bandwidth available right now is 2.5 GBit/s and higher bandwidth can be anticipated very soon.
Um... easier said than done there, hotshot.
When a program asks for memory there's a reasonable amount of loops it has to go through in the processor to get the memory, because the processor manages memory. Making a program that toys with memory over the internet wouldn't be slightly exciting.
DMA channels let something, usually a video card, sound card, IDE bus, etc. do what it needs to do with the system's memory without bothering the processor. The speed gained by not bothering the processor when accessing memory is what makes UltraDMA hard drives so fast, video cards accelerated (in addition to a lot of other l337 tricks), etc.
Now, you take a cluster, connected via gigabit network, in which each computer can directly access each other's memory as opposed to using a program to do it that just takes the target processor's cycles. THAT is slightly exciting.
FreeBSD already supports gdb over firewire using
the firewire bridge ability to DMA to/from any
location of memory. Very handy for remote kernel
debugging.
First, what the headline would have you believe has been invented is making it appear as though the RAM of one machine is really the RAM of another machine. This technology has been around and used for quite some time in clustered/distributed/parallel computing communities since at least the 1980s.
If you look at a brief summary of the spec, http://www.rdmaconsortium.org/home/PressReleaseOct 30.pdf, you'll find that all that's happening is that more of the network stack's functionality has been pushed into the NIC. This prevents the CPU from hammering both memory and the bus as it copies data between buffers for various layers of the networking stack.
I'll also note that the networking code in the linux kernel was extensively redesigned to do minimal (and usually no) copying between layers, thereby providing very little advantage of pushing this into hardware.
Please, folks, don't drink and submit!
This article defines NUMA as
which seems to cover all of this.The proc device serves a two-level directory structure. The first level contains numbered directories corresponding to pids of live processes; each such directory contains a set of files representing the corresponding process.
The mem file contains the current memory image of the process. A read or write at offset o, which must be a valid virtual address, accesses bytes from address o up to the end of the memory segment containing o. Kernel virtual memory, including the kernel stack for the process and saved user registers (whose addresses are machine-dependent), can be accessed through mem. Writes are permitted only while the process is in the Stopped state and only to user addresses or registers.
The read-only proc file contains the kernel per-process structure. Its main use is to recover the kernel stack and program counter for kernel debugging.
The files regs, fpregs, and kregs hold representations of the user-level registers, floating-point registers, and kernel registers in machine-dependent form. The kregs file is read-only.
The read-only fd file lists the open file descriptors of the process. The first line of the file is its current directory; subsequent lines list, one per line, the open files, giving the decimal file descriptor number; whether the file is open for read (r), write, (w), or both (rw); the type, device number, and qid of the file; its I/O unit (the amount of data that may be transferred on the file as a contiguous piece; see iounit(2)), its I/O offset; and its name at the time it was opened.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter