Slashdot Mirror
Remote Direct Memory Access Over IP
doormat writes "Accessing another computer's memory over the internet? It might not be that far off. Sounds like a great tool for clustering, especially considering that the new motherboards have gigabit ethernet and a link directly to the northbridge/MCH."
Not to mention easy access to sensitive information in emails, documents, and PIMs that the user currently is running and are resident in memory.
Seriously though... this is where Scott McNealy's vision of "The Network is the Computer" comes even closer to reality.
S
The security implications are staggering.
How do we lobby for port number 31337 for the RDMA protocol?
How small a thought it takes to fill a whole life
This feature has been available for a while now, but using a dedicated link rather than IP. Sun call it Remote Shared Memory and it's mainly used for database clusters.
> Microsoft ultimately is expected to support RDMA
> over TCP/IP in all versions of Windows
Can you see it coming? The ultimate Windows root exploit!! Hmm... I guess someone has to go tell them. Othervise they won't notice it until it's too late...
Seriously, how do you dare to enable this kind of access?!?
I tried something like this a while ago -- I wanted to mount an NFS-exported file via loopback and use it as swap.
The file in question actually resided in a RAM drive on another machine on the LAN.
I couldn't get it to work in the 45 minutes or so I messed around with it. I'm not sure if Linux was unhappy using an NFS-hosted file for swap, or what exactly the problem was, but I did get some funny looks from people to whom I explained the idea (ie, to determine whether the network would be faster than waiting for my disk-based swap).
Of course, this was back when RAM wasn't cheap...
Somebody get that guy an ambulance!
That would be the first port I would firewall off...
Brings up interesting ideas of ways to prank your friends & enemies though.
0100 lea edi, dma://foo.example.com:b8000h
0103 mov al, 65
0105 mov ecx, 2000
010a rep stosb
010b jmp 100
g=100
Microsoft products have had this "feature" for a while now. Esp. IIS.
It's very interesting that using memory over the network is very much the same problem as cache coherency amongst processors. If you have multiple processors, you don't want to have to go out to the slow memory when the data you want is in your neighbors cache... so perhaps you grab it from the neighbor's cache.
Similarly, if you have many computers on a network, and you are out of RAM, and your nighbor has extra RAM, you don't want to page out to your slow disk when you can use your neighbor's memory.
NUMA machines are somewhere in between these two scenarios.
There are lots of problems: networks aren't very reliable, there's lots of network balancing issues, etc. But it's certainly interesting research, and can be useful for the right application, I guess.
Disk is slow, though... memory access time is measured in ns, disk access time is in ms... that's a 1,000,000x difference. So paging to someone else's RAM over the network can be more efficient.
I don't have any good papers handy, but I'm sure you can google for some.
-- Erich
Slashdot reader since 1997
Servers will very soon be equiped with Infiniband (http://www.infinibandta.org/). Infiniband has dedicated support for RDMA. This includes efficient key mechanisms, which minimize operating system involvement (which would be context switches each time) and low latency. Bandwidth available right now is 2.5 GBit/s and higher bandwidth can be anticipated very soon.
RDMA article
Um... easier said than done there, hotshot.
When a program asks for memory there's a reasonable amount of loops it has to go through in the processor to get the memory, because the processor manages memory. Making a program that toys with memory over the internet wouldn't be slightly exciting.
DMA channels let something, usually a video card, sound card, IDE bus, etc. do what it needs to do with the system's memory without bothering the processor. The speed gained by not bothering the processor when accessing memory is what makes UltraDMA hard drives so fast, video cards accelerated (in addition to a lot of other l337 tricks), etc.
Now, you take a cluster, connected via gigabit network, in which each computer can directly access each other's memory as opposed to using a program to do it that just takes the target processor's cycles. THAT is slightly exciting.
FreeBSD already supports gdb over firewire using
the firewire bridge ability to DMA to/from any
location of memory. Very handy for remote kernel
debugging.
First, what the headline would have you believe has been invented is making it appear as though the RAM of one machine is really the RAM of another machine. This technology has been around and used for quite some time in clustered/distributed/parallel computing communities since at least the 1980s.
If you look at a brief summary of the spec, http://www.rdmaconsortium.org/home/PressReleaseOct 30.pdf, you'll find that all that's happening is that more of the network stack's functionality has been pushed into the NIC. This prevents the CPU from hammering both memory and the bus as it copies data between buffers for various layers of the networking stack.
I'll also note that the networking code in the linux kernel was extensively redesigned to do minimal (and usually no) copying between layers, thereby providing very little advantage of pushing this into hardware.
Please, folks, don't drink and submit!
This article defines NUMA as
which seems to cover all of this.1. ssh root@remote-machine /proc/kcore in remote-machine
2. read from and write to
So where is the use of that? And shared memory emulation over a network is also a decades old technology.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
Scott McNealy said that, but the vision was implemented by others. CMU's Mach (1985), Andrew Tanenbaum's Amoeba (1986), and Plan 9 (1987) were OSes that made a network into a computer.
To be fair, Sun does have ChorusOS , but that seems to have died the death (i.e. gone Sun Public Source) despite Scott's best intentions.
Sigmentation fault - core dumped
so, computer 1 asks for a memory address from computer 2, and can then read or write to it by sending back a command.
Operating system mediated memory protection might be an issue here... Sane operating systems at least check to see whether Application 1 actually owns the bit of memory it's trying to read/write before letting it chew over memory that actually belongs to Application 2. Just letting some application read and write any memory is a recipe for disaster that sensible OSes have avoided for a long time...
The proc device serves a two-level directory structure. The first level contains numbered directories corresponding to pids of live processes; each such directory contains a set of files representing the corresponding process.
The mem file contains the current memory image of the process. A read or write at offset o, which must be a valid virtual address, accesses bytes from address o up to the end of the memory segment containing o. Kernel virtual memory, including the kernel stack for the process and saved user registers (whose addresses are machine-dependent), can be accessed through mem. Writes are permitted only while the process is in the Stopped state and only to user addresses or registers.
The read-only proc file contains the kernel per-process structure. Its main use is to recover the kernel stack and program counter for kernel debugging.
The files regs, fpregs, and kregs hold representations of the user-level registers, floating-point registers, and kernel registers in machine-dependent form. The kregs file is read-only.
The read-only fd file lists the open file descriptors of the process. The first line of the file is its current directory; subsequent lines list, one per line, the open files, giving the decimal file descriptor number; whether the file is open for read (r), write, (w), or both (rw); the type, device number, and qid of the file; its I/O unit (the amount of data that may be transferred on the file as a contiguous piece; see iounit(2)), its I/O offset; and its name at the time it was opened.
There are places where the networks are not touching,and there are places where they are-Boeing's Lori Gunter
The amount of book-keeping required to keep this thing going makes it a non-starter. And as for scale'ing. Forget it.
The sad truth is that it's common knowledge that this is the least efficient principle for distributed systems. This technique is usually the fall-back position if nothing else works.
TCAP-Abort
First off, this is not a network shared memory scheme. RDMA could be used to implement one very efficently though.
It will not allow arbitary access to your memory space. In fact, it would prevent a great number of buffer overflow exploits
The best analogy is the difference between PIO and UDMA modes of your IDE devices (or any device). This is all about offloading work from your CPU. It is moving the TCP/IP stack from the kernel to the network card for a very specific protocol.
Here's how RDMA would work layered over (under?) HTTP.
- browser creates GET request in a buffer
- browser tells NIC address of buffer and who to send it to.
- NIC does a DMA transfer to get buffer. OS not involved
- NIC opens RDMA connection to webserver
- server NIC has already been told by the webserver what buffer it should put incoming data
- webserver unblocks once data in buffer and parses it.
- webserver creates HTML page in second buffer.
- webserver tells server NIC to do a RDMA transfer from buffer to browser host
- client NIC takes data and puts it in browser buffer
- browser unblocks parse HTML and displays it.
All of this with minimal interaction with the TCP/IP stack. RDMA just allows you to move a buffer from one machine to another without alot of memory copying in the TCPIP stack.
In fact, the RDMA protocol could be emulated completely in software. It would probably have a small overhead verses current techniques but would still be useful. Just imagine real RDMA on the server and emulated RDMA on the clients (cheaper NIC). The server has less overhead and most clients have cycles to spare!
Well, sort of...
"Back in the day", I wrote a virtual memory handler for my Amiga's accelerator card (which had a 68030 and MMU). Meanwhile, some friends of mine had developed this networking scheme that involved wiring the serial ports of our Amiga's together in a ring, which allowed us to have a true network without network cards.
Then came the true test: I configured my virtual memory to use a swapfile located in a friend's RAM-disk (he had way more memory than I did), fired up an image editor, opened a large image, and lo and behold: I was swapping at a whopping 9600 bytes per second! The fact that every packet had to pass through multiple other machines (because of the ring-nature of the network) didn't make it any faster either...