Slashdot Mirror
SuSE and Siemens Release Linux Memory Extension
hussar noted that SuSE and Siemens have developed a memory extension that will allow Linux to use up to 4GB of memory. Linus has reportedly approved its inclusion in kernel 2.3.15. The strangest part is excite has taken to posting about Linux Kernel Patches. Pretty crazy stuff.
Not on my desktop sadly, but the latest UNIX server we bought here has 2GB of memory and that didn't cause too much excitement.
That kind of memory is pretty common place - and Linux needs to support it.
Has anyone played with SGI's journaled file system for Linux? I am glad to see this major shortcoming being looked at.
Thinking of Maud you forget everything else. -- hack v1.0.3 [] Who was that Maud person anyway? -- nethack v3.1.0
Why did Linus change his mind about "kludged" memory models? Is this patch elegant, or was it a pragmatic decision?
Hello,
:) In the right crowds, I'm sure that's an important feature. :)
/just/ the right circumstances.
:)
Yes, it was key to supporting IRIX.
What's important to note and understand is that everyone, and I mean EVERYONE, has a differnet idea of what is and isn't a good feature. And believe me, I get a fair amount of feedback on my articles and I rarely can please everyone. There's always the person that just isn't happy until the PC internal speaker is supported, or until the kernel can do ISA PnP (it can now), or until it can do some other completely bizarre task that seems trivial to us at first, but them you realize how critical this can be under
That's what makes Linux great: we have such a selection of great things that we have to please everyone a little; but we'll never please everyone entirely.
As for me, this 4G thing seems pretty silly.
Joe
Part of the FreeBSD kernel are actually under GNU...
Most certainly not!
Berlin-- http://www.berlin-consortium.org
DNA just wants to be free...
HP-UX 11, being a 64-bit OS, breaks the 4 Gig limit, but IIRC the actual limit varies a bit between "patch levels". They're moving it upwards all the time, I seem to recall.
--
Linux user since early January 1992.
wow that's a user of my irc client! ;)
just a typo...
The issue may be one of whether the extra 4 physical-address lines are actually connected to the world off the chip; that might not be the case on all P6-series processors, even though the processor on the die might contain all the hardware necessary to put non-zero values on those lines.
Yes, NT (unless you turn on the right boot flag, which might work even in non-enterprise editions) has 2GB of the virtual address space of the machine available to user-mode code and 2GB for kernel-mode code, just as had been the case for Linux all along.
However, I'm not sure that limited you to 2GB of physical RAM on NT, as NT might not require that all of physical memory be mapped into the kernel's address space.
Linux has, for a long time, supported "unlimited user space" in the sense that user-mode code can map stuff into and out of its address space, so that the no-more-than-4GB window you get on a processor with 32-bit linear addresses (e.g., an x86 processor) can be moved around a more-than-4GB set of data. The patch doesn't change that.
However, that's true of most OSes that run on x86 processors, these days.
You don't need to use segmentation to get at >4GB of physical memory.
In fact, segmentation doesn't even help, given that the x86 MMU maps 48-bit segmented addresses to 32-bit linear addresses before running said addresses through the page table and translating them to 32-bit or 36-bit physical addresses.
However, for any single process to access more than 4GB, it does have to do something that amounts to bank switching, i.e. map stuff into and out of its no-more-than-4GB linear address space as necessary; the same applies to kernel-mode code and the kernel-mode portion of the address space.
NT 4.0, or NT 5.0^H^H^H^H^H^HW2K? I don't think NT 4.0 supports more than 4GB of physical memory, but W2K, at least in the DataCenter Server edition, will.
See, for example, this note on "Address Windows Extensions and Windows 2000 DataCenter Server", and this press-release-like document which indicates that this is new in W2K and not in NT 4.0, but also seems to imply that memory above 4GB will be available to the page pool (I'd seen stuff that gave me the impression that it would be wired-down memory that had to be specially mapped into a process's address space, but perhaps that's not the case).
No, it doesn't. It just has a 32-bit base linear address, a 20-bit length (which can either be in units of bytes or 4K pages), and a bunch of other flags. A 48-bit far address gets translated to a 32-bit linear address, and that is what gets translated to a 32-bit or 36-bit physical address via the page table.
See (this page that has a link to the PDF document) Intel Architecture Software Developers Manual Volume 3: System Programming for the full story.
thanks for the update - I think. Would be better to hear that it was working great and I should switch to it now. Not having a decent journaled FS is a major downside to Linux in my opinion. Who wants to put a lot of disk on a server and then have it take 3 weeks (ok a slight exaggeration) to fsck. Not me.
Actually I do have a couple of servers with a small amount of disk on them (16GB each as I recall) and the main problem is how slow NFS is.
still kernal level appletalk support is nice. Is all swings and roundabouts.
Thinking of Maud you forget everything else. -- hack v1.0.3 [] Who was that Maud person anyway? -- nethack v3.1.0
Windows98 supports more than a gig.
God, I like FreeBSD (maybe even a bit more than I like Linux, in fact), and I use it in both my gateway and my laptop, but looking at some of its supporters, maybe they should rename it FreeBSE and be done with it...
I really expected better from the user base than a bad case of Linux Envy...
"Be nice, veer left, and never stop thinking" Iain Banks - Walking On Glass
yep. for all the fears people had (and FUD that got spread) about the commercialization of Linux, here we see the Open Source (and specifically the GPL) magic working just right. Large companies like Siemens suddently find it (financially) worthwhile to contribute to an open project. this is a huge ball that got rolling; it's not just a bunch of hackers competing with MS (and Solaris and ...), now it's large companies and their resources, too. and to think that some MS drone was quoted saying that "the linux hype has peaked" just a few days ago... boy he's in for a surprise :)
Yes, P6's have 36-bit physical memory addressing; this does not mean that virtual (linear) addresses are 36 bits. (Some other 32-bit processors, e.g. 32-bit SPARC processors or processor modules with the SPARC Reference MMU, also supported more than 32 bits of physical address. Heck, PDP-11s with MMUs often supported either 18 or 22 bits of physical address, even though they only supported 16 - or 17, sort of, if you count split I&D space - bits of virtual address.)
Dynix, on Sequent's servers, might - they support much more than 4GB of physical memory, and if the NUMA part of "NUMA-Q" just means that accessing "other people's" memory is slower, not that you have to play bus-mapping games to get at it, they may use the 36-bit features to do that - although I thought I saw something indicating that some of their machines supported 128GB, which requires 37 bits.
what's the point of giving 2GB to the OS? I know the NT kernel is obsese compared to Linux, but still... that sounds like some serious overkill to me, and using almost half of your machine's memory as page cache (in Linux terms, I'm sure NT has an equivalent that's calle dsomething else) doesn't strike me as a very balanced use of a large box.
on a completely unrelated point, I wonder if this new 4GB support is a compile-time option, or if it can be somehow enabled at runtime without a performance hit when it's disabled. that would be neat, but it does sound a bit improbable for a change like this.
How will this effect IA/64 Linux? Doesn't 64bit processors allow up to 1TB RAM...Will this help there project out or are they using something else?
yeah it does. it has to involve mapping memory zones on demand instead of keeping the same lot of things mapped at the same time. the good old equation that 32bit = 64Gb doesn't mean that you can put 4Gb of RAM in your memory and use it: you need virtual addresses for all kinds of pages that are either paged out or mapped to disk and set to load on demand. and Linux so far has been mapping kernel memory while userlevel code runs too, with just the permissions changed, b/c it's cheaper to do this than to remap it for every syscall and unmap it on the way back. I have no idea how much of this scheme this new patch changes.
woohoo! moderated up! i know, i know, pointless comment, moderate me down i dare ya. i'd like to point out however (before you do) that each app gets 2GB and not all apps sharing 2GB in userspace like my previous otherwise wonderful post *cough* seems to imply. somebody else posted somewhere that the PIII has 36bit addressing letting it potentially use 4 terabytes of RAM. who wants to help me write the linux kernel patch for that :)
Satan, oscillate my metallic sonatas.
2 GB is not much in the company world, our 4 years old HP9000 SAP/Oracle server has 2 GB, our new one has 8 GB memory.
We need 3rd party support by manufacturers for products that they sell to make linux more profesisonal
Shine on, you crazy diamond.
...and Andrea works for SuSE, you ... (ok, I don't wanna get downgraded for this psoting)
--
Michael Hasenstein
http://www.csn.tu-chemnitz.de/~mha/
My best example of this was when I saw a rehash of Pravenich's 2.4 kernel thing on a tech news site - they'd duplicated his mistakes, and were brainwashing the masses with the stuff! ;)
The GNU thing is certainly untrue.
And *when did I ever say bsd has no drivers*.
-- dieman - Scott Dier
That would be funnier if what you said about MS were true and didn't contradict.
Office 2000 can now run on Linux with 4GB.
W2K only supports 128M?
None since w2k is no longer supported on non-intel architecture (Compaq recently stopped NT development on Alpha)
core dumped.
I'm interested if anybody knows what are the limits of Win 9x/NT, BEOS, *BSD, MacOS,...
Linux rox!!!
But how many poeple have the experence using, that's say, 2GB memory?
A big company does something, they send out a press release. Nothing new there, it is just that it is about Linux.
Now, that someone has come up with a bigmem patch that Linus will live with, THAT is news!
This is a big deal for some users. A real shocker will be if someone comes up with a patch to use the 36 bit addressing on the P6 cpu, for up to 64GB ram on Intel machines.
Plato seems wrong to me today
Its not unusual really. It fits into excite's usual MO.
:)
For the non-journalists/editors in the crowd, I'll point out that this story was A) from a newswire service (check the DATELINE--ITS IN ALL CAPS LIKE THIS) and B) the story was obviously a press release (witness the "About SuSE" section of the article -- an obvious shameless plug for the company putting out the press release).
Excite's news section consists almost entirely of newswires because its cheaper than writing your own stories and they generally don't require too much editing (because they are written by professionals who know what newspaper editors like to see in an article in terms of structure, content, syntax etc.)
FWIW, I used to design/edit/publish several newspapers for non-profit veterans groups like the AMVETS and PVA...at least until I got a "real" job in the IT field.
My journal has hot
I find it somewhat telling that the article didn't mention Andrea Arcangeli and Gerhard Wichert, workin at SuSe and Siemens respectively and wrote this patch pretty much dual-handedly. I suppose with the corpratization of linux, the companies are more important now than the actual people who make linux what it is. At the very least, a link to Andrea's archived message on l-k would give credit where it is due.
I vaguely remember reading that Solaris now supports it, although finding detailed technical information on Sun's shiny new Marketing-Driven(TM) Web site looks as if it'd demand more patience than I have.
It's probably a question of whether any "commodity" x86 machines support it; if not, then Solaris for Intel, and UnixWare, may not support it either.
Not in the case of P6-core machines; segmentation turns a 48-bit segmented address into a 32-bit linear address. They're using paging, instead, i.e. the page table entries, in one of the 36-bit-physical-address modes, generate more than 32 bits of physical address from 32 bits of (linear) virtual address.
Any one process would have to map stuff into and out of its address space to use more than 4GB (or 4GB minus what kernel-mode code takes) of physical memory, but
Kernel-mode code could also map stuff into and out of its part of the address space.
This is great work. Just think what will happen when the SGI big memory project is ready. Check out http://oss.sgi.com/projects/bigmem/. Wow. Two big memory solutions. I just don't know which to choose. Oh. Hang on. I don't have four *fricking* gigabytes of RAM....
Oh man don't even compare the way linux handles mem compared to BSD's. The code the BSD's use for enormous amounts of memory is more mature. Also, linux has problems withn neighbor table overflows.
- The hardware vendor gets to show off their hardware.
- The hardware vendors gains some purely positive publicity and goodwill from the community
- Everyone else benefits from the contribution.
Plus, a significant contribution can be merely the seed for further developments, since anyone in the world can read the patch and contribute their own. It's a win-win situation all around.However, it appears that most of the 64 bit architectures (Alpha and UltraSPARC) don't use 64-bit addressing. They use something more like 39-bit or 42-bit -- something in that general neighborhood.
:-)
39-bit = 549755813888 = 512 gigabytes.
42-bit = 4398046511104 = 4 terabytes.
45-bit = 35184372088832 = 32 terabytes.
You only need 4 terabytes of memory if you're altavista or fedex and you want to keep your entire database in memory at once. Otherwise a couple-hundred gigabytes should do you fine.
(I wish I could remember the specific figure...)
I believe the 36 bit addressing only allows you to use memory beyond 32 bits for paging; buffers, cache, etc. Logical addressing (program addresses) is still limited to 2GB or 3GB or whatever. So if you put 64G on your quad Xeon factory heater (tm), programs still max out at 4G. But you get plenty of disk buffers....
--
Infuriate left and right
2GB user, 2GB system, though you can use an MS supported hack to get 3GB user, 1GB system.
I'm happy about this too, but just to play devil's advocate, this whole major corporations contributing to the kernel could backfire, you know. Couple of scenarios, from most to least benign: 1) hardware company contributes driver code 2) software company contributes code to provide connectivity to their package 3) software company contributes code to provide connectivity to this package. only this time, it precludes the use of a competitor's package 4) software company tries to contribute something to the kernel to provide priveleged access for their software. linus blocks it. company goes off in a huff, forks the kernel to include their stuff, and tells their large installed user base (who are locked to this software company's product) that they can only use their version of the kernel 5) software company from #4 is { Sun, Microsoft, etc } Food for thought.
...version 6.2 is finally available online at
ftp://ftp.suse.com/pub/suse/i386/
Enjoy.
(Anyone know where one can find the ISO's?)
Your Brain + EEG + LEGO Robots = Brainstorms
I would hate to call you a "stupid idiot" but the point is:
WE DONT CARE ABOUT 4GB OF RAM.
We care about our AudioPCI cards, out ultra tnt2 boards, the newest-and-greatest technology.
And.. we like the GNU license because we think it promotes people to add to the kernel, whereas the BSD license lets people take too much from the code and put it into EvilSoft coprate apps.
My $.02
-- dieman - Scott Dier
It doesnt have hardware support, but it does have ram support =) FreeBSD: A Better Windows NT Than Linux
Just a quick note to everybody. It is not about being able to use 4GB of physical memory. It is to enable process to use more than 2GB of memory. Traditional Linux memory model to is split the lower half for kernel memory and upper half for user memory. To check if the pointer is pointing to a kernel memory you just need to check its MSB.
test %eax,0x8000000
je user_mem_label
I think they have worked on 3GB prior to this. SOrry, been a while since I checked the kernel lists.
Anyhow, this is only of practical use to database developers. maybe some but not many. In any case, you might as well use a 64-bit architecture.
Hasdi
...from Siemens, not a media story.
Well, seeing as chips after the Pentium Pro can access up to 64 GB, they seem to have a ways to go before the software is there to support the full hardware. Ah well, I only have 128 MB, so I guess I have a long ways to go too. :)
IIRC, a swap partition or swap file can only be about 128 MB under linux, with a maximum of 16 swap spaces, leading to a total of 2GB of swap space. Since we can now use 4GB, how are we supposed to allocate enough swap space (I prefer 2 times the physical memory).
Has this annoying restriction of 128MB been removed, or can we use more swap spaces (lets see, 4 GB divided by 128 MB gives way too much swap spaces to be practical), or are we not supposed to use virtual memory any more ?
If the restriction of 128 MB per swap space still exists, is there anybody working on removing this so Linux can become practical for modern computers? If the restriction is removed, is it possible to create a single 8GB swap space ?
Well, from expearance, win 98 wont be stable with 1 gig. I once put 512Meg in a Win 98 system (we had 4 128M DIMMS at the shop, and I just wanted to see). Strange things started to happen, it blue screened quite a bit. It wasn't prity. I would say that you could stick 4 Gigs in a 98 machine, but it wouldent stay running for more then 5 minutes.
I first was pretty excited to see a big corporation like Siemens embrace Linux is a big way. Then i checked out the linux-kernel archives just to discover that the 'Big Mem patch' is only 500 lines long - much of which are #ifdef CONFIG_BIGMEM-s. The 2.3.15 kernel patch itself adds more than 90000 new lines to the kernel! Big Mem is a useful (but apparently quite trivial, probably due to Linux's generic VM architecture) addition, but SuSE really got the scale wrong this time. Does this really deserve a press release when there are much bigger contributions going into the Linux kernel on a daily basis? Is SuSE going to IPO or what?
You're not counting lines of code to determine relative merit, are you?
Would you have pointy hair, by any chance?
640Mb should be enough for anybody!
So what was the difficulty in allowing Linux to use 4G? I understand their was some issue with flushing certain buffers so was this a x86 only issue?
How is it that FreeBSD handles more RAM? Do they merely take a performance hit or do they have better kernel architecture?
Marriage is the "pseudo-ethics" that cloaks the messy truth of sexuality in the raiment of propriety -- it's "Don't Ask,
Umm. Only the Xeon CPUs have the 36-bit address bus (versus 32-bit for the standard PIIs - and probably the standard PIIIs also) allowing addressing up to 64 GB of RAM. Also, using those top 4 address bits takes special OS support, and AFAIK Linux doesn't support that as of yet, unless these Siemens/SuSE kernel mods also allow for that.
Sam: "That was needlessly cryptic."
Max: "I'd be peeing my pants if I wore any!"
Well, don't discount hardware (in)stability. Very often, pc motherboards deal with (lots of) ram, especially in lots of slots, quite poorly.
What type of motherboard were you using?
Try the same eact setup with linux, you'll probably have problems there too.
What am I missing.. I remember ages ago a friend of mine put Linux on 2 4 gig quad processor Xeon boxes which at the time needed kernel patches to have it recognize the rest of the mem but they got it working.. We only run solaris on here (w/ only a gig) but don't have this problem on intel boxes.. but (i'm at home now) but this is what my xconfig 2.0.38 says :: under "General Setup" subsection "Memory Configuration" There are three memory configurations available the standard allows use of just under a gig of ram w/ 3 gigs of virt space per process , the enterprise uses 2 gigs of mem but limits process space to 2 gigs.. the custom option allows you to specify the split, subject to kernel constraints.. Linux/x86 can use up to ~3.4 gigs of physical memory... Anyway it sounded vaguely like the NT kernspace userspace splitting you could set.. I know, no one is reading this anymore :(
Where do you get 64 Gb from 32 bits of address space? It'd be more like 32 Gb (4 GBytes).
Sam: "That was needlessly cryptic."
Max: "I'd be peeing my pants if I wore any!"
A. Arcangeli is a long time kernel developer. So no third part developers.
For the limited class of applications which this patch is designed to address, there is going to be a performance hit. I understand very well that IA-32 is a lower cost platform and for the vast majority of applications is very cost effictive, but with the overhead of remapping memory, what happens to the cost/performance ratio when your overall throughput goes down ? Running a Linux or *BSD app. on an Alpha for example will provide better system throughput as well as larger memory.
Ditto.
I'm surprised to see this sort of thing again. I can remember the first time I saw this kind of stuff, on the old Atari 130xe: 128 Kbytes through 'bank switching'...
chris
San Francisco values: compassion, tolerance, respect, intelligence
Sorry you lose :P
moderate me troll if you like.
it's still true.
That is the weirdest part.
Could somebody explain to me how (or perhaps why) some of the top-of-the-line Intel-based servers claim to support even more than 4 Gigs of ram? The page for the new IBM Netfinity 8500R, for example claims that it supports "Up to 16GB ECC SDRAM". How is that even possible, since I thought 32-bit architecture made 4 Gigs the max addressable limit, period. Am I wrong? Are they simply selling a capacity nobody can use? This has confused me ever since I first heard about it.
Thanks.
Is this the patch SGI has had on their web page
for 4 months?
5) Office 2000 memory requirements now supported by Linux, making port much easier.
4) Enhances sales potential of Windows 2000 -- WINE now able to run W2K under Linux.
3) Yet another fun Linux feature to deny and obfuscate.
2) Can complain before tech-unclued journalists about Linux's memory requirements -- 4G compared to W2K's 128M.
1) Now that Linux supports 4G of RAM, it will be competition on the everyday Joe's desktop, thus making MS-DOJ trial irrelevant.
(Darn Excite, slashdotted again.)
We want endless gardens of data, where the bits can flower, flourish and reproduce. -- Andy Mueller-Maguhn
1024MB Does me just fine ;-)
since linux can't run with more then 960 megs....
"Subtle mind control? Why do all these HTML buttons say 'Submit' ?"
ReadThe ReflectionEngine, a cyberpunk style n
I though they were talking about garden variety IA-32 boxen. Not even close. I went to the siemens website and read about the Primergy server clusters. Now I know why the need the 4 Gigs. This is gonna rock!
it's a good thing that this was an AC post, it's a lot easyer to loose karma then it is to get it back. This is may be the real reason linux dosn't support large memory models.... (or didn't)
"Subtle mind control? Why do all these HTML buttons say 'Submit' ?"
ReadThe ReflectionEngine, a cyberpunk style n
geez, you guys are as bad as the Mac people. Now the linux kernel can do this to. We all know FreeBSD can access 4gb of memory. Who cares?
"Subtle mind control? Why do all these HTML buttons say 'Submit' ?"
ReadThe ReflectionEngine, a cyberpunk style n
1. Diamond Viper 770 Ultra works fine under XFree86 3.3.4 on FreeBSD 3.2. 2. Most soundcards are supported. 3. Part of the FreeBSD kernel are actually under GNU...
The correct term for this is, of course, "code slush".
That's not what my Intel book says. Section 3.8 of the Intel Architecture Software Developer's Manual: Volume 3, Using the Physical Address Extension flag in the CR4 register extends the address space to 36 bits for processors after and including the Pentium Pro. :P
That means your heavy-duty stand-alone applications require you to purchase a very expensive product, which is tuned exclusively for network server and database work. You'll get an expensive set of server apps and licenses "for free" which you'll never use, and wait several weeks after the initial release for each and every service pack (they're always delayed for the high-end NT versions).
If you're interested in spending this kind of money, run Linux or *BSD on hardware that's designed to have 64Gb of memory and 12 CPUs (eg UltraSparc or Alpha) and don't worry about stupid kludges from Intel or any other 32-bit vendor. When even Intel are telling you that IA32 is a dead-end, it's time to get off.
In fact, if you've really got this type of job to do, your priority is probably scalability and performance, in which case a Proprietary Unix on it's own native hardware is going to look much more attractive. Want to buy some E450s?
Nick.
> Oh man don't even compare the way linux handles mem compared to BSD's.
:)
Why not? Afraid to learn something?
> The code the BSD's use for enormous amounts of memory is more mature.
Yes, although it's also far more complex than Linux, which disanvantageous in its own way.
The guy working on FreeBSD memory management took the time to look over the Linux code (which you obviously haven't) and noted that simplicity was one of its main strengths. He said that he weas hoping to simplify the FreeBSD code at some point
> Also, linux has problems withn neighbor table overflows.
What version of the kernel had this bug and what does this have to do with memory?
The site where I work has a 4-processor HP/UX machine with 4GB of RAM, running an Oracle backend for one of the webservers. It's woefully inadequate during peak times, and there are plans to upgrade it to something more robust in a few months.
Linux has *ALWAYS* supported the full address range on 64 bit platforms (Alpha and Usparc). Alpha linux sytems can currently have 2^42 bytes of ram (limitation of hardware) Unfortuantly some stupid PCI devices need dma into the whole addr range. ARGH! We're back to ISA's 16mb dma limit again!
AFAIK, from reading www.kt.opensrc.org, there is already a patch for this.
" Andrea Arcangeli posted a patch and announced that he and Gerhard Wichert had co-developed the patch to allow nearly 4 gigs of memory on 32-bit systems. A big discussion followed."
Located here
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
In short, way too much to be talking about ram for now.
My Freakin Blog
Catching up. Now we'll go further.
How long has BSD been around? What's
it done with the huge lead it's had?
I'm just wondering if it involves more than changing a lot of `int's to `unsigned int's.
My Freakin Blog