The problem here is no call exist to force writes to disk to be "ordered"
Right. Atomic rename is a special case of ordered write, really. Atomic-but-asynchronous-rename is great, but something more powerful would be nice too.
So what if ext3 is relatively slow? It's not the fucking bottleneck. If you're tuning a mailserver, a compile box, or a database machine, and you show with fucking benchmarks that filesystem metadata updates are causing significant performance problems, then use a different filesystem. But for most people, filesystem writes are not the bottleneck, and the data integrity lost by giving up ordered writes just isn't worth it.
With apologies to Benjamin Franklin, those who would give up essential data integrity for a little unimportant performance deserve neither.
So you're asking for all open-write-close operations to do an fsync at the end?
Of course not. That's silly.
Or only if a rename is done?
You're getting closer. Please try to follow. The difference between what you're thinking and what I'm saying is subtle.
What I'm demanding is that when renaming file B over file A, that file B's data blocks be written to disk before the record of the rename itself. That way, if the system is shut down at any point, the restarted system sees under the name A either the original contents of A or the complete contents of B.
That is not the same as asking for a sync to disk on every rename. It's asking for the operating system to introduce a dependency on the relative ordering of data block writes and renames.
Say you perform a open-write-close-rename operation, then sleep. The operating system might write out the changes to disk after five seconds, after 120 seconds, or after six months, but no matter when the changes are written, the data blocks for the opened file come first, and only after these blocks are written should the record of the rename itself be written. You could write the data blocks after three seconds and the rename operation after six months, and it wouldn't matter: so long as the relative order is preserved, a restarted system sees either the original contents of the file or the complete new contents.
What you're imagining, which is simply forcing rename to sync to disk, would satisfy the constraints above, true. But it's certainly not the best way to satisfy them. In fact, sync-on-rename would perform as badly as the fsync approach I'm railing against. The filesystem can do a better job than an application armed with fsync can, and dammit, they should. "Applications should use fsync" is not an excuse for breaking atomic rename.
Thank you! I'm glad someone else finally gets it! I disagree slightly, however: I think open-write-close-rename is a perfectly good way of asking for atomic replacement without an immediate commit. I think that should be the default behavior, since it's the most useful. The ext4/XFS behavior, in terms of safety, is just equivalent to open-write-close on the original file, and I think it's okay to make that operation the unsafe one.
In short, you're right, but in addition, I think the POSIX vocabulary is just fine as it is. Also, the delay behavior you're talking about is essentially what soft-updates does. It seems to work fine for the BSD people.
You're right that the problem is below the application layer, but fsync-on-close (even after two seconds) is a phenomenally stupid idea. Plenty of applications open a file each time they write something to a log, then close it again. While this is slightly inefficient, it's not something that should cause a disk-grinding full sync.
No! Atomic rename is how atomic updates work under unixish systems. The filesystem needs to properly implement the existing vocabulary. open-write-close-rename (with no goddamn fsync) is a perfectly reasonable thing to do.
All that's required is for the system to guarantee that when A is renamed on top of B, all data blocks for A are committed before the rename is. That way, an application looking at B will either see the contents of the old B or A, and never an empty file or a mixture of the two, even if the system dies in the seconds between that close and rename.
Userspace apps should be written to not have even a 5 second risk window.
No, filesystems shouldn't force the application developer to choose between unsafe and unusable. read-write-close-rename is a perfectly reasonable way for an application developer to ask for atomicity without durability, and it's the filesystem's job to do what application developers actually want. fsync here asks for something entirely different and imposes a stupefying performance penalty. No, it's the filesystem's job to give this sequence of operations atomic properties because that's what the application developer obviously requested.
Telling application developers to use a database is bullshit. The filesystem is a database, albeit not a relational one. A open-write-close-rename sequence merely asks for atomicity without durability, something that's perfectly reasonable. As other posters have mentioned in vain, all the application wants is for either the old version of a file or the entire new version to appear on a reboot. He doesn't care at the instant of the rename whether that replacement has been recorded on disk, just that eventually, when the filesystem does record that replacement, that it's recorded atomically.
You might want the open-write-fsync-close-rename behavior for a mailserver, in which you must acknowledge receipt (i.e., you need durability), but asking for that same durability in a multi-file configuration setup is just stupidly degrading performance.
open-write-close-rename is saying something fundamentally different from open-write-fsync-close-rename, and it's perfectly reasonable for a filesystem to act sanely in response to both kinds of request.
Data committed to disk as soon as an fwrite or fclose returns is not something you can or should expect
First of all, those are C buffered IO functions and have NOTHING TO DO WITH POSIX. They have buffer issues of their own. You meantwrite(2) and close(2), so let's talk about those. It's true that POSIX doesn't require IO to hit the disk until the system sees an fsync(2) or sync(2) call. However, it is perfectly reasonable to for atomic filesystem updates, and this feature can be accommodated in the existing POSIX API. There's no good reason a filesystem can't guarantee that any data blocks for a file renamed on top of another file aren't synced before the rename. Filesystems can implement this guarantee far less expensively than fsync(2). That's how the atomic rename facility was meant to be used, and it's a very elegant conceptual mechanism for asking the filesystem for atomicity without durability.
In short, requiring an fsync before an atomic rename removes an important word from the vocabulary of application developers, and ultimately does more harm than good. Performance will be horrid, and for no good reason.
If I, the application, write this new file and then rename it over the old, and a crash happens sometime after that sequence, then upon reboot the filesystem will either have the old file, the old file and the new file or the old file replaced by the new file.
Exactly! That's a conceptually elegant guarantee, and I think it's worth tweaking the filesystem to ensure it works. Look: open-write-fsync-close-rename is not merely the "correct" way to spell open-write-close-rename. The sequences mean entirely different things:
open-write-close-rename is asking that either the old or the new version should be in place at any one time, but also saying it's not important right now that the change happen right away. In database terminology, this sequence asks for atomicity without durability.
open-write-fsync-close-rename is asking for durability as well as atomicity. It's a much stronger guarantee. It's asking for durability as well --- the filesystem has no way of knowing that the rename is coming after it sees fsync-close, so of course it needs to write the new data right away.
You can't simply say "add an fsync!". That converts a weaker request into a stronger request, and results in a absofuckinglutely stupid performance hit for no good reason. Sequence 1 is perfectly reasonable, and it's the filesystem's job to ensure it works as intended.
When I write data to a file (either through a descriptor or FILE*)...[delaying] 150 seconds is brain-damaged.
You have a flawed and dangerous conception of how C's buffered IO functions work. Consider the following program: #include <stdio.h> #include <unistd.h>
int main() {
fprintf(stdout, "hi");
sleep(150);
_exit(0); }
This program will output nothing.fprintf, fwrite, etc. write to C streams (a FILE*). Each stream can be under one of three buffering schemes, selected by setvbuf:
unbuffered
line-buffered
fully buffered
(By default, standard output is typically fully buffered unless it is connected to a terminal, in which case it is fully buffered. Incidentally, this default buffering is one reason for the pty(7) system.)
When a stream is unbuffered, IO will be sent directly to the underlying system (in the POSIX case, write()) after each C stream operation. However, in the other two modes, data will be buffered forever unless certain conditions are met. These conditions are:
Forcing a flush explicitly with fflush
Closing a stream with fclose
An orderly shutdown of the process
In line-buffered mode, writing a newline
Filling up the internal stream buffer with data to be written
There is no timed flush. If a program terminates (say, by pulling the plug) without these conditions having been met, buffered data are lost.
Classic coke wasn't the same as original coke anyway - Coca-Cola used the confusion to change the sweetener from cane sugar to high fructose corn syrup. (Mmm... diabetes.)
No he cannot, as he cannot write that interpreter to a place where it can be executed.
There are plenty of interpreters on the system already --- or do you plan to uninstall Firefox?
Note that trying to do this trick doesn't work, as your linker then needs to mmap this code with PROT_EXEC which is not allowed for files residing in a noexec mounted fs.
I think the point of the G...GP post was that you can't easily push this out remotely, and on Linux you have to write it, support it and debug it yourself, including all the niggly corner cases.
That's a good point, but the kind of huge organization you mention will have in-house IT people who can that anyway, and I still think the advantage of a FOSS platform outweighs the relatively lack of ready-to-go deployment facilities.
WSUS. Centrally administer the set of updates permitted to clients and servers. Linux version: Maybe set up a repository for your corp distro - but how to sync and manage the updates is what I don't know here.
Any of the major repository systems can be set up in a custom configuration with client machines automatically sucking packages up from a central company repository. Redhat's up2date and satellite systems are especially geared toward this kind of deployment.
SCCM / Zenworks / Others. Roll out an application to user desktops whether they're on-net or not. I can push Office to a machine 500mi from one of my offices
If I'm understanding this correctly, you get application installation automation for free with your centralized repository, perhaps automated with cfengine, puppet, or even ssh-in-a-loop.
Group Policy...
This is hard, and I'll admit Windows has an edge here, though personally, I feel like that's a little bit about North Korea having an edge in oppression compared to the US; it's not necessarily something desirable.
That said, if you must do something like this, there are ways. Other comments for this article address this point better than I do. For starters, there's kiosk mode "KDE's Kiosk Mode, allows a system administrator to configure all aspects of the desktop for an end user and optionally prevent the end user from making modifications to the provided setup."
If it does lock up you have to kill all the processes, resulting in lost data.
That doesn't sound right. You should complain to the network administers. NFS is supposed to be robust against server overloads and even reboots. (I recall one story about how a client machine dutifully waited six months for a damaged server to be shut down and shipped across the country and back for repairs.)
That said, nfs4 works well for us. POHMELFS and CRFS look even more promising.
Maybe that data entry clerk wants to take a five-minute break from typing to keep her wrists healthy. Maybe she wants to send an email to her kid she let stay home from school to see how he's feeling. Maybe she wants to check the local diner's menu and phone in an order. These things don't affect her overall productivity.
Incidentally, productivity is something easily and objectively measured. You're paying the employee for her output, so just use that to evaluate her performance.
You can't technologically tell the distractions from the legitimate personal tasks, so you'd choose to ban everything. I bet you'd make her clock out to use the bathroom if you could.
Replacements are easy.
You know what? Fuck you. People like you are the reason we need unions in the world.
what did you expect? the linux world is full of retards who debate shit endlessly and never do anything productive. in fact the majority of the linux world has been retards who produce nothing.
Am I getting older, or is Slashdot even more puerile than usual lately?
I just don't want anyone to get the wrong idea and think that noexec provides a kind of security it really can't. Really, noexec provides a minimum security benefit. nosuid, now that's important.
As a bonus, you can easily add web filtering and block things like Slashdot at work.
Actually, browsing Slashdot, The Old New Thing, lwn.net and so on has made me more productive overall. Preventing users from accessing "time wasters" is a losing strategy: not only is the blocking technically futile, but by treating employees like children, you kill morale. Instead of micromanaging their days, treat employees like responsible adults and evaluate them based on their work and its results.
This kind of stuff is why NFS-mounted home directories are just wonderful. If my machine kicks the bucket, I can grab a new one, install an OS on it, and get back to where I was before in half an hour. In a larger organization, an imaged system would work even better.
Now, as for mass configuration changes, cfengine is your friend.
Err, you can still run interpreted programs on a filesystem mounted noexec:
~$ python myprogram.py
A sufficiently clever user could use an interpreter to write his own dynamic linker and thereby run binaries too.
But I agree: locking down the desktop is the wrong approach. Better is to separate sensitive information from things that aren't sensitive, and have a standard user environment to restore to if the user does manage to mess up his configuration.
Our species is not a unique and special snowflake. We're likely to see all kinds of convergent evolution. An example from biology: Cephalopods. (Squids, octopuses, and so on.) We can use Cephalopods to test theories about extraterrestrial life like we can use Antarctica to test Mars rovers.
The most developed of these is the Octopus. Not only do these guys have eyes that are better than our own, but they have brains. This is important because our last common ancestor with these guys had neither brains nor eyes, and was as complex as yeast. Yet the Octopus nervous system has quite a few similarities to our own:
The findings emerging from recent electrophysiological studies in the octopus suggest that a convergent evolutionary process has led to the selection of similar networks and synaptic plasticity in evolutionarily very remote species that evolved to similar behaviors and modes of life. These evolutionary considerations substantiate the importance of these cellular and morphological properties for neural systems that mediate complex forms of learning and memory. In particular, the similarity in the architecture and physiological connectivity of the octopus MSF-VL system to the mammalian hippocampus and the extremely high number of small interneurons in its areas of learning and memory suggest the importance of a large number of units that independently, by en passant innervation, form a high redundancy of connections. As these features are found in both the octopus MSF-VL system and the hippocampus, it would appear that they are needed to create a large capacity for memory associations.
Any technological alien civilization would face the same mathematical evolutionary pressures described by game theory, and would develop along lines close to our own. The differences we see between alien cultures and our own will be on the order of the differences between human cultures, and not something radically different.
Why would you suppose that the distance between us and extraterrestrial life would be any greater than that between us and the octopus? We can be reasonably confident that:
The laws of physics and the mathematical realities of game theory are the same everywhere
Life will be carbon-based, because there aren't really any good alternatives. (No, silicon won't work.)
Some kind of organic polymer will be used to encode each orgasm's genetic information, because there really isn't a good alternative. DNA is good choice because it's stable and cheap to make.
Carbon-based life will require roughly the same environment we do. For instance, there will be no creatures that require bio-suits kept at the temperature of molten lead because any reasonable enzyme would denature at that point. Some kind of liquid environment will be needed for chemical reactions to take place in, probably water because it's abundant, simply, and is liquid in the right temperature range.
Organisms will face roughly the same environment challenges we do, and will produce similar results.
Really, we're not going to see off-the-wall organisms. They'll have eyes. They'll have brains. Anything that required technology will require air, fire, and water. Fire requires oxygen, so our aliens will have roughly the same atmospheric needs we do.
Slashdot Mirror
Right. Atomic rename is a special case of ordered write, really. Atomic-but-asynchronous-rename is great, but something more powerful would be nice too.
What we really need is a user-level fbarrier. I'm not the first person to think of this syscall.
Also,
When you think about it, that's a very powerful guarantee. (Personally, though, I'd rather have fbarrier.)
So what if ext3 is relatively slow? It's not the fucking bottleneck . If you're tuning a mailserver, a compile box, or a database machine, and you show with fucking benchmarks that filesystem metadata updates are causing significant performance problems, then use a different filesystem. But for most people, filesystem writes are not the bottleneck, and the data integrity lost by giving up ordered writes just isn't worth it.
With apologies to Benjamin Franklin, those who would give up essential data integrity for a little unimportant performance deserve neither.
Of course not. That's silly.
You're getting closer. Please try to follow. The difference between what you're thinking and what I'm saying is subtle.
What I'm demanding is that when renaming file B over file A, that file B's data blocks be written to disk before the record of the rename itself. That way, if the system is shut down at any point, the restarted system sees under the name A either the original contents of A or the complete contents of B.
That is not the same as asking for a sync to disk on every rename. It's asking for the operating system to introduce a dependency on the relative ordering of data block writes and renames.
Say you perform a open-write-close-rename operation, then sleep. The operating system might write out the changes to disk after five seconds, after 120 seconds, or after six months, but no matter when the changes are written, the data blocks for the opened file come first, and only after these blocks are written should the record of the rename itself be written. You could write the data blocks after three seconds and the rename operation after six months, and it wouldn't matter: so long as the relative order is preserved, a restarted system sees either the original contents of the file or the complete new contents.
What you're imagining, which is simply forcing rename to sync to disk, would satisfy the constraints above, true. But it's certainly not the best way to satisfy them. In fact, sync-on-rename would perform as badly as the fsync approach I'm railing against. The filesystem can do a better job than an application armed with fsync can, and dammit, they should. "Applications should use fsync" is not an excuse for breaking atomic rename.
Thank you! I'm glad someone else finally gets it! I disagree slightly, however: I think open-write-close-rename is a perfectly good way of asking for atomic replacement without an immediate commit. I think that should be the default behavior, since it's the most useful. The ext4/XFS behavior, in terms of safety, is just equivalent to open-write-close on the original file, and I think it's okay to make that operation the unsafe one.
In short, you're right, but in addition, I think the POSIX vocabulary is just fine as it is. Also, the delay behavior you're talking about is essentially what soft-updates does. It seems to work fine for the BSD people.
You're right that the problem is below the application layer, but fsync-on-close (even after two seconds) is a phenomenally stupid idea. Plenty of applications open a file each time they write something to a log, then close it again. While this is slightly inefficient, it's not something that should cause a disk-grinding full sync.
No! Atomic rename is how atomic updates work under unixish systems. The filesystem needs to properly implement the existing vocabulary. open-write-close-rename (with no goddamn fsync) is a perfectly reasonable thing to do.
All that's required is for the system to guarantee that when A is renamed on top of B, all data blocks for A are committed before the rename is. That way, an application looking at B will either see the contents of the old B or A, and never an empty file or a mixture of the two, even if the system dies in the seconds between that close and rename.
No, filesystems shouldn't force the application developer to choose between unsafe and unusable. read-write-close-rename is a perfectly reasonable way for an application developer to ask for atomicity without durability, and it's the filesystem's job to do what application developers actually want. fsync here asks for something entirely different and imposes a stupefying performance penalty. No, it's the filesystem's job to give this sequence of operations atomic properties because that's what the application developer obviously requested.
Telling application developers to use a database is bullshit. The filesystem is a database, albeit not a relational one. A open-write-close-rename sequence merely asks for atomicity without durability, something that's perfectly reasonable. As other posters have mentioned in vain, all the application wants is for either the old version of a file or the entire new version to appear on a reboot. He doesn't care at the instant of the rename whether that replacement has been recorded on disk, just that eventually, when the filesystem does record that replacement, that it's recorded atomically.
You might want the open-write-fsync-close-rename behavior for a mailserver, in which you must acknowledge receipt (i.e., you need durability), but asking for that same durability in a multi-file configuration setup is just stupidly degrading performance.
open-write-close-rename is saying something fundamentally different from open-write-fsync-close-rename, and it's perfectly reasonable for a filesystem to act sanely in response to both kinds of request.
First of all, those are C buffered IO functions and have NOTHING TO DO WITH POSIX. They have buffer issues of their own. You meant write(2) and close(2), so let's talk about those. It's true that POSIX doesn't require IO to hit the disk until the system sees an fsync(2) or sync(2) call. However, it is perfectly reasonable to for atomic filesystem updates, and this feature can be accommodated in the existing POSIX API. There's no good reason a filesystem can't guarantee that any data blocks for a file renamed on top of another file aren't synced before the rename. Filesystems can implement this guarantee far less expensively than fsync(2). That's how the atomic rename facility was meant to be used, and it's a very elegant conceptual mechanism for asking the filesystem for atomicity without durability.
In short, requiring an fsync before an atomic rename removes an important word from the vocabulary of application developers, and ultimately does more harm than good. Performance will be horrid, and for no good reason.
Exactly! That's a conceptually elegant guarantee, and I think it's worth tweaking the filesystem to ensure it works. Look: open-write-fsync-close-rename is not merely the "correct" way to spell open-write-close-rename. The sequences mean entirely different things:
You can't simply say "add an fsync!". That converts a weaker request into a stronger request, and results in a absofuckinglutely stupid performance hit for no good reason. Sequence 1 is perfectly reasonable, and it's the filesystem's job to ensure it works as intended.
You have a flawed and dangerous conception of how C's buffered IO functions work. Consider the following program:
#include <stdio.h>
#include <unistd.h>
int main() {
fprintf(stdout, "hi");
sleep(150);
_exit(0);
}
This program will output nothing. fprintf, fwrite, etc. write to C streams (a FILE*). Each stream can be under one of three buffering schemes, selected by setvbuf:
(By default, standard output is typically fully buffered unless it is connected to a terminal, in which case it is fully buffered. Incidentally, this default buffering is one reason for the pty(7) system.)
When a stream is unbuffered, IO will be sent directly to the underlying system (in the POSIX case, write()) after each C stream operation. However, in the other two modes, data will be buffered forever unless certain conditions are met. These conditions are:
There is no timed flush. If a program terminates (say, by pulling the plug) without these conditions having been met, buffered data are lost.
Classic coke wasn't the same as original coke anyway - Coca-Cola used the confusion to change the sweetener from cane sugar to high fructose corn syrup. (Mmm... diabetes.)
There are plenty of interpreters on the system already --- or do you plan to uninstall Firefox?
So? MAP_PRIVATE, PROT_EXEC, read() and jmp.
Fair enough. For me, the single most useful Windows administration tool is cygwin. :-)
Different scenarios. What if your user is using his account on a central machine via remote X11?
That's a good point, but the kind of huge organization you mention will have in-house IT people who can that anyway, and I still think the advantage of a FOSS platform outweighs the relatively lack of ready-to-go deployment facilities.
Any of the major repository systems can be set up in a custom configuration with client machines automatically sucking packages up from a central company repository. Redhat's up2date and satellite systems are especially geared toward this kind of deployment.
If I'm understanding this correctly, you get application installation automation for free with your centralized repository, perhaps automated with cfengine, puppet, or even ssh-in-a-loop.
This is hard, and I'll admit Windows has an edge here, though personally, I feel like that's a little bit about North Korea having an edge in oppression compared to the US; it's not necessarily something desirable.
That said, if you must do something like this, there are ways. Other comments for this article address this point better than I do. For starters, there's kiosk mode "KDE's Kiosk Mode, allows a system administrator to configure all aspects of the desktop for an end user and optionally prevent the end user from making modifications to the provided setup."
Gnome also supports a lockdown system.
And as a last resort, you can always patch the software and distribute the patched version to all your machines.
That doesn't sound right. You should complain to the network administers. NFS is supposed to be robust against server overloads and even reboots. (I recall one story about how a client machine dutifully waited six months for a damaged server to be shut down and shipped across the country and back for repairs.)
That said, nfs4 works well for us. POHMELFS and CRFS look even more promising.
Maybe that data entry clerk wants to take a five-minute break from typing to keep her wrists healthy. Maybe she wants to send an email to her kid she let stay home from school to see how he's feeling. Maybe she wants to check the local diner's menu and phone in an order. These things don't affect her overall productivity.
Incidentally, productivity is something easily and objectively measured. You're paying the employee for her output, so just use that to evaluate her performance.
You can't technologically tell the distractions from the legitimate personal tasks, so you'd choose to ban everything. I bet you'd make her clock out to use the bathroom if you could.
You know what? Fuck you. People like you are the reason we need unions in the world.
A wrapper around perl is one obvious approach.
Strawman, strawman, strawman. Why do you keep bringing up ssh in a loop while willfully ignoring puppet and cfengine?
Am I getting older, or is Slashdot even more puerile than usual lately?
Oh, I agree.
I just don't want anyone to get the wrong idea and think that noexec provides a kind of security it really can't. Really, noexec provides a minimum security benefit. nosuid, now that's important.
Actually, browsing Slashdot, The Old New Thing, lwn.net and so on has made me more productive overall. Preventing users from accessing "time wasters" is a losing strategy: not only is the blocking technically futile, but by treating employees like children, you kill morale. Instead of micromanaging their days, treat employees like responsible adults and evaluate them based on their work and its results.
This kind of stuff is why NFS-mounted home directories are just wonderful. If my machine kicks the bucket, I can grab a new one, install an OS on it, and get back to where I was before in half an hour. In a larger organization, an imaged system would work even better.
Now, as for mass configuration changes, cfengine is your friend.
Err, you can still run interpreted programs on a filesystem mounted noexec:
~$ python myprogram.py
A sufficiently clever user could use an interpreter to write his own dynamic linker and thereby run binaries too.
But I agree: locking down the desktop is the wrong approach. Better is to separate sensitive information from things that aren't sensitive, and have a standard user environment to restore to if the user does manage to mess up his configuration.
Our species is not a unique and special snowflake. We're likely to see all kinds of convergent evolution. An example from biology: Cephalopods. (Squids, octopuses, and so on.) We can use Cephalopods to test theories about extraterrestrial life like we can use Antarctica to test Mars rovers.
The most developed of these is the Octopus. Not only do these guys have eyes that are better than our own, but they have brains. This is important because our last common ancestor with these guys had neither brains nor eyes, and was as complex as yeast. Yet the Octopus nervous system has quite a few similarities to our own:
Any technological alien civilization would face the same mathematical evolutionary pressures described by game theory, and would develop along lines close to our own. The differences we see between alien cultures and our own will be on the order of the differences between human cultures, and not something radically different.
Why would you suppose that the distance between us and extraterrestrial life would be any greater than that between us and the octopus? We can be reasonably confident that:
Really, we're not going to see off-the-wall organisms. They'll have eyes. They'll have brains. Anything that required technology will require air, fire, and water. Fire requires oxygen, so our aliens will have roughly the same atmospheric needs we do.
The differences we may see are in the arbitrary choices evolution has made: I think we'll see extraterrestrial life use some of the amino acids that don't occur in nature here. Maybe their proteins and carbohydrates have opposite orientations. But the fundamental structures will be very similar because the problem is similar everywhere!
Also, cultu