An illumina can do many more samples in a single run, in batch. But you might not want to take it into the field and your latency would be higher since you would accumulate samples until you had enough to justify one run.
But they are still (probably) burning one nanopore per sample, so that's $900 per run (yeah, I'm sure ONP gave them a hefty discount). So the overall cost is much higher than with Illumina, but you are right about the latency. In cases where that matters, I would go with PacBio (cheaper and faster).
Another way this thing is superior is in read-length (50kbases)
Well, let's say "up to 50 kb". The average is a different story, especially if you need to get the higher-quality 2D reads for your downstream analysis. ONP has been promising >100 kb reads for a long time but have yet to deliver. Much better than Illumina, as you say, but not better than PacBio.
I'm an outsider, so I've just gotta be misunderstanding something.
Well, like pretty much all press coverage of the Oxford Nanopore sequencer, there is a ton of hype but questionable value. I'll give the nanopore portability. It is an incredible feat compared to the large sequencers (even the benchtop MiSeq). But here's the thing:
1) The accuracy is terrible. This is especially important when you are looking at SNP variants. You need accuracy.
2) The sequencer may be portable, but data analysis in this version currently uses a cloud service that (obviously) requires an Internet connection, so I'm not sure the hype about service in rural areas is really that great.
3) The throughput is ok, but not great. For virus genomes this might be fine, but for bacterial and larger genomes, it's a no go.
4) The speed isn't all that great. It's around 24 hrs. to complete a 2D run. That is right in line with what is offered by other sequencers.
5) Yes, you DO need a library prep, contrary to what the proponents claim. It might be a little bit simpler than some conventional protocols, but you cannot just drop DNA into the pore.
All of this, in my mind, comes down to two features that matter most for any sequencer: cost and speed.
Cost: The best cost/bp currently, by far, comes from Illumina technologies. This will never compete with that. That said, Oxford Nanopore has an advantage in read length that Illumina will never have. However, PacBio has been competing in the read length niche for a while now and is well-established. So is the cost of Oxford Nanopore better than PacBio? Cost is mostly a function of yield x read length. For PacBio, the cost of a sequencing unit (a SMRT cell) is ~$600 (the library prep cost is ~$400, but is a one-time cost for each sample). One SMRT cell yields ~0.5 - 1 Gb per run, so $0.0000006/bp (Note: this is with the older RS II system. With the newer Sequel system the throughput is better). The Oxford Nanopore site claims up to 1 Gb per chip, at $900/chip, but the reality is a bit less. Based on a recent paper where they assembled the E. coli genome with nanopore data, the proportion of actual usable data is closer to 150 Mb. So that's about 10 times the cost of PacBio sequencing.
Speed: The Oxford Nanopore site claims they are fast, but to get the higher quality 2D reads that you need for assembly, the run times are typically about 18-24 hrs. For a MiSeq, the run time can be as low as 12 hrs, and for PacBio it is 3 hrs. So the nanopore is not really winning with speed either.
It seems to me that portability is the biggest strength of the nanopore, but the majority of groups are still going to get their sequencing done at core facilities, so I have doubts about how that will play out in the market. What they really need to focus on is cost. But everyone is doing that at the same time, so it is a hard race to keep up with.
FreeBSD Jails and Linux Containers are really different beasts. Jails are great if security is your primary consideration. Hence the name: Jails effectively isolate processes and go to great lengths to prevent them from accessing anything outside the jail. Containers use separate kernel namespaces to give groups of processes separate views of kernel global variables. Security (especially with user namespaces) is a bonus, but the primary goal is efficient os-level virtualization and isolation of resources. A more apt comparison is with the BSD VPS project rather than Jails.
At a guess the jail support didn't make it.
Correct. XNU does not have support for Jails, and it likely won't because it requires some pretty severe changes to kernel data structures to make them work.
Ummm...ok. So when your SMART detects a failing drive in your RAID0 array and you decide you want to replace it, how do you do that exactly? Oh, that's right, wipe the entire array and restore from backup, which, depending on the size of your array can take anywhere from several hours to days, more if you decided to use your array to run the OS as well. RAID0 is just a plain terrible idea, period. It doesn't matter if you don't think you need uptime, an N disk RAID0 is N times more likely to fail catastrophically as a standalone hard disk (assuming the failure rates on all of the hard disks are equal), and without redundancy getting back up and running is a long process.
Did you actually read that report? The very first comment:
The only way to deal with journal corruptions, currently, is to ignore them: when a corruption is detected, journald will rename the file to.journal~, and journalctl will try to do its best reading it.
Further down in comment #3, some more detail:
Journal files are mostly append-only files. We keep adding to the end as we go, only updating minimal indexes and bookkeeping in the front earlier parts of the files. These files are rotated (rotation = renamed and replaced by a new one) from time to time, based on certain conditions, such as time, file size, and also when we find the files to be corrupted. As soon as they rotate they are entirely read-only, never modified again. When you use a tool like "journalctl" to read the journal files both the active and the rotated files are implicitly merged, so that they appear as a single stream again.
Now, our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse. After all, in the case the often-run writing code really fucks something up, then it is not necessarily a good idea to try to make it better by running a tool on it that tries to fix it up again, a tool that is necessarily a lot more complex, and also less tested.
Now, of course, having corrupted files isn't great, and we should make sure the files even when corrupted stay as accessible as possible. Hence: the code that reads the journal files is actually written in a way that tries to make the best of corrupted files, and tries to read of them as much as possible, with the the subset of the file that is still valid. We do this implicitly on every access.
(emphasis added)
So, in other words, there can be journal corruption, as there can be corruption in any data file. The way journald deals with that is by rotating the log, so that a new log can start in its place, and the contents of the old log preceding the corruption are preserved. The journalctl utility knows how to read the corrupted files and extract useful information from them.
Google shows up plenty with little effort.
Google shows up people ranting about something they don't understand. The way journald handles corruption is probably the best possible way any software can handle corruption.
You're kidding right? That presentation is based entirely on developers not sanitizing user input, from the Internet no less, which will of course lead to vulnerabilities. I don't buy that it is an inherent weakness of the language. If you blindly submit SQL queries passed around as variables, you deserve what you get. That is just really bad programming in any language. Did they do a PHP jam?
They could have added binary logging to any existing syslogd. Instead, NIH, because Poettering.
If binary logging was the sole intention, then yes. But, as you know, the journal does a lot more than just a binary output of rsyslog, and would not be easily achieved by patching rsyslog. One can argue that those features are unnecessary, but that is a different subject.
The normal adoption process for something like systemd would be to roll a new distribution using it, and then once it is tested, debugged, and proven, for it to propagate throughout the community.
Well, systemd was being used by Fedora and Arch for quite some time. RHEL was on the way to adopting to it already. Ubuntu already had Upstart, so maybe they were forced into systemd by the upstream changes, but if you ask me, I think systemd is an improvement over upstart. It sounds to me that enough people liked it that distributions started adopting it. Maybe it was a faster change than you would have liked, but it is not a conspiracy. It comes down to who is doing the work to maintain the distributions and dependencies, and they decided that systemd was the way forward. An alternate way may be shown to us by Gentoo. We'll see, but I doubt there will be much traction.
Basically, you need to be able to read the logs with the most minimal of tools, because you are going to be diagnosing it in a downed state most likely-- You cannot bank on having a full suite of binary manipulation tools on hand. You will be lucky if you have more than vi.
This is the biggest load of canned bullsh**t argument. Where do you guys come up with this crap? We don't work on 1970s mainframes anymore. Why would vi be the only thing available? Why would an emergency shell not have the journalctl utility? Why would there not be serial logging capability? Why would you not be able to boot with a rescue disk (in case all other things fail)? The answer is, there is no reason. It is a made up scenario that doesn't exist.
Also, text based logs compress REAALLY well for long term storage for audit purposes! Binary logs? Probably not so much.
More bulls**t. There is no reason to think the journal won't compress well. Being binary, or not, has little if anything to do with compressibility. Remember that ASCII is itself a binary format! Also, auditing is one of the primary strengths of journald. It is designed specifically with auditing in mind.
if the binary logs are in some stupid "easily damaged" format, then having a process suddenly die horribly from abnormal termination will result in corrupt logs
That makes no sense at all. A process dying does not result in a corrupt log. There would be an error message, or at the very least a sigterm message, printed in the log just as it would be on the console if it had been run from an interactive terminal.
Hell, the file chain can be damaged from FS corruption, and parts of the log will still be readable.
The journal is designed to preserve as much of the log as possible when corruption happens. Anything written before or after the corruption event will be accessible by journalctl.
systemd notices that the log is corrupt, and... deletes it
That's not what happens.
See: the long and angry (and unfixed) bugzilla tickets.
Which one? The one that says journald will rotate the log on detection of corruption, thereby preserving it in an un tampered state while still allowing the uncorrupted portion to be read?
NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs.
Unlikely. If there was an error running the mount command, it was definitely recorded in the log. Did you use journalctl -u nfs or journalctl -b?
With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing
With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be).
I'm sorry, but that's bulls**t. If your system is randomly hanging, then it's not configured correctly. While the boot sequence of systemd can be characterized as non-deterministic, it is not random. If services are hanging because dependencies have not been met, then you need to specify your dependencies properly.
Another thing, is the broken compatibility with what came before. Example: I edit/etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent?
Since you should be using a >3.5 kernel, nothing. The rpc idmapper has been replaced with the nfsidmap system call in the kernel. So there is no need to start/restart an idmapper service.
Yes, it's partly me lacking familiarity with the new way of doing things;
Understatement. Have you read any of the systemd docs or transition/setup guides available? There are a ton. Just do it.
This is how the FreeBSD kernel devs describe BSD Jails. Each jail get's it's own kernel network stack, kernel memory allocator, and almost every other kernel datastructure.
What you are describing is VPS (Virtual Private System), not Jails. VPS is the successor to Jails, written to address some of the shortcomings of Jails and make them more useful in situations where you want true virtual environments, rather than just the extra security that Jails has to offer. Incidentally, the mechanisms used to implement VPS in FreeBSD are nearly identical to the mechanisms for implementing containers on linux. Here is the relevant description from the whitepaper (http://2010.eurobsdcon.org/fileadmin/fe_user/klaus/37R5uB.pdf):
3.4 Multiplexing global variables A FreeBSD kernel without VPS maintains global variables like the process table, the hostname, number of currently existing processes, and much more. In a VPS enabled kernel, global variables are replaced by variables private to a VPS instance. Therefore even if no VPS instance is explicitly created, the system knows the instance “vps0”, which is the “main system”. This instance is created very early at kernel boot and has all privileges. VPS instances can be created in a hierarchical way, allowing one VPS instance to manage its child instances and pass on part of their resource quotas. Each “struct ucred” keeps a pointer to the real VPS instance, and each “struct thread” keeps the pointer to the effective VPS instance. A “struct ucred” contains user credentials and is referenced by threads, processes, sockets, some devices as well as some other resources.
Sounds familiar? That is basically how linux namespaces work. The primary difference is that on linux you have several independent namespaces (cpu, mem, ipc, net, etc) that must be used together to create a container, whereas with VPS they are integrated more-or-less. On linux you have the flexibility to use namespaces for other purposes besides containers, on BSD you have VPS and only VPS.
BSD Jails do not use anything like chroot. "chroot" is being used as a verb that described the intention, but not the implementation.
For VPS, you are correct. For Jails it is a chroot, albeit a hardened version of the chroot system call, not chroot the userspace program. Read about it here in the whitepaper written by the author of Jails, http://phk.freebsd.dk/pubs/san...
They said this is nearly identical to paravirtualization.
This is not true at all. For Jails there are two principle isolation features. Once a process is jailed, the "struct proc" has a copy of the "prison" data structure and a reference counter. Every child process that it creates references this data structure in its own "struct proc" and increments the reference counter. In this way, the processes that belong to a jail are tracked by their references to the "prison" data structure owned by the root jailed process. Various kernel systems are then altered to block messages between processes not in the same jail, to limit process listing to the tree described by references to the root railed process (via the sysctl interface, for example), to limit pty access to processes in the same jail, etc. Among the things recorded in the "prison" data structure are the ip address and hostname, and the kernel blocks attempts to bind an ip address or hostname that does not fall in this specified range.
Despite these mechanisms to isolate processes, there is still one global process table and one network stack. The partitioning is effected by hiding parts of the process table and making every attempt to block communication outside of the allowed process list, but it is a fairly blunt instrument. For example, IPC is completely blocked by default in a Jail because this is th
The difference is BSD Jails are entirely separate environments with their own unshared kernel datastructures, and the jail communicates with the host via an API. Linux namespaces is just metadata added to shared environments.
I'm sorry, but this notion is completely wrong. A BSD Jail is a forked process (the "jail process"), which calls the "jail" kernel system call and then executes a chroot. The jail syscall serves to attach the "prison" data structure to the "proc" data structure of the jail process, allowing the kernel to identify the process as "jailed" and treat it accordingly. The isolation of the environments is dependent entirely on the kernel recognizing that the process is jailed and putting the appropriate restrictions on it. https://www.freebsd.org/doc/en...
An LXC container is a forked process with a chroot that is assigned a set of namespaces, which are then restricted by kernel cgroups and kernel capabilities. The end result is that the forked process is only able to communicate with other processes within its namespace and is only allowed to access resources that cgroups allow that namespace to access. The only way for a containerized process to communicate with the host is through a vnet bridge device, which is similar to way it works with Jails. https://libvirt.org/drvlxc.htm...
So in other words, the specific implementations details are different, but the result is nearly identical. A forked process executes a chroot and then is assigned a special status by the kernel which determines which resources it can access and which processes it can communicate with.
You're able to run as-root / Set-UID binaries with-in them? Nope. LXC emulates this by mapping UID-0 in the container to UID-x on the host via namespaces.
No, that is not correct. Root is root in an lxc container subject to some limitations (ex: making device entries), just like it is with BSD Jails. The mapping that you are referring to is a security mitigation feature, should an attacker manage to break out of the container. If a root-user within the container breaks out of the chroot (containers are essentially chroot with cgroups added in), but are still within the container process (iow, no buffer overflow or similar vulnerability), they will be subject to unprivileged status on the host (basically, the same as an unprivileged shell user). That is good, and is not something that BSD Jails do afaik. So, one might say lxc is more secure than jails in this respect.
From one of the maintainers of Docker (as of June 2014):
You do know that Docker and LXC are not the same thing, right? Docker is built on LXC, but they are not synonymous. Also, the quote is more of a "be careful with this" rather than a "containers can't handle this" type of comment. The thing about Docker specifically that makes it different from LXC, is the docker user space process, which is larger and possibly subject to more attack vectors, hence the conservatism about security. Just plain old LXC containers, though, should be as secure as anything else on the system (sans kernel vulnerabilities, etc).
there has been only one advisory about escaping out of a jail (and it was because of a devfsd bug, not jails itself)
Right, so aside from a kernel vulnerability (devfsd) a kernel-provided capability (jails) is perfectly secure? Nice bit of sophistry there. Jails is fundamentally no more or less secure than lxc containers. They are both "operating system-level virtualization" techniques implemented in similar ways (using chroot combined with kernel capabilities to separate userspace processes and resource limits). They are effectively the same.
but what Linux calls "containers" are crappy attempts to containerize.
Not sure what you mean. Jails have been around for a long time, but LXC/LXD containers have almost identical functionality.
container templates...check
filesystem snapshot integration (ZFS, btrfs) with cloning operations...check
resource limits...check
unprivileged containers...check
network isolation...more flexible under LXC than Jails, in my opinion
bind mounts in containers...check
nice management utilities...check
live migration...in development
Only difference I can see really is that LXC doesn't support nested containers...
It's been hard to find details on which clients support TLS1.1 - and perhaps there's some aspect here I'm missing
While there is some variability in how clients may implement or require certain features, 99% (yes, I just made that up, but a high%) of applications will use a library for tls connections, so capability is determined by the library it uses, and there aren't that many. For example, the majority of Windows applications, and certainly the ones provided by Microsoft will use SChannel which supports TLS 1.2 as of Win7, although it may not be enabled by default. Likewise, Apple applications like Mail.app use Secure Transport which has TLS 1.2 as of OS X 10.9. Mozilla applications use NSS which has TLS 1.2, as does GnuTLS which is used by Gnome applications like Evolution on Linux. So as far as support is concerned, I think you should be good. Some applications will use the protocol to differing extents, for example differing cipher suites or supporting earlier protocol versions, but if your server requires TLS 1.1, the client should be able to handle it, in the majority of cases. Considering that it's a standard from nearly 10 years ago, I would consider it a downright requirement. In my experience it is usually the servers that fail to support later versions of lts, not the clients, unless it is legacy custom code from random vendor.
Well, it's a prefab kit. Everything difficult (designing the dna templates, PCR, cloning) is already done. So you just mix, plate, and incubate. Not much different than other similar ed kits that have existed for ages. You aren't going to be able to make arbitrary mutations with this. For that you will still need a lab.
Personally, I think recombineering is much easier than CRISPR. Transform in RED plasmid, induce, transform in targeting fragments, done. Whereas with CRISPR, identify protospacer with appropriate PAM sequence, synthesize this and subclone into a plasmid, design editing template, synthesize this and clone into a plasmid, transform with cas9 and plasmids, pick colonies, sequence to confirm mutations. For some things CRISPR is nice, but when other methods like recombineering are available, those are usually easier. The main appeal of CRISPR is in eukaryotic systems where other tools are not available.
What are you talking about? Banana plants are a monoculture because of the (industrial) way they are cultivated. The cultivars like Cavendish are sterile because they are a hybrid between two species. They can be grown from seeds, it's just the seeds cannot be propagated. You have to go back to the original crosses to get viable seeds. Even if they weren't sterile, they would still be grown as a monoculture because that is the most convenient method of organizing a large tract farm. There is no law of nature preventing bananas from being cultivated in a different way. This is just the way the industry developed.
All decoded streams need to have matched sample rates. Pulseaudio doesn't upmix or downmix encoded bitstreams. It needs to know the content because it needs to know what to do with it. Say you are watching a bluray and then you turn on your Bluetooth headphones. What does pulseaudio do? Pass the encoded bitstream through? No because that would be bad. Even decoded 5.1 would be bad. What if you want to play to both devices simultaneously? What about when you receive a skype call? Should that be passed through as hd to your receiver? No. Pulseaudio needs to know the content so it can "route" the signal streams properly. It may not be a feature you care about, but it is the reason pulseaudio was written. In the case of dts-hd, the problem is not so much the format, but the way pulseaudio did the passthrough for the lower bitrate formats. The higher bitrates need more bandwidth, so this has to be engineered into the solution.
No, passthrough exists to allow you to pass the content forward unaltered so that more capable end-points can handle processing it. A high-end receiver knows a *LOT* more about audio, the speaker-configuration and room dynamics than Kodi does and can properly process the data for best quality sound.
What kind of processing? Equalizing to the room, yes. Decoding the bitstream, no. There is nothing particularly special about the latter. The former is done after decoding regardless of whether it happens at the receiver or the output device. If you have hdmi and a decoder elsewhere in the chain, you don't need passthrough to the receiver.
Uhm, alsa supports truehd and dts-hd because somebody has written the support for those. New format support does not just magically appear. Part of the problem is getting the format recognized, part of the problem is changing the sink passthrough code to allocate the higher bandwidth needed for these formats, and part of the problem is a smooth transition between passthrough and PCM, allowing different clients to handle things differently. I agree that it definitely needs to be there, but it is not trivial.
Anyway, if you are using hdmi, why do you need passthrough? Passthrough was created as a workaround to the limitations of s/pdif. They don't exist anymore if you are using hdmi.
Oh please. The only problem with pulseaudio was that sound on linux has always been a clustf$&k. Whether needing to detect your hardware capabilities and auto configure/dev entries, or setting something like a default output device, allowing per-user settings that are saved, allowing users to use the audio device without being root, mixing audio channels, whatever...it was the drivers/audio subsystem/system device configuration that created the problems. Pulseaudio was ONE attempt at a solution, and there were many others (esd, arts, dmix, etc). The only thing that really distinguishes pulseaudio is that A) it actually worked to some extent, and B) it does things the other projects didn't do, like network audio. The sound latency is still an issue that I don't particularly like, but there is no solution that doesn't compromise some other aspect (like using JACK), so meh. If we ever move off alsa, I thnk linux sound issues will probably disappear.
Or you could just, you know, not use it. It's been there for months. I've never even noticed it. I don't use a Firefox account, so who cares about Pocket integration anyway?
Plenty of other people manage to live and work in San Francisco at the same pay scales. They live there because they want to live there, that's why living costs are high. Sure, if they want to move to Kansas they can find cheaper living, but that doesn't seem to be what people are doing. The problem here is that there aren't enough workers, because companies have to poach workers from other companies to fill their vacancies. It's a good situation for developers because they can demand inflated salaries, but it's not sustainable and companies will fight it one way or another.
Well, first of all they don't. I think it should be abundantly obvious that the laws of economics have little to nothing to do with compensation of CEOs and star athletes or celebrities. Second, a CEO can be paid a large salary (which isn't always the case, btw, they usually get other forms of compensation) because he is one person, rightly or wrongly. You can't pay 10,000 people the salary of a CEO. Even the majority of board members don't even come close.
Slashdot Mirror
An illumina can do many more samples in a single run, in batch. But you might not want to take it into the field and your latency would be higher since you would accumulate samples until you had enough to justify one run.
But they are still (probably) burning one nanopore per sample, so that's $900 per run (yeah, I'm sure ONP gave them a hefty discount). So the overall cost is much higher than with Illumina, but you are right about the latency. In cases where that matters, I would go with PacBio (cheaper and faster).
Another way this thing is superior is in read-length (50kbases)
Well, let's say "up to 50 kb". The average is a different story, especially if you need to get the higher-quality 2D reads for your downstream analysis. ONP has been promising >100 kb reads for a long time but have yet to deliver. Much better than Illumina, as you say, but not better than PacBio.
I'm an outsider, so I've just gotta be misunderstanding something.
Well, like pretty much all press coverage of the Oxford Nanopore sequencer, there is a ton of hype but questionable value. I'll give the nanopore portability. It is an incredible feat compared to the large sequencers (even the benchtop MiSeq). But here's the thing:
1) The accuracy is terrible. This is especially important when you are looking at SNP variants. You need accuracy.
2) The sequencer may be portable, but data analysis in this version currently uses a cloud service that (obviously) requires an Internet connection, so I'm not sure the hype about service in rural areas is really that great.
3) The throughput is ok, but not great. For virus genomes this might be fine, but for bacterial and larger genomes, it's a no go.
4) The speed isn't all that great. It's around 24 hrs. to complete a 2D run. That is right in line with what is offered by other sequencers.
5) Yes, you DO need a library prep, contrary to what the proponents claim. It might be a little bit simpler than some conventional protocols, but you cannot just drop DNA into the pore.
All of this, in my mind, comes down to two features that matter most for any sequencer: cost and speed.
Cost:
The best cost/bp currently, by far, comes from Illumina technologies. This will never compete with that. That said, Oxford Nanopore has an advantage in read length that Illumina will never have. However, PacBio has been competing in the read length niche for a while now and is well-established. So is the cost of Oxford Nanopore better than PacBio? Cost is mostly a function of yield x read length. For PacBio, the cost of a sequencing unit (a SMRT cell) is ~$600 (the library prep cost is ~$400, but is a one-time cost for each sample). One SMRT cell yields ~0.5 - 1 Gb per run, so $0.0000006/bp (Note: this is with the older RS II system. With the newer Sequel system the throughput is better). The Oxford Nanopore site claims up to 1 Gb per chip, at $900/chip, but the reality is a bit less. Based on a recent paper where they assembled the E. coli genome with nanopore data, the proportion of actual usable data is closer to 150 Mb. So that's about 10 times the cost of PacBio sequencing.
Speed:
The Oxford Nanopore site claims they are fast, but to get the higher quality 2D reads that you need for assembly, the run times are typically about 18-24 hrs. For a MiSeq, the run time can be as low as 12 hrs, and for PacBio it is 3 hrs. So the nanopore is not really winning with speed either.
It seems to me that portability is the biggest strength of the nanopore, but the majority of groups are still going to get their sequencing done at core facilities, so I have doubts about how that will play out in the market. What they really need to focus on is cost. But everyone is doing that at the same time, so it is a hard race to keep up with.
FreeBSD Jails and Linux Containers are really different beasts. Jails are great if security is your primary consideration. Hence the name: Jails effectively isolate processes and go to great lengths to prevent them from accessing anything outside the jail. Containers use separate kernel namespaces to give groups of processes separate views of kernel global variables. Security (especially with user namespaces) is a bonus, but the primary goal is efficient os-level virtualization and isolation of resources. A more apt comparison is with the BSD VPS project rather than Jails.
At a guess the jail support didn't make it.
Correct. XNU does not have support for Jails, and it likely won't because it requires some pretty severe changes to kernel data structures to make them work.
Ummm...ok. So when your SMART detects a failing drive in your RAID0 array and you decide you want to replace it, how do you do that exactly? Oh, that's right, wipe the entire array and restore from backup, which, depending on the size of your array can take anywhere from several hours to days, more if you decided to use your array to run the OS as well. RAID0 is just a plain terrible idea, period. It doesn't matter if you don't think you need uptime, an N disk RAID0 is N times more likely to fail catastrophically as a standalone hard disk (assuming the failure rates on all of the hard disks are equal), and without redundancy getting back up and running is a long process.
Did you actually read that report? The very first comment:
The only way to deal with journal corruptions, currently, is to ignore them: when a corruption is detected, journald will rename the file to .journal~, and journalctl will try to do its best reading it.
Further down in comment #3, some more detail:
Journal files are mostly append-only files. We keep adding to the end as we go, only updating minimal indexes and bookkeeping in the front earlier parts of the files. These files are rotated (rotation = renamed and replaced by a new one) from time to time, based on certain conditions, such as time, file size, and also when we find the files to be corrupted. As soon as they rotate they are entirely read-only, never modified again. When you use a tool like "journalctl" to read the journal files both the active and the rotated files are implicitly merged, so that they appear as a single stream again.
Now, our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse. After all, in the case the often-run writing code really fucks something up, then it is not necessarily a good idea to try to make it better by running a tool on it that tries to fix it up again, a tool that is necessarily a lot more complex, and also less tested.
Now, of course, having corrupted files isn't great, and we should make sure the files even when corrupted stay as accessible as possible. Hence: the code that reads the journal files is actually written in a way that tries to make the best of corrupted files, and tries to read of them as much as possible, with the the subset of the file that is still valid. We do this implicitly on every access.
(emphasis added)
So, in other words, there can be journal corruption, as there can be corruption in any data file. The way journald deals with that is by rotating the log, so that a new log can start in its place, and the contents of the old log preceding the corruption are preserved. The journalctl utility knows how to read the corrupted files and extract useful information from them.
Google shows up plenty with little effort.
Google shows up people ranting about something they don't understand. The way journald handles corruption is probably the best possible way any software can handle corruption.
But Perl, well, let me just leave this here,
https://www.youtube.com/watch?... [youtube.com]
or Perl Jam 2, the Camel Strikes Back
You're kidding right? That presentation is based entirely on developers not sanitizing user input, from the Internet no less, which will of course lead to vulnerabilities. I don't buy that it is an inherent weakness of the language. If you blindly submit SQL queries passed around as variables, you deserve what you get. That is just really bad programming in any language. Did they do a PHP jam?
They could have added binary logging to any existing syslogd. Instead, NIH, because Poettering.
If binary logging was the sole intention, then yes. But, as you know, the journal does a lot more than just a binary output of rsyslog, and would not be easily achieved by patching rsyslog. One can argue that those features are unnecessary, but that is a different subject.
The normal adoption process for something like systemd would be to roll a new distribution using it, and then once it is tested, debugged, and proven, for it to propagate throughout the community.
Well, systemd was being used by Fedora and Arch for quite some time. RHEL was on the way to adopting to it already. Ubuntu already had Upstart, so maybe they were forced into systemd by the upstream changes, but if you ask me, I think systemd is an improvement over upstart. It sounds to me that enough people liked it that distributions started adopting it. Maybe it was a faster change than you would have liked, but it is not a conspiracy. It comes down to who is doing the work to maintain the distributions and dependencies, and they decided that systemd was the way forward. An alternate way may be shown to us by Gentoo. We'll see, but I doubt there will be much traction.
Basically, you need to be able to read the logs with the most minimal of tools, because you are going to be diagnosing it in a downed state most likely-- You cannot bank on having a full suite of binary manipulation tools on hand. You will be lucky if you have more than vi.
This is the biggest load of canned bullsh**t argument. Where do you guys come up with this crap? We don't work on 1970s mainframes anymore. Why would vi be the only thing available? Why would an emergency shell not have the journalctl utility? Why would there not be serial logging capability? Why would you not be able to boot with a rescue disk (in case all other things fail)? The answer is, there is no reason. It is a made up scenario that doesn't exist.
Also, text based logs compress REAALLY well for long term storage for audit purposes! Binary logs? Probably not so much.
More bulls**t. There is no reason to think the journal won't compress well. Being binary, or not, has little if anything to do with compressibility. Remember that ASCII is itself a binary format! Also, auditing is one of the primary strengths of journald. It is designed specifically with auditing in mind.
if the binary logs are in some stupid "easily damaged" format, then having a process suddenly die horribly from abnormal termination will result in corrupt logs
That makes no sense at all. A process dying does not result in a corrupt log. There would be an error message, or at the very least a sigterm message, printed in the log just as it would be on the console if it had been run from an interactive terminal.
Hell, the file chain can be damaged from FS corruption, and parts of the log will still be readable.
The journal is designed to preserve as much of the log as possible when corruption happens. Anything written before or after the corruption event will be accessible by journalctl.
systemd notices that the log is corrupt, and... deletes it
That's not what happens.
See: the long and angry (and unfixed) bugzilla tickets.
Which one? The one that says journald will rotate the log on detection of corruption, thereby preserving it in an un tampered state while still allowing the uncorrupted portion to be read?
NFS filesystems fail to be mounted at boot. Why? I have no bloody clue. There's nothing in the logs.
Unlikely. If there was an error running the mount command, it was definitely recorded in the log. Did you use journalctl -u nfs or journalctl -b?
With sysvinit I could boot to an interactive shell in the late initramfs and step through every script by hand, and pinpoint exactly where something was failing
You can get a debug shell in systemd. The process is just a little bit different,
http://freedesktop.org/wiki/So...
With systemd, even with correctly configured systems, I still experience occasional unrecoverable hangs at boot as it screws up its nondeterministic boot ordering and waits forever on something or other (who knows what it might be).
I'm sorry, but that's bulls**t. If your system is randomly hanging, then it's not configured correctly. While the boot sequence of systemd can be characterized as non-deterministic, it is not random. If services are hanging because dependencies have not been met, then you need to specify your dependencies properly.
Another thing, is the broken compatibility with what came before. Example: I edit /etc/nfsidmap.conf to configure NFSv4 id mapping. Previously I'd reload/restart the nfs-common service with 'service' or 'invoke-rc.d'. Job done. What's the systemd equivalent?
Since you should be using a >3.5 kernel, nothing. The rpc idmapper has been replaced with the nfsidmap system call in the kernel. So there is no need to start/restart an idmapper service.
Yes, it's partly me lacking familiarity with the new way of doing things;
Understatement. Have you read any of the systemd docs or transition/setup guides available? There are a ton. Just do it.
This is how the FreeBSD kernel devs describe BSD Jails. Each jail get's it's own kernel network stack, kernel memory allocator, and almost every other kernel datastructure.
What you are describing is VPS (Virtual Private System), not Jails. VPS is the successor to Jails, written to address some of the shortcomings of Jails and make them more useful in situations where you want true virtual environments, rather than just the extra security that Jails has to offer. Incidentally, the mechanisms used to implement VPS in FreeBSD are nearly identical to the mechanisms for implementing containers on linux. Here is the relevant description from the whitepaper (http://2010.eurobsdcon.org/fileadmin/fe_user/klaus/37R5uB.pdf):
3.4 Multiplexing global variables
A FreeBSD kernel without VPS maintains global variables like the process table, the hostname, number of currently existing processes, and much more.
In a VPS enabled kernel, global variables are replaced by variables private to a VPS instance. Therefore even if no VPS instance is explicitly created, the
system knows the instance “vps0”, which is the “main system”. This instance is created very early at kernel boot and has all privileges. VPS instances can be created in a hierarchical way, allowing one VPS instance to manage its child instances and pass on part of their resource quotas. Each “struct ucred” keeps a pointer to the real VPS instance, and each “struct thread” keeps the pointer to the effective VPS instance. A “struct ucred” contains user credentials and is referenced by threads, processes, sockets, some devices as well as some other resources.
Sounds familiar? That is basically how linux namespaces work. The primary difference is that on linux you have several independent namespaces (cpu, mem, ipc, net, etc) that must be used together to create a container, whereas with VPS they are integrated more-or-less. On linux you have the flexibility to use namespaces for other purposes besides containers, on BSD you have VPS and only VPS.
BSD Jails do not use anything like chroot. "chroot" is being used as a verb that described the intention, but not the implementation.
For VPS, you are correct. For Jails it is a chroot, albeit a hardened version of the chroot system call, not chroot the userspace program. Read about it here in the whitepaper written by the author of Jails,
http://phk.freebsd.dk/pubs/san...
They said this is nearly identical to paravirtualization.
This is not true at all. For Jails there are two principle isolation features. Once a process is jailed, the "struct proc" has a copy of the "prison" data structure and a reference counter. Every child process that it creates references this data structure in its own "struct proc" and increments the reference counter. In this way, the processes that belong to a jail are tracked by their references to the "prison" data structure owned by the root jailed process. Various kernel systems are then altered to block messages between processes not in the same jail, to limit process listing to the tree described by references to the root railed process (via the sysctl interface, for example), to limit pty access to processes in the same jail, etc. Among the things recorded in the "prison" data structure are the ip address and hostname, and the kernel blocks attempts to bind an ip address or hostname that does not fall in this specified range.
Despite these mechanisms to isolate processes, there is still one global process table and one network stack. The partitioning is effected by hiding parts of the process table and making every attempt to block communication outside of the allowed process list, but it is a fairly blunt instrument. For example, IPC is completely blocked by default in a Jail because this is th
The difference is BSD Jails are entirely separate environments with their own unshared kernel datastructures, and the jail communicates with the host via an API. Linux namespaces is just metadata added to shared environments.
I'm sorry, but this notion is completely wrong. A BSD Jail is a forked process (the "jail process"), which calls the "jail" kernel system call and then executes a chroot. The jail syscall serves to attach the "prison" data structure to the "proc" data structure of the jail process, allowing the kernel to identify the process as "jailed" and treat it accordingly. The isolation of the environments is dependent entirely on the kernel recognizing that the process is jailed and putting the appropriate restrictions on it.
https://www.freebsd.org/doc/en...
An LXC container is a forked process with a chroot that is assigned a set of namespaces, which are then restricted by kernel cgroups and kernel capabilities. The end result is that the forked process is only able to communicate with other processes within its namespace and is only allowed to access resources that cgroups allow that namespace to access. The only way for a containerized process to communicate with the host is through a vnet bridge device, which is similar to way it works with Jails.
https://libvirt.org/drvlxc.htm...
So in other words, the specific implementations details are different, but the result is nearly identical. A forked process executes a chroot and then is assigned a special status by the kernel which determines which resources it can access and which processes it can communicate with.
You're able to run as-root / Set-UID binaries with-in them? Nope. LXC emulates this by mapping UID-0 in the container to UID-x on the host via namespaces.
No, that is not correct. Root is root in an lxc container subject to some limitations (ex: making device entries), just like it is with BSD Jails. The mapping that you are referring to is a security mitigation feature, should an attacker manage to break out of the container. If a root-user within the container breaks out of the chroot (containers are essentially chroot with cgroups added in), but are still within the container process (iow, no buffer overflow or similar vulnerability), they will be subject to unprivileged status on the host (basically, the same as an unprivileged shell user). That is good, and is not something that BSD Jails do afaik. So, one might say lxc is more secure than jails in this respect.
From one of the maintainers of Docker (as of June 2014):
You do know that Docker and LXC are not the same thing, right? Docker is built on LXC, but they are not synonymous. Also, the quote is more of a "be careful with this" rather than a "containers can't handle this" type of comment. The thing about Docker specifically that makes it different from LXC, is the docker user space process, which is larger and possibly subject to more attack vectors, hence the conservatism about security. Just plain old LXC containers, though, should be as secure as anything else on the system (sans kernel vulnerabilities, etc).
there has been only one advisory about escaping out of a jail (and it was because of a devfsd bug, not jails itself)
Right, so aside from a kernel vulnerability (devfsd) a kernel-provided capability (jails) is perfectly secure? Nice bit of sophistry there. Jails is fundamentally no more or less secure than lxc containers. They are both "operating system-level virtualization" techniques implemented in similar ways (using chroot combined with kernel capabilities to separate userspace processes and resource limits). They are effectively the same.
but what Linux calls "containers" are crappy attempts to containerize.
Not sure what you mean. Jails have been around for a long time, but LXC/LXD containers have almost identical functionality.
container templates...check
filesystem snapshot integration (ZFS, btrfs) with cloning operations...check
resource limits...check
unprivileged containers...check
network isolation...more flexible under LXC than Jails, in my opinion
bind mounts in containers...check
nice management utilities...check
live migration...in development
Only difference I can see really is that LXC doesn't support nested containers...
It's been hard to find details on which clients support TLS1.1 - and perhaps there's some aspect here I'm missing
While there is some variability in how clients may implement or require certain features, 99% (yes, I just made that up, but a high%) of applications will use a library for tls connections, so capability is determined by the library it uses, and there aren't that many. For example, the majority of Windows applications, and certainly the ones provided by Microsoft will use SChannel which supports TLS 1.2 as of Win7, although it may not be enabled by default. Likewise, Apple applications like Mail.app use Secure Transport which has TLS 1.2 as of OS X 10.9. Mozilla applications use NSS which has TLS 1.2, as does GnuTLS which is used by Gnome applications like Evolution on Linux. So as far as support is concerned, I think you should be good. Some applications will use the protocol to differing extents, for example differing cipher suites or supporting earlier protocol versions, but if your server requires TLS 1.1, the client should be able to handle it, in the majority of cases. Considering that it's a standard from nearly 10 years ago, I would consider it a downright requirement. In my experience it is usually the servers that fail to support later versions of lts, not the clients, unless it is legacy custom code from random vendor.
Well, it's a prefab kit. Everything difficult (designing the dna templates, PCR, cloning) is already done. So you just mix, plate, and incubate. Not much different than other similar ed kits that have existed for ages. You aren't going to be able to make arbitrary mutations with this. For that you will still need a lab.
Personally, I think recombineering is much easier than CRISPR. Transform in RED plasmid, induce, transform in targeting fragments, done. Whereas with CRISPR, identify protospacer with appropriate PAM sequence, synthesize this and subclone into a plasmid, design editing template, synthesize this and clone into a plasmid, transform with cas9 and plasmids, pick colonies, sequence to confirm mutations. For some things CRISPR is nice, but when other methods like recombineering are available, those are usually easier. The main appeal of CRISPR is in eukaryotic systems where other tools are not available.
What are you talking about? Banana plants are a monoculture because of the (industrial) way they are cultivated. The cultivars like Cavendish are sterile because they are a hybrid between two species. They can be grown from seeds, it's just the seeds cannot be propagated. You have to go back to the original crosses to get viable seeds. Even if they weren't sterile, they would still be grown as a monoculture because that is the most convenient method of organizing a large tract farm. There is no law of nature preventing bananas from being cultivated in a different way. This is just the way the industry developed.
All decoded streams need to have matched sample rates. Pulseaudio doesn't upmix or downmix encoded bitstreams. It needs to know the content because it needs to know what to do with it. Say you are watching a bluray and then you turn on your Bluetooth headphones. What does pulseaudio do? Pass the encoded bitstream through? No because that would be bad. Even decoded 5.1 would be bad. What if you want to play to both devices simultaneously? What about when you receive a skype call? Should that be passed through as hd to your receiver? No. Pulseaudio needs to know the content so it can "route" the signal streams properly. It may not be a feature you care about, but it is the reason pulseaudio was written. In the case of dts-hd, the problem is not so much the format, but the way pulseaudio did the passthrough for the lower bitrate formats. The higher bitrates need more bandwidth, so this has to be engineered into the solution.
No, passthrough exists to allow you to pass the content forward unaltered so that more capable end-points can handle processing it. A high-end receiver knows a *LOT* more about audio, the speaker-configuration and room dynamics than Kodi does and can properly process the data for best quality sound.
What kind of processing? Equalizing to the room, yes. Decoding the bitstream, no. There is nothing particularly special about the latter. The former is done after decoding regardless of whether it happens at the receiver or the output device. If you have hdmi and a decoder elsewhere in the chain, you don't need passthrough to the receiver.
Uhm, alsa supports truehd and dts-hd because somebody has written the support for those. New format support does not just magically appear. Part of the problem is getting the format recognized, part of the problem is changing the sink passthrough code to allocate the higher bandwidth needed for these formats, and part of the problem is a smooth transition between passthrough and PCM, allowing different clients to handle things differently. I agree that it definitely needs to be there, but it is not trivial.
Anyway, if you are using hdmi, why do you need passthrough? Passthrough was created as a workaround to the limitations of s/pdif. They don't exist anymore if you are using hdmi.
Oh please. The only problem with pulseaudio was that sound on linux has always been a clustf$&k. Whether needing to detect your hardware capabilities and auto configure /dev entries, or setting something like a default output device, allowing per-user settings that are saved, allowing users to use the audio device without being root, mixing audio channels, whatever...it was the drivers/audio subsystem/system device configuration that created the problems. Pulseaudio was ONE attempt at a solution, and there were many others (esd, arts, dmix, etc). The only thing that really distinguishes pulseaudio is that A) it actually worked to some extent, and B) it does things the other projects didn't do, like network audio. The sound latency is still an issue that I don't particularly like, but there is no solution that doesn't compromise some other aspect (like using JACK), so meh. If we ever move off alsa, I thnk linux sound issues will probably disappear.
Sounds like a problem with Kodi.
Oh wait,
http://kodi.wiki/view/PulseAud...
Summary: if you intentionally break your sound, it doesn't work, surprise!
Or you could just, you know, not use it. It's been there for months. I've never even noticed it. I don't use a Firefox account, so who cares about Pocket integration anyway?
Plenty of other people manage to live and work in San Francisco at the same pay scales. They live there because they want to live there, that's why living costs are high. Sure, if they want to move to Kansas they can find cheaper living, but that doesn't seem to be what people are doing. The problem here is that there aren't enough workers, because companies have to poach workers from other companies to fill their vacancies. It's a good situation for developers because they can demand inflated salaries, but it's not sustainable and companies will fight it one way or another.
Well, first of all they don't. I think it should be abundantly obvious that the laws of economics have little to nothing to do with compensation of CEOs and star athletes or celebrities. Second, a CEO can be paid a large salary (which isn't always the case, btw, they usually get other forms of compensation) because he is one person, rightly or wrongly. You can't pay 10,000 people the salary of a CEO. Even the majority of board members don't even come close.