OK, now it's starting to make more sense looking at the use cases
Here is a description of a few of the popular use cases for Apache Kafka. For an overview of a number of these areas in action, see this blog post.
Messaging Kafka works well as a replacement for a more traditional message broker. Message brokers are used for a variety of reasons (to decouple processing from data producers, to buffer unprocessed messages, etc). In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications. In our experience messaging uses are often comparatively low-throughput, but may require low end-to-end latency and often depend on the strong durability guarantees Kafka provides.
In this domain Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.
Website Activity Tracking The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting. Activity tracking is often very high volume as many activity messages are generated for each user page view.
Metrics Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.
Log Aggregation Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. In comparison to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-to-end latency.
Stream Processing Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. For example, a processing pipeline for recommending news articles might crawl article content from RSS feeds and publish it to an "articles" topic; further processing might normalize or deduplicate this content and published the cleansed article content to a new topic; a final processing stage might attempt to recommend this content to users. Such processing pipelines create graphs of real-time data flows based on the individual topics. Starting in 0.10.0.0, a light-weight but powerful stream processing library called Kafka Streams is available in Apache Kafka to perform such data processing as described above. Apart from Kafka Streams, alternative open source stream processing tools include Apache Storm and Apache Samza.
Event Sourcing Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.
Commit Log Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper project.
I've got no idea what Kafka does, and the summary really doesn't tell you much at all. I was about to put in a helpful post saying what it is, but even after visiting their home page I've still got no idea.
Apparently Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
How about the Intro We think of a streaming platform as having three key capabilities: It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
It lets you store streams of records in a fault-tolerant way.
It lets you process streams of records as they occur.
What is Kafka good for? It gets used for two broad classes of application:
Building real-time streaming data pipelines that reliably get data between systems or applications
Building real-time streaming applications that transform or react to the streams of data
I have a 2013 Mac Pro and a new 2016 MacBook Pro 13". Whilst multi-thread performance is a different matter altogether, single core performance is pretty much on par (with a slight edge to the laptop) when comparing the two machines. The vast majority of software I run is single threaded, as I don't do video editing, 3D or gaming.
This is a Intel Xeon E5-1620 quad-core versus an i7-6567U
I honestly don't know. At which point do you need to pay for Java? Is it you need to buy a licence to write code in Java? Run the code you've written? Distribute your Java code to others?
$5k per 20 seats per year. This works out at $250 per seat per year. This is less than a dollar a day. If you're running a company with 20 developers and can't afford $1 per person per day then you're doing something wrong.
A possible solution is to not ban sales to bots per-se, but instead verify that the identity of the person redeeming the ticket at the door is the same as the person who purchased the ticket (via verifying CC details, or even something as basic as their name).
If tickets have conditions on them that prevent their usage by anyone other than the person who originally bought them, then there can be no market for resold tickets. Let the scalpers buy as many tickets as they want, but eliminate the market for them to be resold.
Ticket Australia now state as part of their conditions of sale "This ticket may not, without the prior written consent of Ticketek or the Seller, be resold at a premium or used for advertising, promotion or other commercial purposes (including competitions and trade promotions) or to enhance the demand for other goods or services. If a ticket is sold or used in breach of this condition, the bearer of the ticket will be refused admission."
If you knowingly purchase a scalped ticket, you're taking a huge risk that you won't get in to the event.
I remember this - I had a cron job running once every 5 minutes that would use curl to get the current weather report, parse that for the temperature and update the LCD on the printer. Good times indeed...
With my car, once it's been locked with the button on the key fob, after a certain amount of time, it deadlocks the doors - they can not be opened from the inside or outside without being unlocked. The unlock button on the driver's door will no longer function either after the car has been locked from the fob.
This means I could, if I wanted to, lock the car with the windows partially down and after a minute or so the car would be deadlocked - even if someone reached in to open the door, they would be unable to.
Apple are doing what they have done every single year - retiring old models from their supported lineup. Film at 11.
Every year, a range of Macs pass through the range of support status from "Supported" to "Vintage" to "Obsolete"
Vintage products are those that have not been manufactured for more than 5 and less than 7 years ago. Apple has generally discontinued hardware service for vintage products in most regions other than the state of California and Turkey. Obsolete products are those that were discontinued more than 7 years ago. Apple has discontinued all hardware service for obsolete products with no exceptions. Service providers cannot order parts for obsolete products through Apple.
Game of Thrones seems to be set on an Earth-like world, however with dragons, undead and a degree of magic. They have winters that are sometimes brief and other times plunge the world into a freeze that lasts for generations...
Consider tidal locking in a system with an M-dwarf star, a "hot Jupiter" and our Planet of Interest (PoI). If in orbit around either the hot Jupiter or the star, the PoI might become locked. But with the three in relatively close interaction, the PoI could be disturbed between locking to one, or the other, or alternating, or spinning irregularly. Feel free to use a planet with an irregular - literally chaotic, even - rotation in an SF scenario of your choice.
So that explains the irregular seasons in the Game of Thrones universe... Winter is coming.
It's trivial to install Samba on OS X. Step 1. Install Xcode Step 2. Install MacPorts Step 3. sudo port install samba3 or sudo port install samba4
Apple can't include Samba out of the box with OS X due to issues linking to GPL v3 libraries and issues due to foregoing patent lawsuits if using GPL v3 code, but there's nothing stopping you from adding it yourself.
Yes, there is Apache licensed code, BSD licensed code etc... The problem with the GPL is that it was previously able to be used as a part of a larger non-open source offering and with the change in the licence to v3, this is no longer possible.
That's the problem with GPL v3. With V2, you could link to GPL'd code, or use GPL'd libraries, in your closed source project and that was OK. If you made changes to the GPL'd code, you needed to make the source for the changes available to anyone who you distributed the software to.
With v3 however, if you use GPL'd code or libraries in your closed source project, you have to make the ENTIRE PROJECT available under the GPLv3 as well. This is the viral nature of it.
From http://choosealicense.com/lice... Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights.
If you include GPL v3 code in your closed source project, under the terms of the licence agreement, you must comply with the licence which demands either you licence the software or you release ALL of the source of your closed-source project under the GPL v3.
If you are producing a large, closed-source, product, option 1 is not an option at all.
If you can't negotiate a licence with the copyright holders, or the copyright holders are unwilling to licence it to you on terms that are acceptable to both parties then option 2 is not an option at all.
So, this leaves only option 3 - remove all GPL'd code from your project and write your own.
I feel that the viral licensing clauses in GPL v3 will ultimately hinder the further development of software. While Apple (as an example) were using GPL v2 licensed code, they were actively contributing patches and changes back to the relevant projects. This was good, it meant that we had professional developers who were paid to work on these projects and the changes they made were contributed back upstream.
Now that no corporation can touch any GPL v3 licensed code, we now have fragmentation and less developers working on open source code bases.
Apple, for example, have had to implement their own SMB stack as smbx, instead of using Samba. For a number of years, SMB compatibility and functionality took a huge step backwards on OS X, all because the Samba project started to use GPL v3. This ended up with developers who would have been working on patches and changes for Samba, instead working on their own closed-source implementation that, quite frankly, was nowhere near as good or as mature as Samba. The end result of this was that Apple's customers suffered with a sub par product and the Samba project suffered with fewer people contributing to it.
" if you have hardware and software that can create video at >4GB/s you're pretty special."
Protip: Most game modern engines can EASILY do that with just the video card. AGP had a maximum throughput of 2166MB/s, half your requirement. That was replaced roughly a decade ago.
Bandwidth of a video card interface and the throughput of a 3D gaming engine is irrelevant to this discussion. We're talking creating video through something like Avid or Premiere/After Effects by compositing multiple streams of video and effects and then mixing that all down to a single stream for output.
Whilst some acceleration of this is done on the GPU (and this is why the 3 year old Mac Pro has 2 GPUs even in it's base configuration) the main requirement here is fast and consistent throughput to mass storage.
There are plenty of video professionals that use a Mac Pro with it's stock 256 GB internal SSD and then hook it up to an external 8- or 12-bay Thunderbolt RAID (or two) that can transfer in excess of 2 GB/sec - something like this http://www.lacie.com/as/en/pro...
I don't care how big your PC workstation case is, if you want 50 or 100 TB of storage, you're looking at external RAID or SAN anyway. The internal storage is only used for the OS, apps and scratch space.
The Adult Population of the USA is something like 194.5 Million people. Let's say that you can get by on $25,000 per year, tax free. Providing UBI for this many people will cost the economy 4.8 Trillion Dollars. Where is this going to come from? OK, let's scale this back a bit. We will give every adult in the USA $200 per week - $10,400 per year. We're still talking about $2.02 Trillion - this is 11% of the entire GDP of the USA. To put this in perspective, the USA spends $810 Billion on public education per year, $1.3 Trillion on pensions and almost $600 Billion on defence.
You're thinking of Services for Unix (SFU) - that was part of it, and the other part was the POSIX-compliant API that Windows NT 3.1 was based on (and remember that NT 3.1 was really NT 1.0).
Windows also incorporated code from BSD (as it was more freely licensed than Linux) - from memory most of the TCP/IP stack was based on BSD's stack. I don't think it came from OpenBSD though - more likely to have come from FreeBSD or one of the other distros.
I think that the POSIX layer was more that it complied with the POSIX standards (for some low numbered version of POSIX) rather than it could run unmodified code targeting POSIX. Either way, the POSIX and OS/2 APIs were killed when NT became XP.
The summary even states that Microsoft still officially consider it beta software. This is not even a v1.0 release. What we are seeing here however is that even a large ship such as Microsoft can turn very slowly. This would have been unthinkable even 5 years ago - just think what it will be like in another 5 years...
Slashdot Mirror
OK, now it's starting to make more sense looking at the use cases
Here is a description of a few of the popular use cases for Apache Kafka. For an overview of a number of these areas in action, see this blog post.
Messaging
Kafka works well as a replacement for a more traditional message broker. Message brokers are used for a variety of reasons (to decouple processing from data producers, to buffer unprocessed messages, etc). In comparison to most messaging systems Kafka has better throughput, built-in partitioning, replication, and fault-tolerance which makes it a good solution for large scale message processing applications.
In our experience messaging uses are often comparatively low-throughput, but may require low end-to-end latency and often depend on the strong durability guarantees Kafka provides.
In this domain Kafka is comparable to traditional messaging systems such as ActiveMQ or RabbitMQ.
Website Activity Tracking
The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type. These feeds are available for subscription for a range of use cases including real-time processing, real-time monitoring, and loading into Hadoop or offline data warehousing systems for offline processing and reporting.
Activity tracking is often very high volume as many activity messages are generated for each user page view.
Metrics
Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.
Log Aggregation
Many people use Kafka as a replacement for a log aggregation solution. Log aggregation typically collects physical log files off servers and puts them in a central place (a file server or HDFS perhaps) for processing. Kafka abstracts away the details of files and gives a cleaner abstraction of log or event data as a stream of messages. This allows for lower-latency processing and easier support for multiple data sources and distributed data consumption. In comparison to log-centric systems like Scribe or Flume, Kafka offers equally good performance, stronger durability guarantees due to replication, and much lower end-to-end latency.
Stream Processing
Many users of Kafka process data in processing pipelines consisting of multiple stages, where raw input data is consumed from Kafka topics and then aggregated, enriched, or otherwise transformed into new topics for further consumption or follow-up processing. For example, a processing pipeline for recommending news articles might crawl article content from RSS feeds and publish it to an "articles" topic; further processing might normalize or deduplicate this content and published the cleansed article content to a new topic; a final processing stage might attempt to recommend this content to users. Such processing pipelines create graphs of real-time data flows based on the individual topics. Starting in 0.10.0.0, a light-weight but powerful stream processing library called Kafka Streams is available in Apache Kafka to perform such data processing as described above. Apart from Kafka Streams, alternative open source stream processing tools include Apache Storm and Apache Samza.
Event Sourcing
Event sourcing is a style of application design where state changes are logged as a time-ordered sequence of records. Kafka's support for very large stored log data makes it an excellent backend for an application built in this style.
Commit Log
Kafka can serve as a kind of external commit-log for a distributed system. The log helps replicate data between nodes and acts as a re-syncing mechanism for failed nodes to restore their data. The log compaction feature in Kafka helps support this usage. In this usage Kafka is similar to Apache BookKeeper project.
I've got no idea what Kafka does, and the summary really doesn't tell you much at all. I was about to put in a helpful post saying what it is, but even after visiting their home page I've still got no idea.
Apparently Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
How about the Intro
We think of a streaming platform as having three key capabilities:
It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system.
It lets you store streams of records in a fault-tolerant way.
It lets you process streams of records as they occur.
What is Kafka good for?
It gets used for two broad classes of application:
Building real-time streaming data pipelines that reliably get data between systems or applications
Building real-time streaming applications that transform or react to the streams of data
OK, I still am not really sure what it does.
I have a 2013 Mac Pro and a new 2016 MacBook Pro 13".
Whilst multi-thread performance is a different matter altogether, single core performance is pretty much on par (with a slight edge to the laptop) when comparing the two machines. The vast majority of software I run is single threaded, as I don't do video editing, 3D or gaming.
This is a Intel Xeon E5-1620 quad-core versus an i7-6567U
https://ark.intel.com/products...
https://ark.intel.com/products...
Power consumption is 130W to the Xeon versus 28W to the i7.
It wasn't me, my cat walked across my keyboard!
I honestly don't know. At which point do you need to pay for Java?
Is it you need to buy a licence to write code in Java? Run the code you've written? Distribute your Java code to others?
$5k per 20 seats per year.
This works out at $250 per seat per year. This is less than a dollar a day. If you're running a company with 20 developers and can't afford $1 per person per day then you're doing something wrong.
Yes, damn that one friend who's late and can't enter with you - they just need to get their act together :D
Never gonna happen? It's already happening...
A possible solution is to not ban sales to bots per-se, but instead verify that the identity of the person redeeming the ticket at the door is the same as the person who purchased the ticket (via verifying CC details, or even something as basic as their name).
If tickets have conditions on them that prevent their usage by anyone other than the person who originally bought them, then there can be no market for resold tickets. Let the scalpers buy as many tickets as they want, but eliminate the market for them to be resold.
Ticket Australia now state as part of their conditions of sale "This ticket may not, without the prior written consent of Ticketek or the Seller, be resold at a premium or used for advertising, promotion or other commercial purposes (including competitions and trade promotions) or to enhance the demand for other goods or services. If a ticket is sold or used in breach of this condition, the bearer of the ticket will be refused admission."
If you knowingly purchase a scalped ticket, you're taking a huge risk that you won't get in to the event.
I remember this - I had a cron job running once every 5 minutes that would use curl to get the current weather report, parse that for the temperature and update the LCD on the printer. Good times indeed...
With my car, once it's been locked with the button on the key fob, after a certain amount of time, it deadlocks the doors - they can not be opened from the inside or outside without being unlocked. The unlock button on the driver's door will no longer function either after the car has been locked from the fob.
This means I could, if I wanted to, lock the car with the windows partially down and after a minute or so the car would be deadlocked - even if someone reached in to open the door, they would be unable to.
Apple are doing what they have done every single year - retiring old models from their supported lineup. Film at 11.
Every year, a range of Macs pass through the range of support status from "Supported" to "Vintage" to "Obsolete"
Vintage products are those that have not been manufactured for more than 5 and less than 7 years ago. Apple has generally discontinued hardware service for vintage products in most regions other than the state of California and Turkey.
Obsolete products are those that were discontinued more than 7 years ago. Apple has discontinued all hardware service for obsolete products with no exceptions. Service providers cannot order parts for obsolete products through Apple.
https://support.apple.com/en-a...
Game of Thrones seems to be set on an Earth-like world, however with dragons, undead and a degree of magic. They have winters that are sometimes brief and other times plunge the world into a freeze that lasts for generations...
Consider tidal locking in a system with an M-dwarf star, a "hot Jupiter" and our Planet of Interest (PoI). If in orbit around either the hot Jupiter or the star, the PoI might become locked. But with the three in relatively close interaction, the PoI could be disturbed between locking to one, or the other, or alternating, or spinning irregularly. Feel free to use a planet with an irregular - literally chaotic, even - rotation in an SF scenario of your choice.
So that explains the irregular seasons in the Game of Thrones universe... Winter is coming.
s/Screaming/Streaking/
A Naked Black Hole Is Streaking Through the Universe
There, that's better.
And, by implication if it is now as secure as the iPhone, then until recently it wasn't?
It's trivial to install Samba on OS X.
Step 1. Install Xcode
Step 2. Install MacPorts
Step 3. sudo port install samba3 or sudo port install samba4
Apple can't include Samba out of the box with OS X due to issues linking to GPL v3 libraries and issues due to foregoing patent lawsuits if using GPL v3 code, but there's nothing stopping you from adding it yourself.
Yes, there is Apache licensed code, BSD licensed code etc... The problem with the GPL is that it was previously able to be used as a part of a larger non-open source offering and with the change in the licence to v3, this is no longer possible.
That's the problem with GPL v3. With V2, you could link to GPL'd code, or use GPL'd libraries, in your closed source project and that was OK. If you made changes to the GPL'd code, you needed to make the source for the changes available to anyone who you distributed the software to.
With v3 however, if you use GPL'd code or libraries in your closed source project, you have to make the ENTIRE PROJECT available under the GPLv3 as well. This is the viral nature of it.
From http://choosealicense.com/lice...
Permissions of this strong copyleft license are conditioned on making available complete source code of licensed works and modifications, which include larger works using a licensed work, under the same license. Copyright and license notices must be preserved. Contributors provide an express grant of patent rights.
If you include GPL v3 code in your closed source project, under the terms of the licence agreement, you must comply with the licence which demands either you licence the software or you release ALL of the source of your closed-source project under the GPL v3.
If you are producing a large, closed-source, product, option 1 is not an option at all.
If you can't negotiate a licence with the copyright holders, or the copyright holders are unwilling to licence it to you on terms that are acceptable to both parties then option 2 is not an option at all.
So, this leaves only option 3 - remove all GPL'd code from your project and write your own.
I feel that the viral licensing clauses in GPL v3 will ultimately hinder the further development of software.
While Apple (as an example) were using GPL v2 licensed code, they were actively contributing patches and changes back to the relevant projects. This was good, it meant that we had professional developers who were paid to work on these projects and the changes they made were contributed back upstream.
Now that no corporation can touch any GPL v3 licensed code, we now have fragmentation and less developers working on open source code bases.
Apple, for example, have had to implement their own SMB stack as smbx, instead of using Samba. For a number of years, SMB compatibility and functionality took a huge step backwards on OS X, all because the Samba project started to use GPL v3. This ended up with developers who would have been working on patches and changes for Samba, instead working on their own closed-source implementation that, quite frankly, was nowhere near as good or as mature as Samba. The end result of this was that Apple's customers suffered with a sub par product and the Samba project suffered with fewer people contributing to it.
Who then benefits from GPL v3?
" if you have hardware and software that can create video at >4GB/s you're pretty special."
Protip: Most game modern engines can EASILY do that with just the video card. AGP had a maximum throughput of 2166MB/s, half your requirement. That was replaced roughly a decade ago.
Bandwidth of a video card interface and the throughput of a 3D gaming engine is irrelevant to this discussion. We're talking creating video through something like Avid or Premiere/After Effects by compositing multiple streams of video and effects and then mixing that all down to a single stream for output.
Whilst some acceleration of this is done on the GPU (and this is why the 3 year old Mac Pro has 2 GPUs even in it's base configuration) the main requirement here is fast and consistent throughput to mass storage.
There are plenty of video professionals that use a Mac Pro with it's stock 256 GB internal SSD and then hook it up to an external 8- or 12-bay Thunderbolt RAID (or two) that can transfer in excess of 2 GB/sec - something like this http://www.lacie.com/as/en/pro...
I don't care how big your PC workstation case is, if you want 50 or 100 TB of storage, you're looking at external RAID or SAN anyway. The internal storage is only used for the OS, apps and scratch space.
The Adult Population of the USA is something like 194.5 Million people.
Let's say that you can get by on $25,000 per year, tax free.
Providing UBI for this many people will cost the economy 4.8 Trillion Dollars. Where is this going to come from?
OK, let's scale this back a bit. We will give every adult in the USA $200 per week - $10,400 per year. We're still talking about $2.02 Trillion - this is 11% of the entire GDP of the USA.
To put this in perspective, the USA spends $810 Billion on public education per year, $1.3 Trillion on pensions and almost $600 Billion on defence.
You're thinking of Services for Unix (SFU) - that was part of it, and the other part was the POSIX-compliant API that Windows NT 3.1 was based on (and remember that NT 3.1 was really NT 1.0).
Windows also incorporated code from BSD (as it was more freely licensed than Linux) - from memory most of the TCP/IP stack was based on BSD's stack. I don't think it came from OpenBSD though - more likely to have come from FreeBSD or one of the other distros.
I think that the POSIX layer was more that it complied with the POSIX standards (for some low numbered version of POSIX) rather than it could run unmodified code targeting POSIX. Either way, the POSIX and OS/2 APIs were killed when NT became XP.
The summary even states that Microsoft still officially consider it beta software. This is not even a v1.0 release.
What we are seeing here however is that even a large ship such as Microsoft can turn very slowly. This would have been unthinkable even 5 years ago - just think what it will be like in another 5 years...