I heard a valid reason from Dean Pesnell (SDO Project Scientist) -- they wanted to compare it to a compressed media, rather than using units of 'library of congress'. Either someone misheard the quote, or someone screwed up their line, as it was supposed to be something like 'downloading 500k songs from iTunes a day'.
For some reason, they weren't willing to go with the PornYear metric.
(and to the other commenter who said they should've said 'MP3' -- iTunes uses AAC, not MP3, and if you said file format only you'd have to mention at what quality, etc. Not to mention that the bad math analogy you went with is for telecommunications pricing, when you could've gone with the bad-metric-conversion analogy.)
It's your basic blog spam -- slashdot user 'coondoggie' submits an article written by 'Michael Cooney'.
Look at the rest of his submissions -- all just links back to Network World. Maybe he's trying to make up for the loss of Roland. (Although, Roland got better in his submissions)
Even if someone made it, you have to go through hoops to be allowed to use peer-to-peer anything at NASA.
I know there's a few multi-GB earth science data sets being distributed using torrents, but when I brought up peer-to-peer anything 2 years ago, before I was working on the data distribution system for SDO, I kept getting push back -- the files are too small to be efficient (~16MB each image); there's too many files to track (with all of the processed forms of the data, we'll be tracking over 400k files per day; more if they recalibrate)... I finally gave up.
Maybe we'll get approval for education & public outreach... we'll likely have to wait to see how hard people hit the servers, and might be able to use the bandwidth savings as justification. (but it'll still take months before we'd be allowed to do it)
The issue is that when you're flying long distances, you have the choice of either flying over the poles, or refueling mid-way. If there's a solar storm going on, everyone's exposed to a fair amount of radiation in a polar flight, and it might affect some of their instruments. Most airlines will take the refueling stop if there's a storm.
The radiation likely won't be enough to affect the average passenger, but it's the pilots who get to decide, and it's the flight crews that are exposed to radiation over and over again on these trips.... but it'll be more important when we move to GPS for air traffic control -- GPS doesn't work when there's too much noise in that frequency band. This would mean that the FAA would have to fall back to radar, and all of the benefits they're claiming for their new system would be wiped out. (ie, need to leave more space around planes, so you can't pack the airspace as well)
MODIS, eh? Earth science (specifically, the EOS-DIS) gets an order of magnitude more money than what we have to work with in solar science.
As for data storage, there's actually more than one system Besides what was already mentioned of the store and forward at the ground station, there's then the 'SDO JSOC' (Joint Science Operations Center) which will provide storage for AIA and HMI... but they'll be pushing the level 0 to tape after only a few days. They won't be archiving level 1 for AIA, and will have to recreate it on demand as they'll only be keeping it on disk for a week or so.. HMI lev1 products will be archived, but they're much smaller than the original data stream and more costly to process. Most scientists, however, will never touch this system. There's also a mirror at Lockheed for the AIA PI team to work off of.
The level 1 AIA and HMI products will then be distributed to a series of systems at various locations (Smithsonian Astrophysics Observatory, National Solar Observatory, Goddard's Solar Data Analysis Center, Royal Observatory of Belgium, Max Planck, U.C.Lancashire, etc.) We still haven't seen the real data yet, so we have no idea how well it'll compress, and so how many days of data we'll be able to store at each site. The Virtual Solar Observatory will then serve the data from these nodes out the scientists.
There will also be a 'cutout service' running at Lockheed to provide sections of the images on demand, for those people who don't need the full-disk images, if they're just studying active regions and other smaller features..
EVE, I'm not familiar with their data system... I think they're packing much of it into a database for direct processing. At the very least, we're planning on packing it into a database for the education and public outreach website so we can generate plots and such.
Oh... and the 1.5TB is the compressed data stream. Between AIA & HMI, there's almost 140k images per day, 16 megapixel, 16 bits per pixel each, so closer to 4TB/day raw.
Almost all of the links on that article refer back to crap at network world -- I'm still trying to figure out what this link is at the bottom, that claims to be "Solar Dynamics Observatory", but seems to just be a 404 to : sdo.gsfc.nasa.gov/images/site/icon.ico
(There's no 'images' directory on that server at the top level)
I'd just appreciate it if someone were going to link to our servers that they didn't link to crap.
My theory on 'white' as a race is simple -- it's a way to maintain the majority.
If you were to divide it out into the specific culture, then those who identify as 'black' or 'hispanic' suddenly outnumber each of the British / Italian / Polish / German / Swedish / Irish / etc.
The 'black' culture, as you point out, was forced upon them... the 'white' culture seems to be more a political choice.
(and for the record, I checked 'Hispanic' on the census forms, due to my 1/16th Argentinean ancestry.)
Sunspots are an indicator of activity, but don't actually affect the earth. The only reason sunspots are even used for comparisons is that there's more than a century of historical record.
The solar-related problems with GPS are CMES that take out the satellites or radio bursts that overwhelm the signal.
And we have no idea at this point when we'll start seeing the same levels of activity as the past solar maximums... but we current consensus is that the past few years have been abnormally low, and thus GPS may not remain so reliable.*
*in general, that is... my GPS has some issue with the antenna getting a static charge, and just sucks, so it's always unreliable.
First, they're college students, so they're at most 20-somethings. And it might not be that bad, but from experience, if you're working with the right type of people, you don't need nearly as much space as people think you do.
And if you hire a bunch of 20-something females, I'm guessing there's a few programmers that might be willing to take a pay cut to be packed into a room with them, so long as they're only overweight and not completely obese. (at least, by the scale that the Wii Fit uses)
You're assuming that the differences are something that someone can keep up with in real time. If someone makes a change in calibration that results in a few month's worth of data changing, it might take weeks or even months to catch up (as you're still trying to deal with the ingest of the new data at the same time). As for bittorrent, p2p is banned in some federal agencies -- and as such, we haven't had a chance to find out how well it scales to dealing with lots (10s of millions) of small files (1 to 16MB).
As for the low-level issues -- it's not even close. The problem is that people build their catalogs to handle the type of science they want to do; they often don't revolve around the same concepts, and they might have one or thousands of tables. See my talk Data Relationships: Towards a Conceptual Model of Scientific Data Catalogs from the 2008 American Geophysical Union.
I've been working for years with people who want to search the data from the systems I maintain, but the way that they want me to describe the data to make it searchable aren't easy to define -- even terms like 'instrument' mean something different between their system and mine. (and I have a paper submitted for the Journal of Library Metadata's special 'eScience' issue, dealing with issues in terminology and other problems that the library field doesn't typically run into, but we have to deal with in science informatics)
Disclaimer : If it's not apparent from the message, I work in this field.
*those* satellites are probably now defunct. It's the satellites that launched in the 1990s that are still dependent on running on old Alpha, Sparc or such hardware that are the current problem.
And there's no budget to hire programmers to port all of the software, run it through the proper testing, code review, etc.
So yes, there's a bunch of old systems hidden in closets and/or empty cubicles, limping along until the mission ends.
Disclaimer : I've seen one of those closets, and more than one of the cubicles. (because I work with folks that support a mission that launched in 1995)
Because the term's already been used for specific types of waves reflected through the sun:
I have reason to believe that this anonymous message in "STEREO Satellites Spot Solar Flare Tsunami" was posted by Joe Gurman, the Project Scientist for NASA's STEREO mission. (and for TRACE, and US Project Scientist for SOHO, and the head of the Solar Data Analysis Center):
You're right -- because it's SQL, which has assumptions about how it's used.
LDAP, on the other hand, you can set up to bind as the individual user, and you adjust which attributes a user is allowed to see or modify in their own entry, and which entries they can see in other entries.
So, part of the solution is using the correct data store for the situation, and SQL isn't always it. (I haven't played with any of the "NoSQL" stuff yet, but much of the behaviour with replication and and flexibility of storage seem rather similar to the LDAP implementations I've worked with.
(it has the voiceover from NASA TV, but doesn't have the launch clock visible... it might've been a camera angle that they didn't use live, as I don't remember seeing this on TV)
As you mention virtual servers, I'm going to guess that part of the problem is that you have large-ish servers.
My suggestion would be that if you have different uptime requirements for different services, to segregate them to different machines with a dedicated UPS.
Our office has between two to four 3000VA MGE Pulsars per rack, depending on how much power they draw and how long we need to keep things up. (Although APC now owns them, the MGEs are more power-dense than the APC Smart-UPS line, as they're only 2U each)
You're looking for a reference management system, not a document management system. (although, they might not deal with all of the stuff that you mentioned that a document management system will)
Zotero should work for a single person, but if you're trying to do this for an office, you might want to take a look at Aigaion.
They're calling for more than a few days of the year:
The disadvantges of this orbit include higher launch and orbit acquisition costs (relative to LEO) and eclipse (Earth shadow) seasons twice annually, During these 2-3 week eclipse periods, SDO will experience a daily interruption of solar observations. There will also be three lunar shadow events each year from this orbit.
These are *not* in situ instruments like what's on ACE, where it has to get hit to be known.
Part of SDO's mission in is space weather prediction -- trying to identify active regions that are likely to flare or throw off a CME, and thus provide more advanced warning so spacecraft can take precautions.
Disclaimer : I'm affiliated with one of the sites that's going to be distributing SDO data.
Slashdot Mirror
Lockheed Martin (AIA PI institution) has some of the 4096 x 4096 images available:
I have no idea why they didn't match the same color tables for the 304/171/19[35] images as SOHO and STEREO.
I heard a valid reason from Dean Pesnell (SDO Project Scientist) -- they wanted to compare it to a compressed media, rather than using units of 'library of congress'. Either someone misheard the quote, or someone screwed up their line, as it was supposed to be something like 'downloading 500k songs from iTunes a day'.
For some reason, they weren't willing to go with the PornYear metric.
(and to the other commenter who said they should've said 'MP3' -- iTunes uses AAC, not MP3, and if you said file format only you'd have to mention at what quality, etc. Not to mention that the bad math analogy you went with is for telecommunications pricing, when you could've gone with the bad-metric-conversion analogy.)
But it's in focus, without needing a servicing mission.
They've booby trapped their sun somehow!
http://www.youtube.com/watch?v=ZWYt8af7C3U
(start about 5:30 in)
It's your basic blog spam -- slashdot user 'coondoggie' submits an article written by 'Michael Cooney'.
Look at the rest of his submissions -- all just links back to Network World. Maybe he's trying to make up for the loss of Roland. (Although, Roland got better in his submissions)
Even if someone made it, you have to go through hoops to be allowed to use peer-to-peer anything at NASA.
I know there's a few multi-GB earth science data sets being distributed using torrents, but when I brought up peer-to-peer anything 2 years ago, before I was working on the data distribution system for SDO, I kept getting push back -- the files are too small to be efficient (~16MB each image); there's too many files to track (with all of the processed forms of the data, we'll be tracking over 400k files per day; more if they recalibrate) ... I finally gave up.
Maybe we'll get approval for education & public outreach ... we'll likely have to wait to see how hard people hit the servers, and might be able to use the bandwidth savings as justification. (but it'll still take months before we'd be allowed to do it)
The issue is that when you're flying long distances, you have the choice of either flying over the poles, or refueling mid-way. If there's a solar storm going on, everyone's exposed to a fair amount of radiation in a polar flight, and it might affect some of their instruments. Most airlines will take the refueling stop if there's a storm.
The radiation likely won't be enough to affect the average passenger, but it's the pilots who get to decide, and it's the flight crews that are exposed to radiation over and over again on these trips. ... but it'll be more important when we move to GPS for air traffic control -- GPS doesn't work when there's too much noise in that frequency band. This would mean that the FAA would have to fall back to radar, and all of the benefits they're claiming for their new system would be wiped out. (ie, need to leave more space around planes, so you can't pack the airspace as well)
MODIS, eh? Earth science (specifically, the EOS-DIS) gets an order of magnitude more money than what we have to work with in solar science.
As for data storage, there's actually more than one system Besides what was already mentioned of the store and forward at the ground station, there's then the 'SDO JSOC' (Joint Science Operations Center) which will provide storage for AIA and HMI ... but they'll be pushing the level 0 to tape after only a few days. They won't be archiving level 1 for AIA, and will have to recreate it on demand as they'll only be keeping it on disk for a week or so.. HMI lev1 products will be archived, but they're much smaller than the original data stream and more costly to process. Most scientists, however, will never touch this system. There's also a mirror at Lockheed for the AIA PI team to work off of.
The level 1 AIA and HMI products will then be distributed to a series of systems at various locations (Smithsonian Astrophysics Observatory, National Solar Observatory, Goddard's Solar Data Analysis Center, Royal Observatory of Belgium, Max Planck, U.C.Lancashire, etc.) We still haven't seen the real data yet, so we have no idea how well it'll compress, and so how many days of data we'll be able to store at each site. The Virtual Solar Observatory will then serve the data from these nodes out the scientists.
There will also be a 'cutout service' running at Lockheed to provide sections of the images on demand, for those people who don't need the full-disk images, if they're just studying active regions and other smaller features..
EVE, I'm not familiar with their data system ... I think they're packing much of it into a database for direct processing. At the very least, we're planning on packing it into a database for the education and public outreach website so we can generate plots and such.
Oh ... and the 1.5TB is the compressed data stream. Between AIA & HMI, there's almost 140k images per day, 16 megapixel, 16 bits per pixel each, so closer to 4TB/day raw.
Almost all of the links on that article refer back to crap at network world -- I'm still trying to figure out what this link is at the bottom, that claims to be "Solar Dynamics Observatory", but seems to just be a 404 to : sdo.gsfc.nasa.gov/images/site/icon.ico
(There's no 'images' directory on that server at the top level)
I'd just appreciate it if someone were going to link to our servers that they didn't link to crap.
If you want movies, see one of :
My theory on 'white' as a race is simple -- it's a way to maintain the majority.
If you were to divide it out into the specific culture, then those who identify as 'black' or 'hispanic' suddenly outnumber each of the British / Italian / Polish / German / Swedish / Irish / etc.
The 'black' culture, as you point out, was forced upon them ... the 'white' culture seems to be more a political choice.
(and for the record, I checked 'Hispanic' on the census forms, due to my 1/16th Argentinean ancestry.)
This was all predicted over 17 years ago -- Toys.
Sunspots are an indicator of activity, but don't actually affect the earth. The only reason sunspots are even used for comparisons is that there's more than a century of historical record.
The solar-related problems with GPS are CMES that take out the satellites or radio bursts that overwhelm the signal.
And we have no idea at this point when we'll start seeing the same levels of activity as the past solar maximums ... but we current consensus is that the past few years have been abnormally low, and thus GPS may not remain so reliable.*
*in general, that is ... my GPS has some issue with the antenna getting a static charge, and just sucks, so it's always unreliable.
Then do it -- vote with your feet, that's the only way that it's really democratic.
... as overpriced, required purchases that the teacher never ends up using in class?
... as ways for the teacher to push up the sales of things they wrote?
First, they're college students, so they're at most 20-somethings. And it might not be that bad, but from experience, if you're working with the right type of people, you don't need nearly as much space as people think you do.
And if you hire a bunch of 20-something females, I'm guessing there's a few programmers that might be willing to take a pay cut to be packed into a room with them, so long as they're only overweight and not completely obese. (at least, by the scale that the Wii Fit uses)
You're assuming that the differences are something that someone can keep up with in real time. If someone makes a change in calibration that results in a few month's worth of data changing, it might take weeks or even months to catch up (as you're still trying to deal with the ingest of the new data at the same time). As for bittorrent, p2p is banned in some federal agencies -- and as such, we haven't had a chance to find out how well it scales to dealing with lots (10s of millions) of small files (1 to 16MB).
As for the low-level issues -- it's not even close. The problem is that people build their catalogs to handle the type of science they want to do; they often don't revolve around the same concepts, and they might have one or thousands of tables. See my talk Data Relationships: Towards a Conceptual Model of Scientific Data Catalogs from the 2008 American Geophysical Union.
I've been working for years with people who want to search the data from the systems I maintain, but the way that they want me to describe the data to make it searchable aren't easy to define -- even terms like 'instrument' mean something different between their system and mine. (and I have a paper submitted for the Journal of Library Metadata's special 'eScience' issue, dealing with issues in terminology and other problems that the library field doesn't typically run into, but we have to deal with in science informatics)
Disclaimer : If it's not apparent from the message, I work in this field.
*those* satellites are probably now defunct. It's the satellites that launched in the 1990s that are still dependent on running on old Alpha, Sparc or such hardware that are the current problem.
And there's no budget to hire programmers to port all of the software, run it through the proper testing, code review, etc.
So yes, there's a bunch of old systems hidden in closets and/or empty cubicles, limping along until the mission ends.
Disclaimer : I've seen one of those closets, and more than one of the cubicles. (because I work with folks that support a mission that launched in 1995)
Because the term's already been used for specific types of waves reflected through the sun:
I have reason to believe that this anonymous message in "STEREO Satellites Spot Solar Flare Tsunami" was posted by Joe Gurman, the Project Scientist for NASA's STEREO mission. (and for TRACE, and US Project Scientist for SOHO, and the head of the Solar Data Analysis Center) :
You're right -- because it's SQL, which has assumptions about how it's used.
LDAP, on the other hand, you can set up to bind as the individual user, and you adjust which attributes a user is allowed to see or modify in their own entry, and which entries they can see in other entries.
So, part of the solution is using the correct data store for the situation, and SQL isn't always it. (I haven't played with any of the "NoSQL" stuff yet, but much of the behaviour with replication and and flexibility of storage seem rather similar to the LDAP implementations I've worked with.
... as the different touch-enabled browsers treat touches a little bit differently:
http://www.quirksmode.org/blog/archives/2010/02/do_we_need_touc.html#more
Closer to the pad, and less shaky:
http://www.youtube.com/watch?v=q9S0z1ofcIc
(it has the voiceover from NASA TV, but doesn't have the launch clock visible ... it might've been a camera angle that they didn't use live, as I don't remember seeing this on TV)
As you mention virtual servers, I'm going to guess that part of the problem is that you have large-ish servers.
My suggestion would be that if you have different uptime requirements for different services, to segregate them to different machines with a dedicated UPS.
Our office has between two to four 3000VA MGE Pulsars per rack, depending on how much power they draw and how long we need to keep things up. (Although APC now owns them, the MGEs are more power-dense than the APC Smart-UPS line, as they're only 2U each)
You're looking for a reference management system, not a document management system. (although, they might not deal with all of the stuff that you mentioned that a document management system will)
Zotero should work for a single person, but if you're trying to do this for an office, you might want to take a look at Aigaion.
If you want to look at others to see what best fits your needs, see:
http://en.wikipedia.org/wiki/Comparison_of_reference_management_software
And , if you still can't find anything -- try asking on the Code4Lib mailing list, as you might need one of the 'integrated' library solutions.
They're calling for more than a few days of the year:
These are *not* in situ instruments like what's on ACE, where it has to get hit to be known.
Part of SDO's mission in is space weather prediction -- trying to identify active regions that are likely to flare or throw off a CME, and thus provide more advanced warning so spacecraft can take precautions.
Disclaimer : I'm affiliated with one of the sites that's going to be distributing SDO data.