Open Source, GIS and Data Visualization?
Mubarmij asks: "A lot of people, including the ex-Vice President, think that Terrain Visualization and the Georeferencing of all kinds of data is the next big thing. Given the broad applications (sims, education, games, GIS, virtual
tourism, etc) that can be derived from such technology, I would tend to agree that if this is not actually the NBG(tm), then at least it is very close. Like the internet, this technology has taken its time in obtaining it's current level of sophistication. However,
there is huge potential here that has yet to be tapped, despite the fact that it currently fills a huge niche market. I had once read that NASA spends more than 70%
of its resources on space imaging and visualization-related activities (unfortunately I link to the article that mentions this, but one should remember that the major goal of all space satellites is to take multihued pictures of Earth and other planets, and you will see that this is not an exaggeration), which is quite a
lot of money." Open Source has provided several frameworks for GIS from which a "killer app" may spring from. Read more on the various Open Source projects on GIS, and feel free to share your thoughts on where this technology may head in the future.
"There are quite a few web sites, commercial and non commercial that tend to this technology. However, it seems like the early nineties, where people are just starting to get aware of the internet, but are still awaiting for the killer app to make this thing fly.
There are two open source projects I am aware of that deal with this area. The first, VTP, is a real open source project attempting to create a real terrain visualizer. The second, OpenSkies, is not really open source yet (despite its claim)... but it is interesting in that it allows networked people to fly or drive through virtual worlds that are reality based.
Here are a few other questions:
- Do you think that this technology will remain a niche market (albeit a big one)? If so, is this likely to occur?
- Are you aware of any open source projects other than the two mentioned above that deal with this area?"
VTP looks really cool.
:)
Source Download Is Here
I'll tell you if I can build it.
---
pb Reply or e-mail; don't vaguely moderate.
pb Reply or e-mail; don't vaguely moderate.
Sorry, Ben; I couldn't find an easy way to get to the source, but google could.
:)
Perhaps in the future you might consider using a robots.txt file to help restrict this as well?
Since it doesn't look like I'll be able to build this on Linux easily, you won't have to worry about my copy of the source.
---
pb Reply or e-mail; don't vaguely moderate.
pb Reply or e-mail; don't vaguely moderate.
You cannot just sit in your room and hack those out.
No, but you can start with US Census TIGER data, and work with dozens of other sources of free data.
I'm currently doing a program that generates databases for a free palm pilot flight planning program called CoPilot. Getting the data and writing scripts to import it is the hard part, in part because new data comes out every 28 days. The only problem I have right now is that the FAA "order form" for digital data doesn't have any feedback, so two weeks after you filled in the form and you still don't see any data, you don't know if that's because they didn't get the order, or they're just really slow processing orders, or they are waiting until the next data update cycle. So I have some older less complete data from another source for now, but I think I know where to get the latest and best data cheap.
The next Cmdr Taco duplicate will be ready soon, but subscribers can beat the rush and see it early!
The really sad thing is that I worked for ESRI's big competition (GeoVision) in the 1987-1992 time-frame. Back in 1991-2 I got interested in Linux, and I actually considered stealing a copy of the source code from work to try and port it to Linux (which I would have given back to the company). It wouldn't have been a big deal, since it ran on SunOS, Ultrix and AIX. The only problem at that time was the lack of a decent SQL database for Linux. Mostly our stuff ran on Oracle, but it also ran on a few others. The other major problem was that the only way I could transfer all the source code for a huge GIS application from work to my home PC was on 3.5" floppies. That would have been more painful that installing SLS 1.03 from 5.25" floppies, which I had also done.
GeoVision's product is still around, after going through a series of owners. It's still a damn good GIS. And it would probably be dead easy to port it to Linux now, since Oracle is on Linux. Heck, I think even Oracle MultiDimension, a product I worked on at Oracle, is available for Linux.
The next Cmdr Taco duplicate will be ready soon, but subscribers can beat the rush and see it early!
--
Care about electronic freedom? Consider donating to the EFF!
Or was that supposed to be NBT (Next Big Thing)?
I see even classic Slashdot is now pretty much unusable on dial up anymore.
Human beings are pattern recognizers. Our hardware is wired best to look/listen/feel/taste and match that stimulus to some other previously established pattern. We put a lot of effort into data visualization because it is easier for us to look at a picture and create a pattern than to look at raw data and create a pattern. Even the most difficult mathematical phenomena (noncommutative groups) make better sense when you give someone a concrete or highly visual example (rubik's cube, arrangements of books, etc.)
Sometimes the most brilliant of mathematical work comes from building a simple model that greatly simplifies the work (think Feynmann diagrams).
This can even apply to theology: when God intersects in our lives, we try to create a pattern to explain what God is, even though God is unfathomable. And thus we end up with many people who end up believing in Jesus or Allah or Nirvana, and then some people who end up believing in the UFO behind the comet. This also uncovers the problem with pattern matching: sometimes the patterns we find are dead fucking wrong (Aristotle). Fortunately we also have scientific method and reason to weed out self-contradicting, nonsensical, or otherwise false patterns.
ObJectBridge (GPL'd Java ODMG) needs volunteers.
Finding God in a Dog
Although it is difficult for many of us to see things, such as The Lord, the fact is that after some proper imagination and hard work, anything is possible.
You made a Hell of an interesting point. Exactly a Hell of it. As there is a WHOLE set of testimonies of visions of Hell. And a few ones of Heaven. Now wouldn't be interesting to MODEL them? Maybe they would help some people to see what is waiting them.... And help morale 'round here on Earth.
Anyway that's not for me. I, like every old Hacker who lived the end of the XXth century, will go to Heaven...
"Ok my son, you are in Heaven, what do you wish?"
"Infinite munition/all weapons... God Mode... and send me to HELL!!!!"
Well, firstly that gawd-awful "shapefile" format was created (and since modified) solely by ESRI.
An "industry standard" perhaps - as ESRI controls 60-80% of the GIS market (depending upon what segment you look at). Secondly, a shapefile is not the standard way in which Arc/Info stores its data - these binary formats (e.g., the "e00" files) are entirely proprietary. There have been some attempts to try and reverse engineer this format - use google to look up "e00 format".
GIS technologies have many applications beyond the arena of geographic information systems. Additionally, it should be noted that GIS itself has benefited from the convergence of many different fields that have been traditionally open source. GIS is essentially a combination of a number of different areas of study that resulted from needs outlined by the DOD and NASA to study planets remotely and to study the earth from airborne platforms and more recently, (starting with Corona) space. Mathematics, geometry, and statistics (all open source at heart) are at the roots of GIS. Where future development of GIS technologies will go is anyone's guess, but should not be limited to traditional areas of GIS application, and realizing the potential benefits of open source paradigms can help a variety of fields that have fairly esoteric uses that can have huge impact upon society. For examples consider that image classification algorithms (both supervised and unsupervised) can be applied to many areas of bioscience to determine crop health, or determine tumor potential in both plants and animals. We use GIS tools, specifically unsupervised classification routines such as k-means and ISODATA to determine the biochemical phenotypes of tissues at cellular resolution, and the whole field of genetic microarrays has recently started using these same clustering algorithms to mine gene data. Other examples include ophthalmology which has benefited along with obviously dermatology and computer vision (in hundreds of applications from food safety to manufacturing), political science, geology, etc etc etc....
Visit Jonesblog and say hello.
One open source project not mentioned is CAVOR, ("Creating A Vision Of Reality" if you insist that stand for something). (See the Cavor project page at SourceForge)
This is primarily the engine for a GIS, the database, graphics display, user interface, scripting, etc, that developers or end-users can further tailor to their specific application. The same engine can be used for other applications that include both a spatial/graphic part and a database part (eg CAD, UML diagrams, PERT charts, etc).
It's still in development -- the display list manager is working (and used in a spin-off, 'cvv'), the database interface and scripting language is in progress.
The architecture is based loosely on GeoVision/SHL/AutoDesk's high end "VISION*" system. (In its time -- before bankruptcy and transfer of the technology to SHL then AutoDesk -- GeoVision pioneered several of the technolgies widely used in GIS systems today, such as storing the spatial data in a relational database.)
The home page is at http://www.cavor.org although I'm not sure the server is up to a slashdot effect.
-- Alastair
GIS just involves a lot of data
Yep, and it used to be that that data was very expensive -- involving lots of hand labor digitizing maps and aerial photos. That's still true to a certain extent, but much less so.
I've gone from one elegant-but-buggy GIS product (VISION*) to just doing "GIS" by connecting my CAD maps (Microstation) to databases,
I'm curious as to when you were using VISION* and it what context. It's been nearly a decade since my involvement with it (I did a lot of the requirements analysis and design for 2.0 -> 2.1).
And connecting CAD to a database is a nightmare -- I was once involved in converting a hybrid AutoCAD+dBase system to VISION*'s forerunner. Yuck!
Oh, and PS, that bodes ill for Open Source GIS software outside the academic world, because big organizations have a positive fondness for the "Microsofts" of any industry they buy from, and an aversion to "unsupported" products.
Sure, that's AutoDesk (who currently own VISION*) isn't going to be worried about CAVOR. Were I major telco or power company or large regional government I'd go with them for the support, training, handholding, etc. (Heck, when I was with GeoVision we wouldn't talk to anyone with less than a quarter million to spend).
But there's still a niche for open source GIS. Heck, look at the "travel map" software out there -- that's a rather simplistic GIS, to be sure, but it shares common elements. Think of what, say, real estate agents could do with a GIS system.
And beyond that, think of the other application domains that share characteristics with GIS. You mention trying to tie a CAD system and a DB together. How about a CAD system with a built-in DB interface? Or any number of other applications that have both a spatial or graphical aspect and a database aspect -- software diagramming tools (UML, other CASE tools), project management (PERT charts), various sorts of CAD, and so on. (This is part of what the CAVOR project is all about -- I always felt that GeoVision never quite understood the potential of their technology. I once prototyped a CASE tool (DFD's, ERD's and the like) on VISION* in about a day.)
Give folks a free toolkit like that and the uses and applications will come, and they won't all (by a long shot) require gigabytes of data. It's when the tool cost is high that it's only used by those with a lot of data to manage.
-- Alastair
Hi Paul!
Take a look at Cavor (http://sourceforge.net/projects/cavor/). It isn't exactly VISION*, but the overall architecture is similar and that's my model (as much as I can remember, anyway).
Cheers,
-- Alastair
(Another ex-GeoVisioner -- and yes, the architecture beat heck out of what ESRI was peddling back then. I haven't been close enough to it lately to know about now.)
-- Alastair
As for the field of displaying quantitative information, the recommended books are Tufte . It is actually quite hard to create intuitively understood data visualisations because our eye-brain can only measure simple things like intensity, distance, etc. That's why things like pie charts where the angle is directly proportional to the propoertion works. All the other data visualisation techniques (parallel coordinates, tensors, etc) require a fair amount of training and patterning before you can pick up the meaning. A geologist (or related discpline) would be able to look at a contour map and be able to "see" in eir mind's eye the slope and elevation. Lesser mortals would probably require a pan of a 3D VRML model and even then have difficulty in recalling specific features. Adding extra layers or texture maps might be aesthetically pleasing (cough*QUAKE*cough) but doesn't really add any extra information.
LL
It's mainly for the folks who need a 2D, Java-based solution. It's pretty darn useful; lots of great projection math utilities, a nicely structured architecture, support for a wide variety of GIS products out there, and has transparent support for delivering imagery over a network. Plus the platform of choice for primary developers (BBN) is Linux. :)
http://openmap.bbn.com
I agree, ESRI is worse than MS. Government agencies all over the world are now all migrating to ESRI as ESRI locks up the market. ESRI now owns the default file standard (just as MS owns the Word .doc standard and can keep messing with it to make life difficult for competitors). But in ESRI's case, a seat at ESRI's ArcInfo costs about $18,000. So if you want to open one of those, now standard file types, better be prepared to pony up some real cash. My small town government can't afford this, and the county is even more financially strapped, but they feel they must in order to keep up to date, technologically.
Really, I wish the GRASS project were moving faster (or moving at all). It was well positioned some years ago, but hast lost badly to ESRI in the past year or two.
Esri HAS competition . For example Smallworld GIS Is available For Linux. Smallworld GIS really beats ESRI in all aspects except for perfomance and price. It has a java-like portable virtual machine, and a smalltalk-like programming language, which probably scare lot's of potential users.
signatures pending - ansa@kos.to - (dont mail there)
everyone remember the challenger explosion? anyone remember that a couple of NASA scientists had done a bunch of research on o-rings and had determined that under specific conditions, that they wouldn't work as they were supposed to? but the challenger launch wasn't stopped, because the research was presented with such poor visualization that the managers of the projects (who didn't really understand the science) couldn't make informed decisions. part of visualization is helping really, really smart people show off their ideas to the rest of us.
I am the king... of No Pants! www.penny-arcade.com
You have a point. Perhaps the Einsteins of the world shouldn't use advanced graphics. However, for the rest of us, it helps a whole lot. So, this is not an effort to help the best and brightest achieve their best, they'll do that no matter what. Rather, this is an effort to help everyone else.
And you really don't have to bring your religon into things to make a point...
All this info about GIS is nice, but it seems to me to be a little high-end: it's not aimed at the needs of the average user.
What I would like to see is somebody making an application like Delorme's AAA MapNGo or Microsoft's Streets and Trips available.
However, those sorts of programs are difficult to do under the Free Software model. The code isn't hard (no harder than a browser or game engine) but the program is worthless without data. And, unlike a program which you can stay in your little room and write, data requires you to have detailed maps of street locations and interconnections, locations of attractions, hotels, restaurants, and gas stations. You cannot just sit in your room and hack those out.
Now, if we could just persuade Delorme that the Free Software community is a good market...
www.eFax.com are spammers
GIS just involves a lot of data - and a lot of problems cross-referencing separate but related datasets. In our case, its the (say) water, sewer, parks, roads and legal maps (and 22 others) that are all drafted separately but must register on top of one another correctly to create synergy.
I've gone from one elegant-but-buggy GIS product (VISION*) to just doing "GIS" by connecting my CAD maps (Microstation) to databases, to a second run at GIS with the Microsoft of the GIS world (ESRI). And they were right; the technology was never my big problem. Staying on top of the ever-mutating dataset is 98% of the work.
I see GIS having big effects in the government and industrial world but almost none at the consumer level, because you have to be a big organization to manage enough *data* to need a GIS to understand it all.
There will be consumer products and services - your PDA will show you the direction to the nearest Thai restaurant - but the only technology you'll need to get this from some remote GIS engine will be a browser.
Oh, and PS, that bodes ill for Open Source GIS software outside the academic world, because big organizations have a positive fondness for the "Microsofts" of any industry they buy from, and an aversion to "unsupported" products. Hate to be the messenger, but its just a corporate-culture fact of life.
When I first read the headline I thought it was about Geek in Space. oops.
Stupid Cheap Guitars
Brought to you by the good folks at
http://www.kitware.com/, yes its' open source.
VTK has been used in a lot of data viz.
I did get around to using in my project because
it *seems* (could just be my lack of expereience) more difficult to visualize dyanmic data, and things like VAS are more suitable.
But it's a really nice piece of software (wrapped in many languages including python!)
I have tried for four years to get the GIS industry leader Environmental Systems Research Institute (ESRI) to port its products to Linux. They have expressed interest, but so far, have ported zero product. The problem is that they have not only leadership, but actual dominance in the industry You can send email to jdangermond@esri.com. Jack Dangermond is the president of the company, and a reasonable man. He has stated in the past that product ports are driven by user demand. Please be polite, ask for the specific product, and refer him to this slashdot site. I think that if he sees that the userbase is considering jumping to Open source, it might help his decision. Thanks Greg
Anyone remember the "Earth" application from Snowcrash (Neal Stephenson)? That seems to be the ideal point towards which all of this is evolving.. check out the Planet-Earth project which is directly inspired by Snowcrash as well as H2G2 and Everything2.
Don't blame me - I voted for Howard Dean. http://dean2004.blogspot.com
this is for 3d representation of gis data
http://www.ai.sri.com/geovrml/
very cool!
the http://www.parallelgraphics.com
cortona vrml plugin has support for it too
back in the day we didnt have no old school
I currently use GIS data to build 3d trail maps, "fly-by" style video, and expansion visualization for ski resorts. There is a lot that can be accomplished with geo data, but I don't necessarily think it'll change the way we all live.
Skiers and Riders -- http://www.snowjournal.com
Opengis.org is the best organizational reference for open GIS standards. They have an international consortium of business and government agencies behind them. They have been around since 1993 and have developed several standards for developing a true open framework for GIS delivery. In fact, GIS is one of the rare applications that demands a very open approach since having geographic data is only useful as it relates to other geographic data.
Opengis.org has done a good job of specing out systems that are truly interoperable because they achieve GIS nirvana: seperating content from visualization. Reading GIS content from multiple servers and displaying it through a single user interface is the heart of open GIS. Amazingly, no major commercial vendors (ESRI, Bentley) are aggressively pursuing this vision. IMHO, this is an opportunity for the open source community to make a mark on a major emerging industry! If you are interested in working on developing an open source version of the server spec that Opengis has released, please contact me!
The reason, is simple.
Most beginning projects focus on the purchase of GIS software, like ESRI's ArcInfo, the purchase of sexy machines to run it, and perhaps some initial staffing to set it up. They almost uniformly massively underestimate the amount of time and money required to actually get a functional landbase into the system, attach data to it, and ensure the necessary level of accuracy of the basemaps.
"It'll be easy, we have DB2 running on an as/400 and lots of maps" - should qualify as one of the most expensive single instances of "famous last words" in computer history.
In reality, very few maps are accurately digitized. Most GIS systems get sold on the premise of scanning and vectorizing dead-tree bluelines, or other formats - from paper. the problem is, that geographic maps are finicky about scale, and the maps you *thought* were accurate inevitably need to be rectified to whatever coordinate system you chose for the system. Its a tedious and enormously time-consuming process that simply doesn't lend itself to any sort of automatic processing. Then, there's the problem that most maps simply aren't accurate to begin with, and you see the beginnings of a problem. "I just bought half a million dollars of GIS software and equipment, and now I discover it's going to cost me three or four million to get my maps in shape to actually be useful".
Picture your average City government. Municipalities are where government really is. About 90% of the civil services provided to you by the sum of all government influence on you, comes from the local level. Cities have huge quantities of maps for things like building plats, subdivision maps, building blueprints, deed records, thoroughfare maps, land-use plans, etc etc etc. And that, coincidentally is where GIS appears to be the most useful. managing all that data is hard for cities - and GIS certainly looks easy in the presentations companies and consultants present - just point and click! What could be easier?
But the numbers of abandoned systems started by cities who bought into these sales pitches are huge - on the order of billions of dollars worth.
GIS companies and consultants know this - yet never warn purchasers, that in most cases, the cost of the software and systems amounts to only a tiny fraction of the actual costs of developing a working GIS.
So when you hear all the gushing success stories, and gee-whiz ideas presented as though GIS was a wonder cure for all sorts of problems, try to bear in mind - GIS works best with single-source maps, and data which can easily be applied to pre-existing points. If you hear the words "scan" or "address-range" - you'll know a bullshit artist is at work. Because for every working system involving simple maps with pre-rectified sources, there are probably 15 that were simply abandoned because of enormous unexpected costs and (salesman provoked) unrealistic expectations. A lot of money has been pissed away on GIS over the years. Real computer professionals know the golden rule - flashy graphics doesn't equal useful purpose.
The other cool thing about it is the protocols and data formats it uses to transfer information in both it's 2D and 3D mode are publically available. The idea is in the same way Palms are popular because anyone can develop for them, that's the goal with this product.
In fact, the 3D client is Open Source!
I hope this might be some where close to the kind of tool you're looking for.
antarctican at trams dot ca
I also have some slick cartograms, which make even boring economic/demographic data seem cool.
andy a.
DEMster? Yes, well, here are the possibilities: Global Data Sharing/Referencing
Actually Napster would a great analogy to follow, most of the terrain data in the world is proprietary and expensive, tightly controlled by the governments and big companies that produce it. Not that i'm suggesting anything , but P2P would be interesting way around that situation. The data only has to leak once for the genie to get out of the bottle..
-Ben>