I think you'd be surprised at the complexity of the applications which use such file structures.:-) When the data itself is simple to store, the complexity of the application isn't really relevant.
Disk space on the mainframe isn't my problem as an applications programmer, and it doesn't seem to be an issue for a freespace file. Continuous space for each record is allocated up front when the file is created, and you are reading/writing data as fixed record sizes. A freespace record is ALWAYS a fixed multiple of a disk allocation size to prevent allocation issues. In our case, I don't actually know what the hardware does (and as an applications programmer that isn't a problem I care about), but logically we're taught that disk is allocated in 28-word sectors. On an OS2200 mainframe, freespace records are multiples of 112 words, and this is somehow tied to the way that operating system allocates data on disk.
What happens when you've pre-allocated, say, 1000 small blocks and 1000 large blocks, and it turns out you actually need 1001 large blocks?
Not an issue with a well-designed database. And historically it hasn't been. Keep in mind that I've been working with this file format almost continually since 1988, so I have plenty of experience seeing it in use in production transaction systems. For well-defined datasets, that isn't an issue.
Out of order writes are not an application issue. All the application cares about is that it can read and write data to a logical record. The underlying record handler has to worry about the specifics of getting that data to and from the disk.
I've never seen a program lose a key, but in our case the base freespace record management system has routines which run periodically against the database to find orphans, etc.
Something like scanning for a passenger name across all flights would be an interesting problem, but I wouldn't btoher to do that in freespace. I'd have a relational database off to the side and populated in parallel for that sort of query. Freespace is about *speed*, remember.
I don't think it would be hard to implement at all. It just takes time, something I can't spend (at work) writing code that isn't directly related to our paying customer needs. But I'm considering it for a home project. Probably in C. We'll see.:-)
Design issue. Optimally you use this sort of thing in places where you will always have an easily-available record number. It isn't a replacement for a relational database in places where relations are nice.:-)
In the airline context in which I work, you tend to see sorted indexes containing keys which are accessed by things like IATA station code (e.g., MSP or ATL), or airline code, or flight-date-origin (e.g., NW1492-24-MSP, which provide unique reference items which are easy to sort.
We also sometimes maintain a relational database for searching/reporting purposes on another lower-usage box just for reporting purposes. That frees the freespace file on the primary box to do it's thing quickly, and transactions against that fast database are also split off and inserted as rows against the relational database.
That allows for the speed of a freespace file in production while also giving reporting/query capabilities for past history... and in a way which doesn't impact production.
We generally use a simple flat file as an index. Field 1 is a sorted index field (say a flight number), field 2 is the key to a freespace record. A simple binary search is fast even on a large sorted list.
No, it isn't useful for certain types of applications. Relational databases exist for a reason.:-) But if you have to store something for a well-known static set of fields (say weather stations or flights in an OAG schedule file), something a lot simpler isn't a bad method at all.
In my experience, most UNIX programmers tend to assume a relational database for almost everything if it isn't a vanilla flat file. That includes programmers in realtime applications, C people, Java people, etc. I can't begin to tell you how many applications I've seen written to use Oracle, Sybase, etc., just to store a simple static table of information. There's no POINT to that!
Most mainframe environments, on the other hand, have many established options, and relational is usually only considered if you actually need that type of functionality. Normally, systems are written with a mix of different database types. Could be flat files, could be freespace, could be RDMS, or could (in our case) be DMS, a network database with some types of set-linking properties but not really table-based.
Again in my experience, working in the airline industry, I've seen a bias towards RDMS for enterprise applications that I simply haven't seen on the mainframe (in my case Unisys transaction processing) side of life.
A web site is a tranctions processing facility. Just replace fancy 3270 or Uniscope screens with HTML. Same idea, forms, etc. It wants in and out, fast. Why use something not really made for that?
01-0127-0001 is the first type 1 record for file 127. 01-0100-0001 would be the first for file 100. That's what I get for doing patchwork re-editing an existing message before sending it...
Oops. Forgot that brackets get eaten. Typical record format is RECORDTYPE/FILENUMBER/RECORDNUMBER. The first Type 1 record for File 100 might look like 01-0127-0001 or whatever (specific binary representation in hex or octal would obviously vary depending on implementation and preference).
In our case, it's a 36-bit word shown as 12 octal digits, probably not a popular choice with UNIX folks.:-)
I've never understood the UNIX world's fascination with relational databases.
Speaking as a programmer in mainframe online transaction environments for the past 20+ years, I've become very familiar with very fast and simple database systems like the "freespace" files we use on the Unisys mainframe platform.
We don't need relations for real-time processing. Most programs just need a place to keep data, and a simple key to retrieve that data. Some efficiency in disk usage is nice, but the primary design factor is performance.
A freespace file is a collection of pre-allocated fixed-length records of various sizes (e.g. 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, and 8192 bytes). Each record size is a assigned a type number (e.g., 1 through 6 in the above case), and a given file is created and pre-allocated with a mix of various records depending on the usage pater for that particular file. If you know all you need is tiny records, create a file containing a few hundred or thousand type 1 and maybe 2 records.
Records not allocated are filled with a deallocated fill pattern.
A program uses a record by performing a Write New operation. That tells the database manager to find a record in that file closest and >= to the size required, stick the presented buffer in the record, save it, and return a key to that record to the calling program. Typical key format is where Record Number is a number from 1... n. If your file has 1000 Type 3 records, it'd be from 1...1000 or 0...999.
To read a record, use a key from a previous Write New (stored away somewhere), perhaps in another file) to read that record from a file. Length is not required.
Programs use a very simple read-and-lock mechanism when modifying existing records. If one program has a record locked, another program must wait. Not a problem with intelligent coding.
We've used this system in airline systems for 40+ years. It works well. Sometimes an environment has robust commit and rollback/recovery features to allow for an entire series of changes to be rolled back on error, sometimes not. It doesn't seem to matter that much, especially for transient data like weather, flight schedule data, etc.
I would LOVE to see a freespace database ported to Solaris, personally. We'd use it heavily.:-)
Having worked on many diverse systems over the past few decades (CDC Cybers, VAXen running VMS, UNIVAC OS1100 mainframes, IBM boxes running TSO, Solaris/HPUX/Linux boxes, etc.), I would say that AIX, Linux, and Solaris are at least within the same basic OS family.
Some experience with the basic command set, shells, and syntax on one of those would be *far* more helpful and relevant than experience on an OS2200 or VMS environment, which would have almost nothing in common with UNIX-like systems.
All he's saying is that he's self-taught, and that what he knows about Linux and other UNIX-like operating systems was largely obtained through self-study and hands-on experience outside of the workplace.
I see this as admirable. It shows proactive interest and initiative, something I wish I could see a lot more of in the IT departments I've worked in. In any case, even general experience with things like sh or csh shell-scripting, vi, awk, etc. will transcend specific environments (for the most part).
There I was... Almost ready to make my winning move... All I needed was that one last harvester full of Tiberium, and my attack could be launched... But wait! Where did it go?? Ah, there it is, gazing longingly across that stupid river (again!) at the far-away Tiberium patch it can't possibly reach (again!).
Rocks had a higher IQ. Dancing fricking PAPERCLIPS had a higher IQ. Who wrote the Harvester AI, anyway??!?
I actually loved the fact that original C&C harvesters were so stupid you had to hold their hands and watch their every move. It made life more interesting, and sometimes provided entertainment as they walked into an enemy installation with no clue in the world about where they were.:-)
There are websites all over the internet that allow you to do country-by-IP-range lookups.
Yup, and those are the same stupid sites that consider me to be located in Switzerland when I'm actually sitting in Atlanta just because the corporate LAN I use part of the time happens to be registered there. That's really useful...
International companies are just that -- international. You can't always derive useful geographical information simply by doing an IP address lookup.
Artists deserve to make money on their work. We dislike the RIAA because they use questionable tactics and have a history of going after individual, largely non-technical defendants and suing them into the stone age.
And no, downloading music without paying for it is not STEALING. It's copyright infringement.
We don't want something for nothing. We want the right to purchase digital music ONCE with the ability to transform that single digital copy into any media or format we choose... and purely for personal use.
I've purchased several thousand dollars worth of music over the past 35 years, and I think I'm justified in making a few MP3 copies of the various music CD's I've legally purchased and the older LP's that I've legally taped, then legally converted to digital media.
No copyright infringement. You aren't copying it.:-)
1978 was a very good year. :-)
on
Jurassic Web
·
· Score: 1
In 1978 I was blowing people up in multiplayer COMBAT and engaging people in real-time conversation all over the state of Minnesota via MMT on MECC's MTS (MECC Timesharing System).
Palm doesn't own PalmOS anymore. They can choose not to use it on their devices, but that doesn't mean the operating system can't continue to develop and be sold to others (in the form of GVM or whatever).
I use mc because I can't be root on the servers at work (I don't play sysadmin, I write software), I don't have the space to create large tarballs, and things like nfs mounts are simply not permitted here between servers.
My limits likely wouldn't exist if I could be root here, but that isn't the case.
Your solutions are very good for the most part, just not for me.:-)
Heh. Yes, that's a good solution at times, but tar is useless when you don't have enough scratch space left to make that 20GB directory tree into a tarball in the first place.:-)
Not a bad solution for many problems, however. Just not large ones.
The function key line showing "10 QUIT" is displayed on the bottom left by default, the "eXit" option is displayed under the File menu in an obvious place at the bottom, and typing "exit" at the command line will exit the program just like it does any other shell.
With all due respect, what other information would you like? And how more obvious could it be?
I agree that quitting isn't obvious if you don't think of it as a shell AND if you don't have your F-keys properly defined in your xterm or ssh client, but that's a user issue, IMO.
Okay, now transfer this six-level-deep application directory tree to the new server using only the command-line ftp client found on a vanilla Solaris 8 server.:-) And no, the software isn't available in package form, the current config data on the host server is required.
Oh, and you have to have it up and running in 15 minutes because we have a customer that want to test it TODAY.
I'm gonna use mc, thank you!
Sometimes prefab building sections possess more utility than a pile of building blocks even though the pile is perhaps more "pure" in its approach to the problem. Welcome to the real world.:-)
The choice of tools that one has is rarely an either/or choice.
That's why I tend to write a mix of shell scripts, Perl, and C code depending on the task at hand.
It's also why I sometimes use Midnight Commander to perform tasks which I could also manage using other (and often simpler) tools. Sometimes file copies in the morning go astray -- I find this happens less often when I use a tool which explicitly shows me the destination. Sometimes using a color-coded editor is nice when my eyes are tired and I'm not on a server which has vim installed (that means most of them here).
It's Windows mentality to assume that there's only one solution for the job, it's not UNIX mentality.:-)
How do I use konqueror with frames and fish on my web tablet?:-)
MC is self-contained, relatively easy to install, fast, and works with vanilla ssh. Not sure what frames and fish are (links?), but I'd be surprised if they were as easy to install for a non-root user (as I often am).
What you say is also true. I think a lot depends on the history of an organization and on its internal structure.
After 9/11 at NWA, for example, many entire teams working on non-core or "noncritical" projects were axed, seemingly without regard for the extensive knowledge in other areas that some of the effected people had. I knew a few of those folks, and they had a lot more talent and experience than people I knew who were spared because they happened to be on the right team at the right time.
When I worked at Unisys, however, it was almost purely based on seniority. Things were so skin tight already at the airline center, though, that most of the fat has long since been trimmed away -- all that was left were a bunch of capable people, and within each team experience was the main legitimate differentiator, at least as far as I could see.
Any admin who judges current software based on dim memories of 10-year-old incarnations needs to get out a little and breathe the flowery air of fresh software development.:-)
Seriously. I've been using it on multiple platforms for close to a decade now, and I'm not sure I can remember it misplacing anything. Without my accidentally telling it to, anyway... These computer thingies are so literal.:-):-)
Slashdot Mirror
I think you'd be surprised at the complexity of the applications which use such file structures. :-) When the data itself is simple to store, the complexity of the application isn't really relevant.
Disk space on the mainframe isn't my problem as an applications programmer, and it doesn't seem to be an issue for a freespace file. Continuous space for each record is allocated up front when the file is created, and you are reading/writing data as fixed record sizes. A freespace record is ALWAYS a fixed multiple of a disk allocation size to prevent allocation issues. In our case, I don't actually know what the hardware does (and as an applications programmer that isn't a problem I care about), but logically we're taught that disk is allocated in 28-word sectors. On an OS2200 mainframe, freespace records are multiples of 112 words, and this is somehow tied to the way that operating system allocates data on disk.
What happens when you've pre-allocated, say, 1000 small blocks and 1000 large blocks, and it turns out you actually need 1001 large blocks?
Not an issue with a well-designed database. And historically it hasn't been. Keep in mind that I've been working with this file format almost continually since 1988, so I have plenty of experience seeing it in use in production transaction systems. For well-defined datasets, that isn't an issue.
Out of order writes are not an application issue. All the application cares about is that it can read and write data to a logical record. The underlying record handler has to worry about the specifics of getting that data to and from the disk.
I've never seen a program lose a key, but in our case the base freespace record management system has routines which run periodically against the database to find orphans, etc.
Something like scanning for a passenger name across all flights would be an interesting problem, but I wouldn't btoher to do that in freespace. I'd have a relational database off to the side and populated in parallel for that sort of query. Freespace is about *speed*, remember.
I don't think it would be hard to implement at all. It just takes time, something I can't spend (at work) writing code that isn't directly related to our paying customer needs. But I'm considering it for a home project. Probably in C. We'll see. :-)
Design issue. Optimally you use this sort of thing in places where you will always have an easily-available record number. It isn't a replacement for a relational database in places where relations are nice. :-)
In the airline context in which I work, you tend to see sorted indexes containing keys which are accessed by things like IATA station code (e.g., MSP or ATL), or airline code, or flight-date-origin (e.g., NW1492-24-MSP, which provide unique reference items which are easy to sort.
We also sometimes maintain a relational database for searching/reporting purposes on another lower-usage box just for reporting purposes. That frees the freespace file on the primary box to do it's thing quickly, and transactions against that fast database are also split off and inserted as rows against the relational database.
That allows for the speed of a freespace file in production while also giving reporting/query capabilities for past history ... and in a way which doesn't impact production.
We generally use a simple flat file as an index. Field 1 is a sorted index field (say a flight number), field 2 is the key to a freespace record. A simple binary search is fast even on a large sorted list.
No, it isn't useful for certain types of applications. Relational databases exist for a reason. :-) But if you have to store something for a well-known static set of fields (say weather stations or flights in an OAG schedule file), something a lot simpler isn't a bad method at all.
It wasn't gone. Just more finely (and widely) distributed. :-)
In my experience, most UNIX programmers tend to assume a relational database for almost everything if it isn't a vanilla flat file. That includes programmers in realtime applications, C people, Java people, etc. I can't begin to tell you how many applications I've seen written to use Oracle, Sybase, etc., just to store a simple static table of information. There's no POINT to that!
Most mainframe environments, on the other hand, have many established options, and relational is usually only considered if you actually need that type of functionality. Normally, systems are written with a mix of different database types. Could be flat files, could be freespace, could be RDMS, or could (in our case) be DMS, a network database with some types of set-linking properties but not really table-based.
Again in my experience, working in the airline industry, I've seen a bias towards RDMS for enterprise applications that I simply haven't seen on the mainframe (in my case Unisys transaction processing) side of life.
A web site is a tranctions processing facility. Just replace fancy 3270 or Uniscope screens with HTML. Same idea, forms, etc. It wants in and out, fast. Why use something not really made for that?
01-0127-0001 is the first type 1 record for file 127. 01-0100-0001 would be the first for file 100. That's what I get for doing patchwork re-editing an existing message before sending it...
Oops. Forgot that brackets get eaten. Typical record format is RECORDTYPE/FILENUMBER/RECORDNUMBER. The first Type 1 record for File 100 might look like 01-0127-0001 or whatever (specific binary representation in hex or octal would obviously vary depending on implementation and preference).
In our case, it's a 36-bit word shown as 12 octal digits, probably not a popular choice with UNIX folks. :-)
I've never understood the UNIX world's fascination with relational databases.
Speaking as a programmer in mainframe online transaction environments for the past 20+ years, I've become very familiar with very fast and simple database systems like the "freespace" files we use on the Unisys mainframe platform.
We don't need relations for real-time processing. Most programs just need a place to keep data, and a simple key to retrieve that data. Some efficiency in disk usage is nice, but the primary design factor is performance.
A freespace file is a collection of pre-allocated fixed-length records of various sizes (e.g. 256 bytes, 512 bytes, 1024 bytes, 2048 bytes, 4096 bytes, and 8192 bytes). Each record size is a assigned a type number (e.g., 1 through 6 in the above case), and a given file is created and pre-allocated with a mix of various records depending on the usage pater for that particular file. If you know all you need is tiny records, create a file containing a few hundred or thousand type 1 and maybe 2 records.
Records not allocated are filled with a deallocated fill pattern.
A program uses a record by performing a Write New operation. That tells the database manager to find a record in that file closest and >= to the size required, stick the presented buffer in the record, save it, and return a key to that record to the calling program. Typical key format is where Record Number is a number from 1 ... n. If your file has 1000 Type 3 records, it'd be from 1...1000 or 0...999.
To read a record, use a key from a previous Write New (stored away somewhere), perhaps in another file) to read that record from a file. Length is not required.
Programs use a very simple read-and-lock mechanism when modifying existing records. If one program has a record locked, another program must wait. Not a problem with intelligent coding.
We've used this system in airline systems for 40+ years. It works well. Sometimes an environment has robust commit and rollback/recovery features to allow for an entire series of changes to be rolled back on error, sometimes not. It doesn't seem to matter that much, especially for transient data like weather, flight schedule data, etc.
I would LOVE to see a freespace database ported to Solaris, personally. We'd use it heavily. :-)
Not sure I understand how this is puzzling.
Having worked on many diverse systems over the past few decades (CDC Cybers, VAXen running VMS, UNIVAC OS1100 mainframes, IBM boxes running TSO, Solaris/HPUX/Linux boxes, etc.), I would say that AIX, Linux, and Solaris are at least within the same basic OS family.
Some experience with the basic command set, shells, and syntax on one of those would be *far* more helpful and relevant than experience on an OS2200 or VMS environment, which would have almost nothing in common with UNIX-like systems.
All he's saying is that he's self-taught, and that what he knows about Linux and other UNIX-like operating systems was largely obtained through self-study and hands-on experience outside of the workplace.
I see this as admirable. It shows proactive interest and initiative, something I wish I could see a lot more of in the IT departments I've worked in. In any case, even general experience with things like sh or csh shell-scripting, vi, awk, etc. will transcend specific environments (for the most part).
There I was... Almost ready to make my winning move... All I needed was that one last harvester full of Tiberium, and my attack could be launched... But wait! Where did it go?? Ah, there it is, gazing longingly across that stupid river (again!) at the far-away Tiberium patch it can't possibly reach (again!).
Rocks had a higher IQ. Dancing fricking PAPERCLIPS had a higher IQ. Who wrote the Harvester AI, anyway??!?
I actually loved the fact that original C&C harvesters were so stupid you had to hold their hands and watch their every move. It made life more interesting, and sometimes provided entertainment as they walked into an enemy installation with no clue in the world about where they were. :-)
I rather liked Lotus WordPro, though. I used it back in the Ami Pro days... And Lotus 1-2-3 was not a bad spreadsheet.
Yup, and those are the same stupid sites that consider me to be located in Switzerland when I'm actually sitting in Atlanta just because the corporate LAN I use part of the time happens to be registered there. That's really useful...
International companies are just that -- international. You can't always derive useful geographical information simply by doing an IP address lookup.
Artists deserve to make money on their work. We dislike the RIAA because they use questionable tactics and have a history of going after individual, largely non-technical defendants and suing them into the stone age.
And no, downloading music without paying for it is not STEALING. It's copyright infringement.
We don't want something for nothing. We want the right to purchase digital music ONCE with the ability to transform that single digital copy into any media or format we choose ... and purely for personal use.
I've purchased several thousand dollars worth of music over the past 35 years, and I think I'm justified in making a few MP3 copies of the various music CD's I've legally purchased and the older LP's that I've legally taped, then legally converted to digital media.
No copyright infringement. You aren't copying it. :-)
In 1978 I was blowing people up in multiplayer COMBAT and engaging people in real-time conversation all over the state of Minnesota via MMT on MECC's MTS (MECC Timesharing System).
Palm doesn't own PalmOS anymore. They can choose not to use it on their devices, but that doesn't mean the operating system can't continue to develop and be sold to others (in the form of GVM or whatever).
Hey! Stop trying to make viable recommendations for my artificially contrived hypothetical situation here... :-)
Besides, in all seriousness:
> rsynch
-bash: rsynch: command not found
Oops. :-(
Next!
I use mc because I can't be root on the servers at work (I don't play sysadmin, I write software), I don't have the space to create large tarballs, and things like nfs mounts are simply not permitted here between servers.
My limits likely wouldn't exist if I could be root here, but that isn't the case.
Your solutions are very good for the most part, just not for me. :-)
Heh. Yes, that's a good solution at times, but tar is useless when you don't have enough scratch space left to make that 20GB directory tree into a tarball in the first place. :-)
Not a bad solution for many problems, however. Just not large ones.
The function key line showing "10 QUIT" is displayed on the bottom left by default, the "eXit" option is displayed under the File menu in an obvious place at the bottom, and typing "exit" at the command line will exit the program just like it does any other shell.
With all due respect, what other information would you like? And how more obvious could it be?
I agree that quitting isn't obvious if you don't think of it as a shell AND if you don't have your F-keys properly defined in your xterm or ssh client, but that's a user issue, IMO.
Equating the mc UI with the vi UI is ... silly.
Okay, now transfer this six-level-deep application directory tree to the new server using only the command-line ftp client found on a vanilla Solaris 8 server. :-) And no, the software isn't available in package form, the current config data on the host server is required.
Oh, and you have to have it up and running in 15 minutes because we have a customer that want to test it TODAY.
I'm gonna use mc, thank you!
Sometimes prefab building sections possess more utility than a pile of building blocks even though the pile is perhaps more "pure" in its approach to the problem. Welcome to the real world. :-)
The choice of tools that one has is rarely an either/or choice.
That's why I tend to write a mix of shell scripts, Perl, and C code depending on the task at hand.
It's also why I sometimes use Midnight Commander to perform tasks which I could also manage using other (and often simpler) tools. Sometimes file copies in the morning go astray -- I find this happens less often when I use a tool which explicitly shows me the destination. Sometimes using a color-coded editor is nice when my eyes are tired and I'm not on a server which has vim installed (that means most of them here).
It's Windows mentality to assume that there's only one solution for the job, it's not UNIX mentality. :-)
How do I use konqueror with frames and fish on my web tablet? :-)
MC is self-contained, relatively easy to install, fast, and works with vanilla ssh. Not sure what frames and fish are (links?), but I'd be surprised if they were as easy to install for a non-root user (as I often am).
What you say is also true. I think a lot depends on the history of an organization and on its internal structure.
After 9/11 at NWA, for example, many entire teams working on non-core or "noncritical" projects were axed, seemingly without regard for the extensive knowledge in other areas that some of the effected people had. I knew a few of those folks, and they had a lot more talent and experience than people I knew who were spared because they happened to be on the right team at the right time.
When I worked at Unisys, however, it was almost purely based on seniority. Things were so skin tight already at the airline center, though, that most of the fat has long since been trimmed away -- all that was left were a bunch of capable people, and within each team experience was the main legitimate differentiator, at least as far as I could see.
Any admin who judges current software based on dim memories of 10-year-old incarnations needs to get out a little and breathe the flowery air of fresh software development. :-)
Seriously. I've been using it on multiple platforms for close to a decade now, and I'm not sure I can remember it misplacing anything. Without my accidentally telling it to, anyway... These computer thingies are so literal. :-) :-)