I have to say, I'm not one of those who think that making programming easier is making programming worse.
I've been programming for over 30 years (first experience being with hex keypads, teletype terminals and batch processing systems back in the 70s!). I love refactoring support, debugging, in-line help, static analysis, code navigation and folding, documentation generation, etc. They make me much, much, more productive. Anyone who thinks using a text editor, command line or punch-cards is superior is welcome to them, but in my opinion they're crazy!
There may be people for whom all this handholding allows them to write poor software, when they couldn't have done it at all without that level of support. It may also be the case that it allows unmotivated developers to plateau in their abilities too early - although if they're that unmotivated I'm not sure how good they would ever have been.
I'm kind of with you on frameworks though, particularly in Java land. I tend to find that frameworks get me 70% of the way with about 30% of the effort. Win! The next 20% takes me 60% of the effort, just breaking even as I struggle to make it do what I need. Wrestling with incomplete documentation, lesser used functionality and mysterious error messages for which a single plaintive post can be found in the blogosphere somewhere, to which there are no replies;)
I often wish I'd just ignored the entire framework and built the damn thing myself. But maybe I'm not building typical systems, or something. I guess if you're doing bread-and-butter work where lots of other people are doing essentially the same thing, they may work out better.
There seems to be some truth to this, in my own experience. I find the solutions my younger colleagues invent are just too complicated and gnarly. They haven't yet found how to see the underlying simplicity in the problem and solution - and more importantly, they don't even understand that they should be doing that.
Mentoring is very satisfying, particularly when someone has a "got-it" moment, and their code improves forever thereafter. But I find that is rare. Many people I've worked with - even really, really bright people - just aren't interested in seeing a bigger picture. In fact, I'd go further. Most people will never do this - they will just solve the problem immediately in front of them, without any regard for how the whole thing hangs together, or the semantics of their construction, or the future ease of maintenance of their code.
I guess what I'm saying is that I'm not sure it's really about inexperience, or hardness of career. It's the difference between being a journeyman or a master, and very few it seems have a genuine desire of mastery in what they do.
I haven't read the book, but I studied cryptography under Professor Keith Martin at RHUL. He was never anything but encouraging of my attempts to design cryptographic protocols. On one occasion I was trying to invent a new symmetric key exchange protocol, reducing the trust required in the trusted third party. He gave me some good pointers, but did observe that the protocol required in the assignment was, by definition, supposed to be a *trusted* third party protocol. Nevertheless, he allowed me to work some of the ideas out a bit more. It was a lot of fun (but a terrible protocol!).
Anyway, I must get a copy of this book. It it's anything at all like his teaching it will be money well spent.
I'm not sure most people do swap distros much after they've found something that works for them.
When I was switching away from Windows about 10 years ago, I tried out a few: Mandriva, Suse, and Gentoo for a while. They all had good points and bad points. Gentoo was fun seeing the system built all around me to my specification, but all that compiling and fiddling became irritating after a while, and I just wanted something that got out of my way. I found Ubuntu, and I've basically stuck with it ever since. I almost switched away from Ubuntu when they started the whole Unity thing, so I delayed upgraded to later versions - but Unity in 12.04 is actually pretty good. Not perfect, but quite usable.
The thing about switching distros is it's like switching to a new version of Windows. It's still basically the same system underneath, so you're not relearning everything, but there's a bunch of different tools and things have moved around. It takes time and energy, so you don't really do it unless you're really curious, or you become uncomfortable with your current environment.
Sorry, I still don't buy it. Most of the people I work with are highly skilled at what they do. Their spreadsheets look fine, and they don't seem to be particularly intimidated.
Logic, in the sense of computer science, isn't really the metric to judge their abilities by - I couldn't do what they do. Or at least, not without a lot of training - and that's something I don't have time for, just like they don't have time to learn in detail what I do.
I'm also not sure why programmers would want their co-workers to specially empathise with them. It just displays a complete lack of awareness of the complexity and challenge in what everyone else does. I started out wanting to say it was an arrogant attitude, but I don't think it's that - it's just lack of awareness.
Now, having been a complete curmudgeon (sorry), I do think that it's good for everyone to learn a bit about what everyone else does. By all means teach the head of sales about lambda calculus, and expect to learn a bit about pipeline development and revenue management strategies. Win-win, as they say:)
I really don't think it's necessary for everyone in a tech company to understand computers in that way.
I work for a small software company. Our head of sales is an awesome guy, really switched on, understands the products, the market, people, what they want, sales process improvements - the works. I can't see how his ability to do his job would be enhanced by learning computer science, unless he was particularly interested in it anyway.
Thanks for your interesting replies. It's not an area I know much about.
I had not considered the requirement to ensure that a patient is actually taking the medicine. I suppose you could indirect it a step further, with people who provide assurance whether a pill is taken (or not), but who have no direct contact with the researchers. Of course, they may still introduce bias into the experiment. I suppose you could use a machine which automatically monitors whether the medicine is taken or not somehow, whose results are not known until the end of the experiment.
I have no doubt that such experiments would not pass ethical muster, or would be against the law in some countries. Still, these protocols were designed to reliably test for effects in active compounds. I still wonder how (setting aside ethical or legal constraints) you could construct reasonably reliable trials which test for clinical effects in placebos.
I mean the second - giving a control group nothing. The first point is well made. In fact, I read somewhere that no doctor or drug ever healed anybody - ultimately it is always the body which heals itself; active drugs only give it a fighting chance to perform its own healing (e.g. by destroying pathogens, or promoting a response to it, or something).
But returning to the second point, I agree you couldn't have double-blind trials as the patient would know whether they got nothing or not. You could still have a blind trial where the researchers did not know who was in the control group or not for the duration of the experiment.
In terms of ethics, if the goal of the trial is to determine whether a placebo has a clinical effect or not, then you are effectively testing a potentially workable treatment (the placebo, even though not pharcologically active) against something known to have no direct effect itself (nothing). So you can still tell patients that there is a 50% chance of getting a treatment that may have a positive effect (but not containing an active ingredient). The other 50% seem to have no hope at all, and they would know it (so you may even invoke the nocebo effect).
It does seem problematic. But if you are interested in whether placebos have clinical effect, how else could it be tested?
It is my understanding that this is still unproven. While they certainly seem to be most effective in treating perceived symtoms or pain, I believe that it is still an open question as to whether they might have any other clinically useful effects.
Can you provide some references to back up your strongly made assertion that they have no other clinical effects?
It's very true that there are often a lot of files with near duplicate content. Detecting near duplicates is much, much harder and will be probably orders of magnitudes slower to do, even if you can figure out how to do it in the first place.
However, there are also often a lot of files with exactly duplicate content. A government agency I worked with figured out they had over 30% identical duplication of files across their file stores. This was a signficant cost for them.
So, while your initial observation has some truth, your conclusion to "forget it" is false. I'm reminded of my old boss, who always used to say "Don't let the perfect be the enemy of the good".
There is no reason to use a crypto-strength hash. This will simply be slower. MD5 should be perfectly fine - it outputs a 128 bit hash, which is more than enough to avoid accidental collisions, and it's fast. You could match on the size as well as the hash, if you really really think you might have a hash match on different content, but it's probably not necessary.
It is true that if you're trying to avoid *intentionally malicious* collisions, you should never use MD5 as it's badly broken for that use - but not for detecting duplicate content. You're correct to avoid using CRC - but that's not a hash algorithm, it's a checksum algorithm. Accidental collisions with that algorithm will be very frequent.
The names of files should never be used to distinguish them. Files are often renamed by applications or during normal work by users. In any case, if you already have a hash match, then why do you care if the names are different? The content is already overwhelmingly likely to be identical. If you're really paranoid, then do a byte comparision of those files.
There is a digital preservation tool called DROID (Digital Record Object Identification) which scans all the files you ask it to, identifying their file type. It can also optionally generate an MD5 hash of each file it scans. It's available for download from sourceforge (BSD license, requires Java 6, update 10 or higher).
It has a fairly nice GUI (for Java, anyway!), and a command line if you prefer scripting your scan. Once you have scanned all your files (with MD5 hash), export the results into a CSV file. If you like, you can first also define filters to exclude files you're not interested in (e.g. small files could be filtered out). Then import the CSV file into your data anlaysis app or database of your choice, and look for duplicate MD5 hashes. Alternetively, DROID actually stores its results in an Apache Derby database, so you could just connect directly to that rather than export to CSV, if you have a tool that an work with Derby.
One of the nice things about DROID when working over large datasets is you can save the progress at any time, and resume scanning later on. It was built to scan very large government datastores (multiple Tb). It has been tested over several million files (this can take a week or two to process, but as I say, you can pause at any time, save or restore, although only from the GUI, not the command line).
Disclaimer: I was responsible for the DROID 4, 5 and 6 projects while working at the UK National Archives. They are about to release an update to it (6.1 I think), but it's not available just yet.
I have to echo your experience here. I really disliked Unity in the earlier incarnations, and kept my main machine on 10.10 until support ran out. Eventually I needed to do a full system re-install due to replacing a hard drive, and decided to give 12.04 a go. Despite all the Unity hate, Ubuntu has been good to me for many years, so I gave it a determined go.
Long story short - I like it. It gets out of my way. It avoids unnecessary chrome. It works.
It took me about 2 weeks of using it to realise I really quite liked it, contrary to my expectations. Again echoing the parent post, it was often the things people were complaining about the most that I ended up appreciating the most.
I am humbled to realise that my prior bitching about Unity was mostly unfounded (at least as it is incarnated in 12.04). And that I am far more change-resistant than I previously believed.
By peter watts is up there. Dysfunctional crew and highly claustrophobic atmosphere in a first contact story that's original and terrifying. Especially the ending, which is a profoundly depressing view of life in the universe.
And the gap series by Stephen donaldson is hard to beat for relentless abuse of and by just about every character on every page of the series....
Agile doesn't pamper developers, and it doesn't codify chaos. It's about getting a good quality product built in the face of uncertain requirements and change. In that sort of environment, treating change as an enemy to be resisted is a sure way to build something that no-one really wants or likes, but fulfils the letter of some requirements document agreed way back when. It's not a panacea, but it can work very well. You still need good people.
It makes partners of the stakeholders. It's very lightweight and done well helps to create a cooperative and respectful environment to work in. It's not appropriate for everything. It doesn't solve all development problems. It has very little to say about actual project management, or how to interface itself with project management. It doesn't really say much about how you get your requirements in the first place.
It's a huge and arrogant mistake to think that stakeholders aren't talented people (well, they may or may not be, like anyone else). It is true that they almost always aren't talented at software development, and often not great abstract thinkers. From their perspective, you are not talented at retail, or industrial processes, or finance, or... well, whatever they actually know about and you don't.
I'm actually quite appalled at your derogatory attitude to your customers and development partners. Were you just trying to make a point, or do you really believe what you said?
Well, just because your development process is working fine doesn't mean Agile is a waste of time.
I've been responsible for scrum / scrumban implemented in two separate and very different organisations over the last 5 years. In each case it's worked very well, but in different ways. In both cases, we adapted it to fit in and provide what each organisation needs. This requires intelligence and an understanding of what the organisation needs and what Agile can (and cannot) provide. It also requires buy-in to experiment and find what works and what doesn't.
It's also not micro-management; quite the opposite. It's about empowering teams to own their work, and to develop a new product in the face of changing requirements. I've seen shy developers suddenly start having great ideas about how to improve the development process, and gaining confidence and stature as they see their ideas implemented.
It also involves the stakeholders more directly in the development process. I've seen stakeholders move from cynical and disengaged to becoming a real development partner (the short iterations and product reviews are wonderful for that). Many people simply can't visualise what they really want from a product until they see something. They are not abstract thinkers. When they can see something, it's amazing what fantastic feedback and ideas you get from the people you're building the product for. It's even better when the same people see the changes they requested appearing a short time later in the product. This empowers the client as much as the development team. Done right, it's amazingly helpful.
I did have a fight with some of the more traditional project managers, who saw getting ongoing good feedback from the customer as a failure of our requirements elicitation stage. But for them, ongoing change is a sign of failure in an earlier stage - very much waterfall thinking.
Anyway, that only scratches the surface. I think Agile has a lot to offer, as long as you understand it's not a magic bullet. You still need good people (and it may be true that good people could use any methodology and produce good results). It does encourage a team spirit and a partnership attitude done right, which I haven't got from any other development or project management methodology.
Genuine question: how do you do development - any good ideas you could share?
I think that you're absolutely correct in that mathematics requires some kind of actual grasp of the relationships being expressed by the maths.
For you it had to be something you could relate to a real world problem. I'm almost the opposite - I have to get something distilled down to pure abstract relationships before I feel I really get it. But either way, you have to do some work and understand what is being expressed by the maths.
I studied cryptography a while ago, and I had some difficulty working through the maths. I got really stuck at one point in the course material and had to ask for help. The professor responded that they'd fudged that part of the maths a bit to avoid confusing people, and they didn't think anyone would get that far into the maths. He also gave a perfectly clear explanation of the source of my confusion - providing the missing bit of information about the relationships involved in the math.
I was left amazed at the realisation that most people pass this stuff without actually understanding it in any real way.
It's specifically designed to first probabilistically identify files, then attempt to verify their format.
Disclaimer: I haven't worked on it directly, but I did spend a number in the digital preservation space, so I probably know some of the people who have contributed to it.
I've wrestled with whether to release under a GPL or BSD style license for two projects. I can see the pros and cons of both, but in the end chose BSD.
The code is of niche appeal, and I'd be happy for anyone to get some use out of it, whether in a proprietary product or anywhere else it's useful. It probably won't have a vast lifetime or attract a large community around it. So the need to preserve the openness of the code over long periods of time just isn't there.
In the shorter term, I'd rather have as many people as free as possible to do anything they like with it. If the code was more generally appealing, I might choose a different license.
Slashdot Mirror
I have to say, I'm not one of those who think that making programming easier is making programming worse.
I've been programming for over 30 years (first experience being with hex keypads, teletype terminals and batch processing systems back in the 70s!). I love refactoring support, debugging, in-line help, static analysis, code navigation and folding, documentation generation, etc. They make me much, much, more productive. Anyone who thinks using a text editor, command line or punch-cards is superior is welcome to them, but in my opinion they're crazy!
There may be people for whom all this handholding allows them to write poor software, when they couldn't have done it at all without that level of support. It may also be the case that it allows unmotivated developers to plateau in their abilities too early - although if they're that unmotivated I'm not sure how good they would ever have been.
I'm kind of with you on frameworks though, particularly in Java land. I tend to find that frameworks get me 70% of the way with about 30% of the effort. Win! The next 20% takes me 60% of the effort, just breaking even as I struggle to make it do what I need. Wrestling with incomplete documentation, lesser used functionality and mysterious error messages for which a single plaintive post can be found in the blogosphere somewhere, to which there are no replies ;)
I often wish I'd just ignored the entire framework and built the damn thing myself. But maybe I'm not building typical systems, or something. I guess if you're doing bread-and-butter work where lots of other people are doing essentially the same thing, they may work out better.
There seems to be some truth to this, in my own experience. I find the solutions my younger colleagues invent are just too complicated and gnarly. They haven't yet found how to see the underlying simplicity in the problem and solution - and more importantly, they don't even understand that they should be doing that.
Mentoring is very satisfying, particularly when someone has a "got-it" moment, and their code improves forever thereafter. But I find that is rare. Many people I've worked with - even really, really bright people - just aren't interested in seeing a bigger picture. In fact, I'd go further. Most people will never do this - they will just solve the problem immediately in front of them, without any regard for how the whole thing hangs together, or the semantics of their construction, or the future ease of maintenance of their code.
I guess what I'm saying is that I'm not sure it's really about inexperience, or hardness of career. It's the difference between being a journeyman or a master, and very few it seems have a genuine desire of mastery in what they do.
You are talking about sousveillance, rather than surveillance:
http://en.wikipedia.org/wiki/Sousveillance
I haven't read the book, but I studied cryptography under Professor Keith Martin at RHUL. He was never anything but encouraging of my attempts to design cryptographic protocols. On one occasion I was trying to invent a new symmetric key exchange protocol, reducing the trust required in the trusted third party. He gave me some good pointers, but did observe that the protocol required in the assignment was, by definition, supposed to be a *trusted* third party protocol. Nevertheless, he allowed me to work some of the ideas out a bit more. It was a lot of fun (but a terrible protocol!).
Anyway, I must get a copy of this book. It it's anything at all like his teaching it will be money well spent.
I meant to reply to the parent, obviously :)
I'm not sure most people do swap distros much after they've found something that works for them.
When I was switching away from Windows about 10 years ago, I tried out a few: Mandriva, Suse, and Gentoo for a while. They all had good points and bad points. Gentoo was fun seeing the system built all around me to my specification, but all that compiling and fiddling became irritating after a while, and I just wanted something that got out of my way. I found Ubuntu, and I've basically stuck with it ever since. I almost switched away from Ubuntu when they started the whole Unity thing, so I delayed upgraded to later versions - but Unity in 12.04 is actually pretty good. Not perfect, but quite usable.
The thing about switching distros is it's like switching to a new version of Windows. It's still basically the same system underneath, so you're not relearning everything, but there's a bunch of different tools and things have moved around. It takes time and energy, so you don't really do it unless you're really curious, or you become uncomfortable with your current environment.
Sorry, I still don't buy it. Most of the people I work with are highly skilled at what they do. Their spreadsheets look fine, and they don't seem to be particularly intimidated.
Logic, in the sense of computer science, isn't really the metric to judge their abilities by - I couldn't do what they do. Or at least, not without a lot of training - and that's something I don't have time for, just like they don't have time to learn in detail what I do.
I'm also not sure why programmers would want their co-workers to specially empathise with them. It just displays a complete lack of awareness of the complexity and challenge in what everyone else does. I started out wanting to say it was an arrogant attitude, but I don't think it's that - it's just lack of awareness.
Now, having been a complete curmudgeon (sorry), I do think that it's good for everyone to learn a bit about what everyone else does. By all means teach the head of sales about lambda calculus, and expect to learn a bit about pipeline development and revenue management strategies. Win-win, as they say :)
I really don't think it's necessary for everyone in a tech company to understand computers in that way.
I work for a small software company. Our head of sales is an awesome guy, really switched on, understands the products, the market, people, what they want, sales process improvements - the works. I can't see how his ability to do his job would be enhanced by learning computer science, unless he was particularly interested in it anyway.
Thanks for your interesting replies. It's not an area I know much about.
I had not considered the requirement to ensure that a patient is actually taking the medicine. I suppose you could indirect it a step further, with people who provide assurance whether a pill is taken (or not), but who have no direct contact with the researchers. Of course, they may still introduce bias into the experiment. I suppose you could use a machine which automatically monitors whether the medicine is taken or not somehow, whose results are not known until the end of the experiment.
I have no doubt that such experiments would not pass ethical muster, or would be against the law in some countries. Still, these protocols were designed to reliably test for effects in active compounds. I still wonder how (setting aside ethical or legal constraints) you could construct reasonably reliable trials which test for clinical effects in placebos.
I mean the second - giving a control group nothing. The first point is well made. In fact, I read somewhere that no doctor or drug ever healed anybody - ultimately it is always the body which heals itself; active drugs only give it a fighting chance to perform its own healing (e.g. by destroying pathogens, or promoting a response to it, or something).
But returning to the second point, I agree you couldn't have double-blind trials as the patient would know whether they got nothing or not. You could still have a blind trial where the researchers did not know who was in the control group or not for the duration of the experiment.
In terms of ethics, if the goal of the trial is to determine whether a placebo has a clinical effect or not, then you are effectively testing a potentially workable treatment (the placebo, even though not pharcologically active) against something known to have no direct effect itself (nothing). So you can still tell patients that there is a 50% chance of getting a treatment that may have a positive effect (but not containing an active ingredient). The other 50% seem to have no hope at all, and they would know it (so you may even invoke the nocebo effect).
It does seem problematic. But if you are interested in whether placebos have clinical effect, how else could it be tested?
It is my understanding that this is still unproven. While they certainly seem to be most effective in treating perceived symtoms or pain, I believe that it is still an open question as to whether they might have any other clinically useful effects.
Can you provide some references to back up your strongly made assertion that they have no other clinical effects?
Would nothing work?
It's very true that there are often a lot of files with near duplicate content. Detecting near duplicates is much, much harder and will be probably orders of magnitudes slower to do, even if you can figure out how to do it in the first place.
However, there are also often a lot of files with exactly duplicate content. A government agency I worked with figured out they had over 30% identical duplication of files across their file stores. This was a signficant cost for them.
So, while your initial observation has some truth, your conclusion to "forget it" is false. I'm reminded of my old boss, who always used to say "Don't let the perfect be the enemy of the good".
There is no reason to use a crypto-strength hash. This will simply be slower. MD5 should be perfectly fine - it outputs a 128 bit hash, which is more than enough to avoid accidental collisions, and it's fast. You could match on the size as well as the hash, if you really really think you might have a hash match on different content, but it's probably not necessary.
It is true that if you're trying to avoid *intentionally malicious* collisions, you should never use MD5 as it's badly broken for that use - but not for detecting duplicate content. You're correct to avoid using CRC - but that's not a hash algorithm, it's a checksum algorithm. Accidental collisions with that algorithm will be very frequent.
The names of files should never be used to distinguish them. Files are often renamed by applications or during normal work by users. In any case, if you already have a hash match, then why do you care if the names are different? The content is already overwhelmingly likely to be identical. If you're really paranoid, then do a byte comparision of those files.
There is a digital preservation tool called DROID (Digital Record Object Identification) which scans all the files you ask it to, identifying their file type. It can also optionally generate an MD5 hash of each file it scans. It's available for download from sourceforge (BSD license, requires Java 6, update 10 or higher).
http://sourceforge.net/projects/droid/
It has a fairly nice GUI (for Java, anyway!), and a command line if you prefer scripting your scan. Once you have scanned all your files (with MD5 hash), export the results into a CSV file. If you like, you can first also define filters to exclude files you're not interested in (e.g. small files could be filtered out). Then import the CSV file into your data anlaysis app or database of your choice, and look for duplicate MD5 hashes. Alternetively, DROID actually stores its results in an Apache Derby database, so you could just connect directly to that rather than export to CSV, if you have a tool that an work with Derby.
One of the nice things about DROID when working over large datasets is you can save the progress at any time, and resume scanning later on. It was built to scan very large government datastores (multiple Tb). It has been tested over several million files (this can take a week or two to process, but as I say, you can pause at any time, save or restore, although only from the GUI, not the command line).
Disclaimer: I was responsible for the DROID 4, 5 and 6 projects while working at the UK National Archives. They are about to release an update to it (6.1 I think), but it's not available just yet.
I have to echo your experience here. I really disliked Unity in the earlier incarnations, and kept my main machine on 10.10 until support ran out. Eventually I needed to do a full system re-install due to replacing a hard drive, and decided to give 12.04 a go. Despite all the Unity hate, Ubuntu has been good to me for many years, so I gave it a determined go.
Long story short - I like it. It gets out of my way. It avoids unnecessary chrome. It works.
It took me about 2 weeks of using it to realise I really quite liked it, contrary to my expectations. Again echoing the parent post, it was often the things people were complaining about the most that I ended up appreciating the most.
I am humbled to realise that my prior bitching about Unity was mostly unfounded (at least as it is incarnated in 12.04). And that I am far more change-resistant than I previously believed.
By peter watts is up there. Dysfunctional crew and highly claustrophobic atmosphere in a first contact story that's original and terrifying. Especially the ending, which is a profoundly depressing view of life in the universe.
And the gap series by Stephen donaldson is hard to beat for relentless abuse of and by just about every character on every page of the series....
Agile doesn't pamper developers, and it doesn't codify chaos. It's about getting a good quality product built in the face of uncertain requirements and change. In that sort of environment, treating change as an enemy to be resisted is a sure way to build something that no-one really wants or likes, but fulfils the letter of some requirements document agreed way back when. It's not a panacea, but it can work very well. You still need good people.
It makes partners of the stakeholders. It's very lightweight and done well helps to create a cooperative and respectful environment to work in. It's not appropriate for everything. It doesn't solve all development problems. It has very little to say about actual project management, or how to interface itself with project management. It doesn't really say much about how you get your requirements in the first place.
It's a huge and arrogant mistake to think that stakeholders aren't talented people (well, they may or may not be, like anyone else). It is true that they almost always aren't talented at software development, and often not great abstract thinkers. From their perspective, you are not talented at retail, or industrial processes, or finance, or... well, whatever they actually know about and you don't.
I'm actually quite appalled at your derogatory attitude to your customers and development partners. Were you just trying to make a point, or do you really believe what you said?
Well, just because your development process is working fine doesn't mean Agile is a waste of time.
I've been responsible for scrum / scrumban implemented in two separate and very different organisations over the last 5 years. In each case it's worked very well, but in different ways. In both cases, we adapted it to fit in and provide what each organisation needs. This requires intelligence and an understanding of what the organisation needs and what Agile can (and cannot) provide. It also requires buy-in to experiment and find what works and what doesn't.
It's also not micro-management; quite the opposite. It's about empowering teams to own their work, and to develop a new product in the face of changing requirements. I've seen shy developers suddenly start having great ideas about how to improve the development process, and gaining confidence and stature as they see their ideas implemented.
It also involves the stakeholders more directly in the development process. I've seen stakeholders move from cynical and disengaged to becoming a real development partner (the short iterations and product reviews are wonderful for that). Many people simply can't visualise what they really want from a product until they see something. They are not abstract thinkers. When they can see something, it's amazing what fantastic feedback and ideas you get from the people you're building the product for. It's even better when the same people see the changes they requested appearing a short time later in the product. This empowers the client as much as the development team. Done right, it's amazingly helpful.
I did have a fight with some of the more traditional project managers, who saw getting ongoing good feedback from the customer as a failure of our requirements elicitation stage. But for them, ongoing change is a sign of failure in an earlier stage - very much waterfall thinking.
Anyway, that only scratches the surface. I think Agile has a lot to offer, as long as you understand it's not a magic bullet. You still need good people (and it may be true that good people could use any methodology and produce good results). It does encourage a team spirit and a partnership attitude done right, which I haven't got from any other development or project management methodology.
Genuine question: how do you do development - any good ideas you could share?
I think that you're absolutely correct in that mathematics requires some kind of actual grasp of the relationships being expressed by the maths.
For you it had to be something you could relate to a real world problem. I'm almost the opposite - I have to get something distilled down to pure abstract relationships before I feel I really get it. But either way, you have to do some work and understand what is being expressed by the maths.
I studied cryptography a while ago, and I had some difficulty working through the maths. I got really stuck at one point in the course material and had to ask for help. The professor responded that they'd fudged that part of the maths a bit to avoid confusing people, and they didn't think anyone would get that far into the maths. He also gave a perfectly clear explanation of the source of my confusion - providing the missing bit of information about the relationships involved in the math.
I was left amazed at the realisation that most people pass this stuff without actually understanding it in any real way.
Actually there is a tool that does all of that already: JHOVE - JSTOR/Harvard Object Validation Environment.
http://hul.harvard.edu/jhove/
It's used in the digital preservation field, for example in an archive to try to figure out what they've got and what state it's in.
The JSTOR/Harvard Object Validation Environment:
http://hul.harvard.edu/jhove/
It's specifically designed to first probabilistically identify files, then attempt to verify their format.
Disclaimer: I haven't worked on it directly, but I did spend a number in the digital preservation space, so I probably know some of the people who have contributed to it.
I've wrestled with whether to release under a GPL or BSD style license for two projects. I can see the pros and cons of both, but in the end chose BSD.
The code is of niche appeal, and I'd be happy for anyone to get some use out of it, whether in a proprietary product or anywhere else it's useful. It probably won't have a vast lifetime or attract a large community around it. So the need to preserve the openness of the code over long periods of time just isn't there.
In the shorter term, I'd rather have as many people as free as possible to do anything they like with it. If the code was more generally appealing, I might choose a different license.
A being a tool that is used for B does not imply that B goes away if A does.
It might imply there is less chance of B if A goes away. Or maybe not - we're good at inventing tools.