Looking At Pretty Graphics Of Dot Com Demographics
I'm unemployed and bored.
With all the talk about dotcom booms and people moving away and all these really general statements people are making; i was curious as to what's really going on now, what went on four years ago and what effect did the boom really have on this community.
So.. i wrote a script that went through the craigslist archive on egroups and tallied the number of postings each day for each category then i stuffed it in a database and made a grapher tool.
It's kinda neat cos you can basically see some somwhat hard data on what the san francisco bay area has been doing over the past four years. you can graph job postings from various industries against things like apartments for rent or housing wanted postings, also for sale postings and resumes.
Some of the interesting things I found is that the number of housing wanted postings seems to be slightly down recently but pretty much unaffected by the drop in jobs. perhaps people are always in a state of wanting to move to a city.
One thing to bear in mind, this data isn't one hundred percent accurate for a few reasons... people do repost their information multiple times in a given month, sometimes people post a number of avaliable apartments in a single posting and craigslist has become significantly more popular over the years in question. however, I do still think that it makes for a decent general indicator of trends in our community. (I'm considering doing some kind of normalizing based on the total volume of posts- I need to think about it some more)
(Unfortunately, personal listings aren't archived. I guess that's a good thing... but I guess it also would have been cool to graph activity on the personals categories against some of the categories that reflect the general state of the economy. (maybe when people aren't tied up in career they start to think about more important things, or maybe not...) )
Either way, the data is there, you can look at it for different time periods and categories. draw your own conclusions and have fun!
The url is here:
http://www.signal11.com/charts/chart-o-m atic.cgi
take it easy..
--adam "
Not just Silicon Valley is being affected by the job crunch - there are a lot of GOOD people out of work in the Los Angeles area too. Not FrontPage monkeys either, real designers and coders.
Now, enough praise...
There seems to be some data missing - most notably the "people" data, which would have been interesting to plot the migration of people to and from the area as opposed to housing availability, jobs availability, etc. The other thing is that the server sometimes seems to return an error for some reason or another, although this may be due to the
But other than that, a good attempt, and certainly some good ideas there.
Smegma.
A lot of people have been laid off - a few of them are talented hackers. But really, the vast majority of dotcom casualties were overpaid frontpage monkeys who will never get a real job outside of bagging groceries or serving fries.
just b/c you lost your job does not mean that you are quite ready to pick up and leave immediately. Most people are going to try and find a job in the general vicinity *first*, then if they cannot find anything a relocation would be necessary. I would assume that is why there is little change in that department.
The guys who are MCSEs and FrontPage Monkeys will be out of work for a LONG time. It seems the trend here in NYC is towards the UNIX/Linux world (seriously). :))
Since there are over 300,000 MCSEs here, and the companies that want them are disappearing, they need a new skill set.
There was always a minority of *NIX people, and THEY are the ones who can still be picky about the pay/job they want.
Since the financial industry is a UNIX world, AIX and HP/UX guys are in HIGH demand.
At the job I got a month ago, they were looking for a good UNIX admin for 10 months before they hired me! That's why their infrastructure in in such a state of disarray.
They hired me on the spot and wanted me to start same day. I am not even the best UNIX admin out there. (I still can't get the UNIX printing system
The market is flooded with Java guys from India who will work for less (Nothing against the East Indians..I'm just stating fact)
I now lead a department. Out of 100 in the department, 97 are from India/Pakistan.
They had NO trouble finding a job.
They all have UNIX skills, DB2 or Oracle and they all have Siebel skills.
If you're not a Liberal in your 20's, then you have no heart.If you're still a Liberal in your 30's you have no brain.
Like I said, this is just jealously, bitterness.. but a social group which lacks the basics of humbleness, and toots its own horn shamelessly, becomes a prick in the side of this lowly helpdesk worker.
oh, hell. I'm posting anonymously.
To the A.C. who posted this great piece of prose: Come out of the Anonymity closet and take credit or give credit for this. It's worthy of saving ... if there's a credit and so on!
BTW, is there a tune to go with this?
I don't feel sad for every one of the dot-com unemployed. Many of them were dreamers, who never lived in the real world.
There are/were thousands of silly dot-coms, with stupid business plans and cobbled-together technology. Many of these shops were not part of any "talent pool".
I remember this one guy who I worked with. Marc was a nice guy. Marc thought of himself as a strong manager who was helping to fuel the dot-com revolution.
But the fact is that Marc didn't have a clue about management OR technology - he was just caught up in a ball of momentum. Marc was in a high-profile position in a very high profile organization. But that fact alone didn't make Marc a capable guy.
Marc left us for the dot coms of Atlanta. The last I heard, Marc was still looking for a job. Not because of the economy, but because he didn't have any skills.
That would be sung to the tune of 'Blame Canada' from South Park: Bigger, Longer and Uncut.
Vintage computer games and RPG books available. Email me if you're interested.
the slashdot effect is in effect. we underestimated the traffic it would generate and are moving the script to a bigger machine now.
if you can't get to it, please bear with us and try again in a few hours.
thanks again
--adam
This is more like a resume for an out-of-work perl-monkey. If he'd really like to do statistical analysis he should should take into account growth of list members, number of postings total, etc.
So what do we have here? Some fellow who just posted a "hey, look at me" resume onto slashdot; not much meat and a sorry excuse for a story.
Shame on you, slashdot...
Ralph Castain of Fort Collins, Colorado wrote the following, which began on the same page (12) as Google's ad, "Google Seeks Expert Computer Scientists"
--
There is no hatred more pure and true than that expressed by children.
After playing with the chart for a while and noticing the indication of an exodus from the bay area I began to get upset that all these "dot.com" people came to to bay area, jacked up the median rental and home prices, then bailed after things started going bad. Although rentals and home prices have started to level off they have, for the most part, not gone down. If they did that would accelerate our decent into resession. That sad thing is that's inevitable. Foreclosure rates have already gone up.
So now what I have to say to all the people who flocked here, threw high-test gasoline on the fires of the economy, burned everything to a crisp then took off, Thanks a whole lot from someone who was here a long time before you! I feel no pain for any of the dot.com loosers.
How about starting a head hunting service linked to the /. amount of people in the techincal field.
/. as a portal for answers. And OSDN could even charge a small subscription fee for all the new services.
Everyone bitches about issues, how about using
--
Computers are useless. They can only give you answers. Pablo Picasso (1881 - 1973)
It was an IT monkey position at a local shop called Powell's books http://www.powells.com - the fax machine to submit applications was sooo busy I couldn't get through.
When I called up to find out what was wrong with their machine (it would just roll over to another line most of the time) they said they were getting applications all over the State of Oregon and as far away as San Francisco all day long.
You have it backwards. It was during the boom when people were wasting their talent, creating useless software and websites, justified by insane business plans. These hard times are a correction. Their unemployment is a sign to people laid off from the boom that they have wasted their previous years, and they need to find something next that truly will be useful.
here's the URL
u rm om=1&from_month=0&from_year=98&c-art=on&to_month=0 &to_year=101&c-eng=on&c-peo=on&c-web=on&c-mar=on
http://charts.signal11.com/chart-o-matic.cgi?yo
Photos.
There's definitely a difference in the quality of the emails I get from software engineers these days. They're much more technically adept.
.coms neither had the interest nor the ability to solve real computer science problems and have now switched to non-technical, and for them, more interesting careers. Wired ran a story about how many armchair engineers from the 90's went back to school and followed their true passion to become actors, artists, and writers.
Seems as if most of the computer scientists working for
You might say the quality of software being written today is slightly up compared to the 90's because the only people programming are the ones who really want to do it.
Hi All,
Another unemplyed geek here. I wrote a set of perl scripts that graphed the # of Jobs available on computerjobs.com for several cities. It is on the main page of AtlantaGeek.com
Visit Savagenumber.com
You just stated above that you had no idea that this was being posted to Slashdot yet in this posting you state the above. Which comment is true?
HagabardLong time reader, first time poster.
Just got my WIRED magazine subscription a short time ago, and what do I see on the cover "Why Linux will lose the desktop war: page 134".
Now before you tell me this has nothing to do with the dotcom boom and subsequent bust, I think you would do well to note these quotes from the article.
If that doesn't relate to the dot-com bust, I don't know what does. He goes on to state the following on the same page:
Pretty damming commentary from someone who supports Linux, don't you think? Or at least he still does, but on the server platform where it has gained the most ground.
Comments?
I'm sure that this will take forever to become a proper thread unto itself, that is why I've bothered to post it here today.
Desiato_Hotblack
** By reading this post, you've agreed to my EULA - which includes not modding-down due to difference in opinion. **
Excuse me. But we're getting a little ridiculous about our titles.
First it was programmer, then it was programmer/analyst, then it was software developer, then it was software engineer, then it was architect...
... now its Computer Scientist?
Puhleeze... anyone who uses that term so loosely needs to read one of Knuth's books to understand what a computer scientist really is.
I could really make good use of those figures that show that an outsourced 1st level support position can cost $5k-$6k. I would be very grateful if you could post a citation on those numbers.
When information is power, privacy is freedom.
As astounding as this data is, it would be infinitely more useful (and accurate) if it were normalized againt actuall traffic usage on craigslist, so the growing/weaning popularity of the site wouldn't skew the demo data.
Anyone wanna get Craig's archived logfiles?
Kevin Fox
I asked the guy at the next machine, "If you had to give a date, when would you say the dot-com collapse happened?" He said, "November '99."
According to this, that's within two months of the point at which marketing people got more popular than engineers (with employers.)
Yes, fascinating.
The source(s) and limitations are stated on the site and any one knowing statistics knows aswell how to use this one.
Good luck on the job hunt!
"The likes of Facebook and WhatsApp are free to those whose privacy is of zero value."
Anyone else look at Adam's resume and notice that he worked at Xing and was part of both Q&A and their MPEG library development? Perhaps he had some small part in the ultimate release of DeCSS. Just a thought. ;P
If he GPL'd his code, numerous people could download and install his scripts, get things messed up, bombard him (and newsgroups) with clueless questions, and then resort to some prebuilt binaries (obviating the need for source in the first place).
Oh, wait... where have I seen this development model before?
Definitely, I would like to have the complete story about Signal11.
I thought his posts were often interesting. However, there was some hassle between him and Slashdot officers that I didn't understand.
- March '98 to August '01
- apartments avaliable
- housing wanted
- resumes
- for sale
You can see that these all tracked each other fairly well up to December '99, when resumes/for-sale went off on their own, and apartments/housing stayed fairly even for a while.Then, apartment availability started shooting up around August '00, kind of tracked resumes/for sale for a while, and then shot through resumes in February, picking up for-sale which has tracked it ever since.
I'll leave it as an excercize for the reader to correlate all this with external economic factors, but I think it's pretty clear that for a while people were selling stuff to make the rent while they were looking for work, and then between August and February people started selling stuff because they were leaving the area.
The real Webmaven is user ID 27463. I don't rate an imposter, because my ID is such a lame-ass high number.
Anyone else notice the "yourmom=1" that is inserted in the URL when you use this? If you set it to zero or remove it, the graph won't use your start/end dates...funny.
-Parts of Tool, "Ænema"
-
And the Angel said unto me, "These are the cries of the carrots! The cries of the carrots!"
My question is, we're just about ready to upgrade to a larger condo or house and I'm wondering if anyone has URL's for historical perspectives of the time lag between unemployment jumps, "for sale" jumps, and drops in prices...
Living in the valley and not being tempted by the big $ offers of startups has meant that my fiancee and I are still employed and ready to upgrade from the 1-bedroom condo to someplace to start the family life, now that the market should be more sane...
(Admittedly, we're starting to see $500k 3-bedroom places in the valley but they're few and not in the best neighborhoods...and we'd much rather pay $400k. Then again, my condo has appreciated ~3x since I bought it in '95 so I suspect there'll be a fixed delta between the sale price of it and the purchase price of the new place.)
Ok, we'll call it slashdotted...
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, adam@signal11.com and inform them of the time the error occurred, and anything you might have
done that may have caused the error.
More information about this error may be available in the server error log.
--- Hindsight is 20/20, but walking backwards is not the answer.
the guy whop put the up the CG/webpage claims that the graphics are his copyright
correct me if I am wrong but he is redistributeing the database in a differant format
he is not doing any analysis or changeing the data in any way but redisplaying it
so he has no copyright claim at all as far as I can see bit of a joke really
regards
john jones
... completely ripped off Red Hat. Some lame journalist works for 6 months to a year on a lame portal that never gets off the ground, and gets tens of thousands of options? I declined an engineering position at Red Hat, mostly because of the insanity of the offer: they offered 2,000 shares for a non-junior engineering position, and a pitiful salary. Knowing your in the same boat as everybody else, this is ok but when the truth comes out a year later, that idiots were making/getting far more only justifies my rejection of that pissy offer. I'm still insulted.
I got sick of seeing license plates like THX IPO on clementina when Red Hat still had their offices in SF. The web-design company that they acquired, Atomic Vision, was very puffed-up, full of self-flattery, and not especially good. For all I know, they bunch of them, headlined by Mattew Butterick, subsisted for a year on a constantly-shifting sand of improvised assignments and high-faluting job titles. I once went in to those offices and swear everybody was reading about soap operas on the web.
A quick gander at Butterick's exercise history on biz.yahoo.com shows that he did indeed cash out. What an ass.
I'm convinced it was a shell company to rip off shareholders. A lot of these people were savvy ex-Wired people accustomed to delusional investors.
Too bad Matthew S. CEO of Red Hat, bought into the hype. Wither, Bob Young?
while i don't have too much knowledge as to the effects of the dotcom boom and bust nationally and globally, i have formed a few opinions on what has happened locally.
while the boom sucked in a number of ways: computer jobs became glamourous and attracted a lot of goldrushers or hollywood ego types, which made working in these types of jobs a little obnoxious. the people who i hated in high school who were stuck up and going to fancy colleges found themselves getting rich in marketing and management positions, riding on the backs of the geeks who actually did the work. the talent pool was suddenly flooded with people who weren't really into what they were doing but they were doing it because that is where the money was. etc..
it also did a few cool things: it provided a way for slackers, artists, writers, etc to actually earn a living wage. (not exemplary salaries, but salaries that actually made it possible for them to live reasonably comfortably in the bay area) suddenly people that had been making peanuts could make a decent salary 30-50K and live comfortably knowing that they could pay their bills and some were gaining a skill that could turn into a career.
unfortunately, things got out of hand when the aforementioned fancy college educated folks had a little too much money to play around with and traders saw a wave of hype which they could ride to grow their assets. idiotic companies with pathetic business plans, idiot management and stupid ideas were recieving insane amounts of venture capital, which they were happy to spend the majority on things like office adornments, lame marketing and other things irrelevant to the core products they were building.
when this all was happening, the rest of the economy grew to support all these new businesses needing services and now that they're all gone, not only is the technology industry affected, but also the industries they relied upon and the industries those industries relied upon. hopefully the ripple won't be catastrophic.
personally, i was happy to see the bubble burst at first for the reasons i listed above. i was sick of hollywood ego internet types and this sudden 'coolness' that surrounded all things internet. i was also sick of seeing people who had no real interest in technology at all get filthy rich off of the hype.
however, now my opinion has changed a little. a lot of my friends are in fairly gnarly positions as a result of the layoffs. they weren't trying to get rich, and never were. they just wanted a honest living wage for a days work. they had the promise of a career and living wage for the years to come laid out before them and now it's all vanished into thin air.
in one way, i think it's a good thing, because the boom distracted people from what they really wanted to do with their lives. while doing operations for a dotcom may pay decent, it's not the most productive thing to do with one's eight hours a day in the grand scheme of things. (given that someone is not completely a geek at heart) now people are being forced to reevaluate themselves and their master plans for life, some are going back to college, some are persuing art, others are getting involved in vocations that they've been meaning to do for a while and i think that that is a very good thing.
i think that it is also a good thing for technology. now that there are less distractions and promise of riches for geeks working on bunk technologies, there will a decent talent pool of geeks looking to work for a decent wage on something that is technologically cool. that is also pretty exciting. while venture capitalists may be stopping the handouts, they're not going to stop completely and now they're not going to have nearly as many proposals to review. i'm hoping that in the next couple years we'll see a number of new companies that are doing stuff that is truly innovative.
Andreesen, perhaps?
We both worked out of his apartment for a number of months on a contract job together, so I should know.
Into programming, eh? XP? A little of the ol' buddy programming? swapping vars; very good, very good. Contract jobs...rim jobs?
wink, wink, nudge, nudge...
Say no more!
yep, good old signal 11, showing his intelligence. Check out the "hire me" link.. it's him.
...Anyone who just skimmed the above and has some mod points to spare, re-read the above post and mod it up.
More realistically, though, thinks UptownGuy...
You'll just have to settle for my high praise, anonymous coward, whoever you are. There is so much truth in what you wrote above... I actually took the time to read it and re-read it over 5 times. Especially the last paragraph. Truth with a capital "T" my friend. Techie or not, dot-com casualty or not, they are still words of wisdom!
I would have to say that explosives are the most abused technology in all of history.
hemos is such a dumb bitch she sucks more than dot coms do
If you go to Atlanta Geek's home page, someone has been tracking the number of job postings for various cities around the USA, with data taken from ComputerJobs.com. If this is reflective of the actual IT market (when in fact it's also reflective of third-party recruiter activity), it would appear that most cities have lost 20% of their jobs in the past month, but Boston! They've leaked 30%!
I own a file with the following MD5SUM:
75fe6d69fcc353c35d5b355d107e7323
If I ever wish to prove my identity, I'll simply present that file, and when the MD5s match, you'll know I'm telling the truth.
Incorrect.
Prices fell in the early nineties and stayed down until 1996. If you had to leave the bay area during that time, you took a loss.
SOMA properties in SF are down on average 20%. The rest of the bay area housing prices are soon to follow.
Which of course, brings up the most obvious question: once you have enough karma to post at +1, what's the point of knowing or caring about how much you have?
THe only thing worse is some low UID persons who assume that everything they say is Interesting or Insightful, simply based on their early arrival.
Your post was Informative.
It doesn't mean much now, it's built for the future.
You just wrote a 9 paragraph statement without using proper capitalization ONCE. How does this reflect on you? To me, pretty damn badly.
It doesn't mean much now, it's built for the future.
Not that I really care. I thought they should have included a filter which would allow you to "tune out a UID" rather than restrict it arbitrarily at 50. I toyed with writing such a feature myself and then submitting a patch. But then work intruded and I was too busy to breathe.
I do really like the messages feature. It allows one to see replies without having to get a high enough score to let the post pass the original poster's filters (I read at +1, and sometimes +2). So I suppose my goal of the reply being seen has been met.
And just to keep this on topic: I wonder what effect /. had on the "I'm a 1337 geek because I read slashdot" during the whole tech expansion. I personally know more than a couple people that should never have worked in a tech industry (and I'm sure a few would argue that my Geology background precludes me from membership into the geek club). Mostly artists and the like. The fellas who called writing web pages "coding".
Anyway, interesting graphics. Did you hear about the Uhaul shortage in Silicon Valley/Bay Area? That's a metric I'd like to see explored.
-B
Ash and Hickory, straight-grained and true, make excellent bludgeons, dandy for the cudgeling of vegetarians.
The economic hardtimes that the dotcom (and other IT-related) companies are seeing now it quite self inflicted.
Those companies has not had a working business-model. The major part of the dot-coms have lived on banners and thats just unrealistic.
In you put ads in a newspaper you can get nice big ad-space that really draws attension to your products. Compare that to a banner at the top of a site, do even someone look at them?
And for other IT-related companies I think the focus on market-share instead of revenue just makes it impossible to keep companies alive. Many seems to have thought "market share is everything, revenue is something we deal with then the VC money is gone".
Many has given their products away for free and have thought that later down the line they are going to be able to charge for it. Personally I think it's very hard to make people pay for something that they got for free before. If not impossible.
Extracting data from a web site and graphing it is a useful skill, but I would expect any reasonably smart college student to be able to figure this stuff out. It takes a lot more than that to succeed and innovate when it comes to computers.
yeah sure it's neat enough, but like others have said, it's not really something that is so mind-blowingly new that companies would line up to hire this guy.
I read the letter, and I admit not knowing who he wrote it to, but if he describes his work like that in a job application i'm not surprised he's not getting a job.
"....kinda neat cos....."
Sorry mate, but that just doens't work.....
-.sig sauer-
learn a new language, and go in another country (Europe is -as usual- slowly catching up with technology) to *start* a new market and use whatever you learned during those dot-com years to avoid the same mistakes, and create something useful.
I refuse to believe that all those people did not learn anything, and new nations *need* IT people.
Think about going to Africa, Europe, or another country. The world is big, there's room for everybody, and there are new markets to create. Move on, Americans!
-- There are two kind of sysadmins: Paranoids and Losers. (adapted from D. Bach)
absolutly, I live in central texas and thgouth about looking for a job is austin, I took a look at the statesman and found: ONE admin job with a minimum 5 years xp and a degree, one programming job with similar req's, and a few (computer)graphic artists positions. This was quite a shock so I did some searching on the web in a few job listing sites and found more job postings but nowhere near what I expected, even after concidering the economy.
Those who REAL talents will not wait for jobs.
Rather, jobs will be waiting for those who have
REAL talents.
In essence - those complaining about being unemployed, and still think that they have _talents_, usually have none.
I may sound cruel, cold, and ruthless, but we all live in REAL WORLD, and REAL WORLD demands REAL talents, not fake "talents", or tinie punie tricks.
It takes next to no talent to build a database. Whoever brags of his or her "talents" on the basis of building a database ought to take a good look of him/herself in front of a mirror, preferably not the "smokey" type, puhleeees.
Muchas Gracias, Señor Edward Snowden !
WELL, now what do I do?? With under two years left it seems kind of late to switch my major, and with little actual experience in the computer industry my future looks bleak, to say the least.
How quickly one's future outlook can change...
"Youuuuuuuu... with your fancy clothes....
Dance all night, till your heart explodes..."
(not posting the link because I don't wanna give Adam more headaches with 2 slashdot effects... hahahah. but look for "club drugs" on morpheus)
-marsh
This charting is more of an indication of the grown popularity of Craig's List over that period, which bumps up some numbers and keeps others from being more depressing than they are.
Remember, dot-coms WEREN'T the computer industry, they were the internet industry.
If, totally hypothetically, chemicals were to somehow become the NEXT BIG THING, there would still be an upsurge in demand for programmers cause so much chemistry/physics/biology is now computer-based as is just about every technical field.
Moreover, there has generally been a strong demand for programmers outside of booms - booms just add gusto to this. Course, if you had enough H1Bs to the mix, then you can get the saleries down again. But it's still not a boom-bust thing for real programmers (front-page monkeys though...?).
Was his last name Andreessen?
This is also the basis of much age-based discrimination -- older and more senior workers cost more. Illegal, but so what?
Here's a prime example that any slashdotter can relate too, even though it's not in our industry. Back in 1994, Paramount had an extremely successful syndicated series. But it had been on the air for seven years, all the regular actors had huge fan bases and plenty of leverage to demand higher and higher salaries. The solution must have seemed obvious: cancel the series and start something close enough to grab the same viewer base. So The Next Generation (not great TV, but it had its moments) was replaced by the putrid Voyager.