Looking At Pretty Graphics Of Dot Com Demographics

chris_robison writes "A talented guy wrote a great example of the quality of talent that is going to waste in these economic "hard times". An unemployed friend of mine put together a kickass tool which lets you query a database of craigslist statistics and generate graphs comparing various things such as job postings and apartments for rent. Although the stats are geared toward San Francisco and the Bay Area, it does make for some interesting reading, even if you aren't from around there. Here's his explaination of what he did (included below)" "hey kids.

I'm unemployed and bored.

With all the talk about dotcom booms and people moving away and all these really general statements people are making; i was curious as to what's really going on now, what went on four years ago and what effect did the boom really have on this community.

So.. i wrote a script that went through the craigslist archive on egroups and tallied the number of postings each day for each category then i stuffed it in a database and made a grapher tool.

It's kinda neat cos you can basically see some somwhat hard data on what the san francisco bay area has been doing over the past four years. you can graph job postings from various industries against things like apartments for rent or housing wanted postings, also for sale postings and resumes.

Some of the interesting things I found is that the number of housing wanted postings seems to be slightly down recently but pretty much unaffected by the drop in jobs. perhaps people are always in a state of wanting to move to a city.

One thing to bear in mind, this data isn't one hundred percent accurate for a few reasons... people do repost their information multiple times in a given month, sometimes people post a number of avaliable apartments in a single posting and craigslist has become significantly more popular over the years in question. however, I do still think that it makes for a decent general indicator of trends in our community. (I'm considering doing some kind of normalizing based on the total volume of posts- I need to think about it some more)

(Unfortunately, personal listings aren't archived. I guess that's a good thing... but I guess it also would have been cool to graph activity on the personals categories against some of the categories that reflect the general state of the economy. (maybe when people aren't tied up in career they start to think about more important things, or maybe not...) )

Either way, the data is there, you can look at it for different time periods and categories. draw your own conclusions and have fun!

The url is here:

http://www.signal11.com/charts/chart-o-m atic.cgi

take it easy..

--adam "

Comments

Nice one, but some oddities.

[Joel Rowbottom]

This is quite a nice tool which could be used as an example to create "live" management statistics for various applications - certainly doing live plotting of figures against each other. I know I'll try a few new things now I've seen the advantage of using checkboxes ;)

Now, enough praise... ;)

There seems to be some data missing - most notably the "people" data, which would have been interesting to plot the migration of people to and from the area as opposed to housing availability, jobs availability, etc. The other thing is that the server sometimes seems to return an error for some reason or another, although this may be due to the /. effect - I can't imagine the live plotting of data will have a positive effect on the resources ;)

But other than that, a good attempt, and certainly some good ideas there.

NYC is bad too

[xtremex]

The guys who are MCSEs and FrontPage Monkeys will be out of work for a LONG time. It seems the trend here in NYC is towards the UNIX/Linux world (seriously).
Since there are over 300,000 MCSEs here, and the companies that want them are disappearing, they need a new skill set.
There was always a minority of *NIX people, and THEY are the ones who can still be picky about the pay/job they want.
Since the financial industry is a UNIX world, AIX and HP/UX guys are in HIGH demand.
At the job I got a month ago, they were looking for a good UNIX admin for 10 months before they hired me! That's why their infrastructure in in such a state of disarray.
They hired me on the spot and wanted me to start same day. I am not even the best UNIX admin out there. (I still can't get the UNIX printing system :))

The market is flooded with Java guys from India who will work for less (Nothing against the East Indians..I'm just stating fact)
I now lead a department. Out of 100 in the department, 97 are from India/Pakistan.
They had NO trouble finding a job.
They all have UNIX skills, DB2 or Oracle and they all have Siebel skills.

Maybe it's just jealosy

[Anonymous Coward]

No, it's probably just jealosy, but the one thing that really got me about the whole dot-com thing, and the people who identified themselves with it, was the constant repetition of the article of faith that dotcommers were exceptionally talented, smart, or worthy of praise. What about chemical engineers? Ingenious construction workers? (God forbid) Politicians? Accountants? And as for the revelattion that talent goes to waste when the invisible hand gets moody, well, welcome to the real world. I would bet all that I own that there are hundreds if not thousands of farmers in any given state who could out-think the best of the dot-commers; that there are people in your city right now on welfare who are smarter, more inventive, and wittier than you. And me, it goes without saying.
Like I said, this is just jealously, bitterness.. but a social group which lacks the basics of humbleness, and toots its own horn shamelessly, becomes a prick in the side of this lowly helpdesk worker.

oh, hell. I'm posting anonymously.

Not all dot-commers are capable.

[standards]

I don't feel sad for every one of the dot-com unemployed. Many of them were dreamers, who never lived in the real world.

There are/were thousands of silly dot-coms, with stupid business plans and cobbled-together technology. Many of these shops were not part of any "talent pool".

I remember this one guy who I worked with. Marc was a nice guy. Marc thought of himself as a strong manager who was helping to fuel the dot-com revolution.

But the fact is that Marc didn't have a clue about management OR technology - he was just caught up in a ball of momentum. Marc was in a high-profile position in a very high profile organization. But that fact alone didn't make Marc a capable guy.

Marc left us for the dot coms of Atlanta. The last I heard, Marc was still looking for a job. Not because of the economy, but because he didn't have any skills.

Re:Not all dot-commers are capable.

[thanq]

many people are desperate for jobs and these companies were promising great salaries, nice relocation spots to live, and promising growth.

I agree with that statement, but isn't that one of the basic human instincts? Hoarding, scrounging, going out there to get more stuff and stash it, repeat. One could say that the whole length of the "dot-com era" or just the huge IT boom we experienced was based on human greed and human gullibility.

Of course, the reasons for wanting more money are ranging from "just to feed the family", through "to pay off my loans", to get a nice car, a nice house.. but soon afterwards the line is deteriorating and people begin to make money just to have more money. You don't need a 7 bedroom house for you and your wife, you don't need a 100k BMW.

But overall, where I am going with this is that humans striving for more stuff and money created the situation in which people were gullible enough to believe that anything that is "e-something" or has to do with computers/internet will bring millions for each cent you put into it. Of course it worked for a while, when the first ones on this path made the money to be made and innovated the ways that were there to be improved.

I know a few of people that went from 12 years of experience in Emergency Medicine into Java by reading a few books because they wanted more money, or 40-year old construction workers that became instant "expert Web specialists." And another dozen of corporate "company representative speaking.. JUST a moment" office space-like lady workers who knew how to put together a memo or a company policy filled to the brim with meaningless bullshit.

A very interesting fact supporting this theory is that as of today a number of CS classes in University of Berkeley in California is 70-80% full, which differes from last few years when there was 20-30% more people than seats in the classes (quoting a co-worker who goes there right now).

Another words, people will go where the hen laying golden eggs is at, and they will make it lay as many as it can a day before they will kill it off by doing so. And unfortunately, those of us who are content with a golden egg once in a while will have to find another ways of doing what they like/love/want to do.

Hopefully, the hen is not dead yet, just out of eggs. ;)

This upsets me

[litewoheat]

After playing with the chart for a while and noticing the indication of an exodus from the bay area I began to get upset that all these "dot.com" people came to to bay area, jacked up the median rental and home prices, then bailed after things started going bad. Although rentals and home prices have started to level off they have, for the most part, not gone down. If they did that would accelerate our decent into resession. That sad thing is that's inevitable. Foreclosure rates have already gone up.


So now what I have to say to all the people who flocked here, threw high-test gasoline on the fires of the economy, burned everything to a crisp then took off, Thanks a whole lot from someone who was here a long time before you! I feel no pain for any of the dot.com loosers.

Slashdot Job Referral Service

[BrookHarty]

How about starting a head hunting service linked to the /. amount of people in the techincal field.

Everyone bitches about issues, how about using /. as a portal for answers. And OSDN could even charge a small subscription fee for all the new services.

--
Computers are useless. They can only give you answers. Pablo Picasso (1881 - 1973)

Another employment graph/script

[atlantageek]

Hi All,
Another unemplyed geek here. I wrote a set of perl scripts that graphed the # of Jobs available on computerjobs.com for several cities. It is on the main page of AtlantaGeek.com

Dotcom mentality and how it relates to Linux

[Desiato_Hotblack]

Long time reader, first time poster.

Just got my WIRED magazine subscription a short time ago, and what do I see on the cover "Why Linux will lose the desktop war: page 134".

Now before you tell me this has nothing to do with the dotcom boom and subsequent bust, I think you would do well to note these quotes from the article.

After 20 years as a business and technology journalist, [Russ Mitchell] joined Red Hat for nearly a year at the height of the tech stock craze. I was editor in chief of Wide Open News. Tough economics led to layoffs and I got whacked; I left with three months' severance and enough stock options to make a down payment on a house in San Francisco."

If that doesn't relate to the dot-com bust, I don't know what does. He goes on to state the following on the same page:

"Like the Japanese soldiers on Guadalcanal who refused to surrender years after the bomb ended World War II, Linux zealots remain obsessed with beating Microsoft in desktop computing. Desktop computing? Don't they know? The war is over. Microsoft has won."

Pretty damming commentary from someone who supports Linux, don't you think? Or at least he still does, but on the server platform where it has gained the most ground.

Comments?

I'm sure that this will take forever to become a proper thread unto itself, that is why I've bothered to post it here today.

Desiato_Hotblack

Re:Yeah

[Anonymous Coward]

Compainies like HP, Lucent, and JDSU

Read my post. I said "dotcom casualties"... are those companies dotcoms?

Source of figures

[Jah-Wren Ryel]

I could really make good use of those figures that show that an outsourced 1st level support position can cost $5k-$6k. I would be very grateful if you could post a citation on those numbers.

Re:Chart doesn't mean much...but...

[Teun]

Still a neat project and interesting to play with for someone from 'out of town' (Europe)
The source(s) and limitations are stated on the site and any one knowing statistics knows aswell how to use this one.

Good luck on the job hunt!

thoughts on dotcom booms and busts

[ajdub]

while i don't have too much knowledge as to the effects of the dotcom boom and bust nationally and globally, i have formed a few opinions on what has happened locally.

while the boom sucked in a number of ways: computer jobs became glamourous and attracted a lot of goldrushers or hollywood ego types, which made working in these types of jobs a little obnoxious. the people who i hated in high school who were stuck up and going to fancy colleges found themselves getting rich in marketing and management positions, riding on the backs of the geeks who actually did the work. the talent pool was suddenly flooded with people who weren't really into what they were doing but they were doing it because that is where the money was. etc..

it also did a few cool things: it provided a way for slackers, artists, writers, etc to actually earn a living wage. (not exemplary salaries, but salaries that actually made it possible for them to live reasonably comfortably in the bay area) suddenly people that had been making peanuts could make a decent salary 30-50K and live comfortably knowing that they could pay their bills and some were gaining a skill that could turn into a career.

unfortunately, things got out of hand when the aforementioned fancy college educated folks had a little too much money to play around with and traders saw a wave of hype which they could ride to grow their assets. idiotic companies with pathetic business plans, idiot management and stupid ideas were recieving insane amounts of venture capital, which they were happy to spend the majority on things like office adornments, lame marketing and other things irrelevant to the core products they were building.

when this all was happening, the rest of the economy grew to support all these new businesses needing services and now that they're all gone, not only is the technology industry affected, but also the industries they relied upon and the industries those industries relied upon. hopefully the ripple won't be catastrophic.

personally, i was happy to see the bubble burst at first for the reasons i listed above. i was sick of hollywood ego internet types and this sudden 'coolness' that surrounded all things internet. i was also sick of seeing people who had no real interest in technology at all get filthy rich off of the hype.

however, now my opinion has changed a little. a lot of my friends are in fairly gnarly positions as a result of the layoffs. they weren't trying to get rich, and never were. they just wanted a honest living wage for a days work. they had the promise of a career and living wage for the years to come laid out before them and now it's all vanished into thin air.

in one way, i think it's a good thing, because the boom distracted people from what they really wanted to do with their lives. while doing operations for a dotcom may pay decent, it's not the most productive thing to do with one's eight hours a day in the grand scheme of things. (given that someone is not completely a geek at heart) now people are being forced to reevaluate themselves and their master plans for life, some are going back to college, some are persuing art, others are getting involved in vocations that they've been meaning to do for a while and i think that that is a very good thing.

i think that it is also a good thing for technology. now that there are less distractions and promise of riches for geeks working on bunk technologies, there will a decent talent pool of geeks looking to work for a decent wage on something that is technologically cool. that is also pretty exciting. while venture capitalists may be stopping the handouts, they're not going to stop completely and now they're not going to have nearly as many proposals to review. i'm hoping that in the next couple years we'll see a number of new companies that are doing stuff that is truly innovative.

Re:jobs - housing.

[Tide]

Maybe elsewhere in the country that would be a fine philosophy, but in the bay area where your paying $1400 a month for a studio, its not that easy. Ive had several friends that were laid off just immediately leave. Just like rats, we all know that the ship is sinking. Just my 2 cents. :)

Here's a multi-state jobs availability chart

[Fulton Green]

If you go to Atlanta Geek's home page, someone has been tracking the number of job postings for various cities around the USA, with data taken from ComputerJobs.com. If this is reflective of the actual IT market (when in fact it's also reflective of third-party recruiter activity), it would appear that most cities have lost 20% of their jobs in the past month, but Boston! They've leaked 30%!