Looking At Pretty Graphics Of Dot Com Demographics

chris_robison writes "A talented guy wrote a great example of the quality of talent that is going to waste in these economic "hard times". An unemployed friend of mine put together a kickass tool which lets you query a database of craigslist statistics and generate graphs comparing various things such as job postings and apartments for rent. Although the stats are geared toward San Francisco and the Bay Area, it does make for some interesting reading, even if you aren't from around there. Here's his explaination of what he did (included below)" "hey kids.

I'm unemployed and bored.

With all the talk about dotcom booms and people moving away and all these really general statements people are making; i was curious as to what's really going on now, what went on four years ago and what effect did the boom really have on this community.

So.. i wrote a script that went through the craigslist archive on egroups and tallied the number of postings each day for each category then i stuffed it in a database and made a grapher tool.

It's kinda neat cos you can basically see some somwhat hard data on what the san francisco bay area has been doing over the past four years. you can graph job postings from various industries against things like apartments for rent or housing wanted postings, also for sale postings and resumes.

Some of the interesting things I found is that the number of housing wanted postings seems to be slightly down recently but pretty much unaffected by the drop in jobs. perhaps people are always in a state of wanting to move to a city.

One thing to bear in mind, this data isn't one hundred percent accurate for a few reasons... people do repost their information multiple times in a given month, sometimes people post a number of avaliable apartments in a single posting and craigslist has become significantly more popular over the years in question. however, I do still think that it makes for a decent general indicator of trends in our community. (I'm considering doing some kind of normalizing based on the total volume of posts- I need to think about it some more)

(Unfortunately, personal listings aren't archived. I guess that's a good thing... but I guess it also would have been cool to graph activity on the personals categories against some of the categories that reflect the general state of the economy. (maybe when people aren't tied up in career they start to think about more important things, or maybe not...) )

Either way, the data is there, you can look at it for different time periods and categories. draw your own conclusions and have fun!

The url is here:

http://www.signal11.com/charts/chart-o-m atic.cgi

take it easy..

--adam "

Comments

NYC is bad too

[xtremex]

The guys who are MCSEs and FrontPage Monkeys will be out of work for a LONG time. It seems the trend here in NYC is towards the UNIX/Linux world (seriously).
Since there are over 300,000 MCSEs here, and the companies that want them are disappearing, they need a new skill set.
There was always a minority of *NIX people, and THEY are the ones who can still be picky about the pay/job they want.
Since the financial industry is a UNIX world, AIX and HP/UX guys are in HIGH demand.
At the job I got a month ago, they were looking for a good UNIX admin for 10 months before they hired me! That's why their infrastructure in in such a state of disarray.
They hired me on the spot and wanted me to start same day. I am not even the best UNIX admin out there. (I still can't get the UNIX printing system :))

The market is flooded with Java guys from India who will work for less (Nothing against the East Indians..I'm just stating fact)
I now lead a department. Out of 100 in the department, 97 are from India/Pakistan.
They had NO trouble finding a job.
They all have UNIX skills, DB2 or Oracle and they all have Siebel skills.

Not all dot-commers are capable.

[standards]

I don't feel sad for every one of the dot-com unemployed. Many of them were dreamers, who never lived in the real world.

There are/were thousands of silly dot-coms, with stupid business plans and cobbled-together technology. Many of these shops were not part of any "talent pool".

I remember this one guy who I worked with. Marc was a nice guy. Marc thought of himself as a strong manager who was helping to fuel the dot-com revolution.

But the fact is that Marc didn't have a clue about management OR technology - he was just caught up in a ball of momentum. Marc was in a high-profile position in a very high profile organization. But that fact alone didn't make Marc a capable guy.

Marc left us for the dot coms of Atlanta. The last I heard, Marc was still looking for a job. Not because of the economy, but because he didn't have any skills.

site slowness

[ajdub]

the slashdot effect is in effect. we underestimated the traffic it would generate and are moving the script to a bigger machine now.

if you can't get to it, please bear with us and try again in a few hours.

thanks again

--adam

on the IT job or worker shortage

[diamond0]

This month's issue of Communications of the ACM features a reader forum titled "How to Address the Global IT Worker Shortage."

Ralph Castain of Fort Collins, Colorado wrote the following, which began on the same page (12) as Google's ad, "Google Seeks Expert Computer Scientists"

As someone who has been involved in industry and academia over the last few years, I find the current IT "shortage" to be somewhat self-inflicted due to several factors:

• Super-specialized requirements. Many companies express no interest in investing in employee training. Hence, we are treated to job listings requiring a combination of skills unlikely to be held by anyone outside that specific company. For example, a recent local job ad required candidates to have at least five years of experience with a highly specialized, industry-specific software package; three years of experience with a specific software-development package; and one to three years experience in that particular industry. When contacted, the human resource manager blamed the IT shortage for her difficulty filling the position.
• Low salaries/limited experience. A scan of recent job ads at one popular Web site showed an average salary range (where quoted) well below industry standards in the U.S. In addition, experienced programmers (for example, those with more than five years) report being rarely contacted for interviews, even when meeting all other knowledge requirements. This raises the question of whether there really is a shortage within the industry, or a shortage only of entry-level personnel willing to work for below-standard wages.
• Inflated job listings. Companies have become notorious for the placement of job listings on popular Web sites they have no real intention of filling. For example, a large multinational company recently placed more than 100 IT job openings on a Web site over the course of a two-week period. At the same time, the company announced massive layoffs, salary reductions, and forced vacations due to declining sales. How likely is it that any of these listings will ever be filled? Yet the IT shortage studies include such listings in their statistics, thereby distorting the overall picture.

As opposed to a shortage of IT professionals in the U.S., the recent experience of a local headhunter is much closer to the norm. After placing a job listing on a Web site for a SQL server DBA, the headhunter was deluged with more than 300 resumes and 100 phone calls within a 24-hour period. A similar experience followed another listing for a C++ programmer.

The fact is many of us in the corporate world outsource our IT needs to foreign companies and professionals simply because this practice is cheaper. Programmers from foreign countries are willing to work for much lower wages than their U.S. counterparts, especially if they stay overseas. The inefficiencies caused by such remote operations are more than covered by the savings in compensation.

IT shortage? Rather than trusting questionable statistics, I'd recommend asking the people in the job market and the head-hunters.

Ralph Castain, Fort Collins, Colorado

Their talent was wasted during the *boom*

[Anonymous Coward]

You have it backwards. It was during the boom when people were wasting their talent, creating useless software and websites, justified by insane business plans. These hard times are a correction. Their unemployment is a sign to people laid off from the boom that they have wasted their previous years, and they need to find something next that truly will be useful.

Re:Chart doesn't mean much...

[ajdub]

Well...

The resume is on there at the suggestion of someone on craigslist who said that I should put it on there. I thought it was a bit slimey but at the same time I figured that it couldn't hurt.

As far as it getting posted to Slashdot, I didn't post it and I *do* know the guy who posted it but I did not ask him to do so, nor did he tell me before he did it. (and his text is a bit cheesy, no?) :)

As far as useful statistics and normalizing based on total number posted. I am planning on adding that stuff. For the second version, I'm planning on adding normalizing, moving averages by day (right now it's only tallied by month) and possibly some other stats. (I've recieved a few good ideas from a few people and I've got some emails out to some math folks about how I might be able to do some interesting forecast graphs)

--adam

Armchair engineers have moved out.

[heroine]

There's definitely a difference in the quality of the emails I get from software engineers these days. They're much more technically adept.

Seems as if most of the computer scientists working for .coms neither had the interest nor the ability to solve real computer science problems and have now switched to non-technical, and for them, more interesting careers. Wired ran a story about how many armchair engineers from the 90's went back to school and followed their true passion to become actors, artists, and writers.

You might say the quality of software being written today is slightly up compared to the 90's because the only people programming are the ones who really want to do it.