Slashdot Mirror
Tracking the IT Job Market with a Bot
atlantageek writes "Is the
IT job market improving? Is the growth in Unix or Windows? Should I
study Data Warehousing or E-Commerce? Identify the recent trends with
CJ Miner, a small tool I've written that has been monitoring the Computer Jobs website for the
last year."
"track how the computerjobs.com website has been doing"?
Ive written an advanced bot, plz hire me.
jobs for unix and windows development are even.
Also they have the similar pattern of availability!
I second that, nice work. Lets see some statistics from this too. Like where jobs are going up and where they are going down. That kind of stuff.
-----BEGIN PGP SIGNATURE-----
12345
-----END PGP SIGNATURE-----
check out the business cycle!
1) How many of these ads are actually real?
2) What do these jobs mean in terms of disposable income?
A bot taking another IT related job! Where will it end?
"There is no time, sir, at which ties do not matter," Jeeves, (Jeeves and the Impending Doom)
Have a look at the entry-level jobs. There's a sharp crash in the beginning of march. Anybody knows why?
This is really interesting. You could use the same data to determine where to find the networking jobs, or to correlate trends in skills or locations. Can I get the raw data?
As Monte Python might put it: "Wait a minute. This isn't data mining... these are just frequency counts!"
But how many jobs have he gotten with his kiddie script monitoring one website? There's no alternative to updating your resume, prowling multiple websites for job listings, submitting your resume, and playing phone message tag until you land a job (or, more likely, a contract).
:P
As for my experience looking for a job recently in Silicon Valley, I would say things are getting better for contract work. From what my friends are telling me, companies don't seem to be hiring for the long-term. I even got an email from a Microsoft recruiter for a contract job.
Could you at least have chosen colors other than red and green for the first two choices!? I'm colorblind, you insensitive clod!
SIGSEGV caught, terminating
wait... not that kind of sig.
One question though: Why computerjobs.com? I'm not real familiar with their site, but are they one of the sites that claims to consolidate complete listings of I.T. jobs from a number of other large job search sites (Monster, CareerBuilder, HotJobs, BrainBuzz, etc. etc.)?
If they really do get a pretty good number of I.T. related listings all collected up in one place, then yes - I think this is a pretty useful little graph/tool.
I've been out of work since the beginning of May, and living in the St. Louis area, it seems to me that there are currently very slim pickings. I keep hearing talk of the economic recovery, but at least around here - I'm not really seeing it.
According to your chart, that would be an accurate accessment too - since it clearly shows a sharp decline in I.T. jobs available in St. Louis since April of 2005. (And worse yet, I'm really mainly interested in the hardware side of things, but if you look at that specifically - you see that in my city, there were only a grand total of about 2 jobs fitting that category, at any given time!) In the whole U.S., it looked like I.T. hardware jobs only averaged around 1,200 *total*, for that matter. Not good... not good at all!
You know if you want to check out how tech jobs are doing why not go here?
Philosophy.
>> Should I study Data Warehousing or E-Commerce
You should study what you enjoy to study and work on. If this means a less trendy job, so be it. I'd rather eat broken glass than do web development, for example. Even if it was the hottest job on the market, I wouldn't do it anyway.
Why the sudden increase in networking jobs in North California from April-May 2005?
For all the liberal bleading heart crap I read on this site, there are some mean people here. I thought the information was fairly interesting.
Read his resume, he's in Atlanta. I went to Georgia Tech and lived in Atlanta for a number of years, and this site was the first I would check when job hunting. It's fairly big there. They don't aggregrate other site's jobs, employers have to pay. With the exception of the head hunters, they were quality jobs. (But I've found that every head hunter posted job I've ever applied for has been a joke anyway)
Just because you have a job doesn't mean you are better than someone who apparently does not. I went over 6 months from my previous job to my current one, even turning down two jobs and getting jerked around by Novell for 2 months to find the right one.
Keeping myself busy with random projects was a great way to practice my skills and learn new ones.
Whats that (Visa Sponsor)?
...I would be pleased to get a line on just ONE "hot IT job". Write me a bot that will do that, and suddenly I'll care about your war3z. I live in Japan though. Interesting position. Or, lack thereof.
Don't trust anyone under thirty.
Timothy now has all power-ups and 30 continues, ICT market unchanged. Konami executives currently giving no comments.
And tomorrow the stock exchange will be the human race
where the f is software on the skill list??
I worked at CMGi for five years, and a good portion of that was working for a daughter company named InfoMation on a project called Echo. It was a web scraping tool that let you create your own personal newspaper, from any source you wanted (web, usenet, mail, rss feeds, etc). Imagine google alerts but you could create custom news feeds to any sites you want.. Have it look each day for new albums from your favorite bands by creating custom searches on music sites.. Have it monitor your competition's website for changes (or look for press releases regarding your competition on the wirefeed services). Track articles in industry trends you want to follow.
I really liked our alert system (ahem), which would put the titles of articles in the subject line, unlike google alerts, which just puts the category.. You could also subscibe via daily summaries which only sent one email per day, or via cellphone/pager sms messages.
I miss it. Unfortunately our sales department tried selling it to the insurance industry, which takes way too long to purchase technology (as opposed to, oh, say, a stock broker, who would want to follow every stock he's got his clients invested in.. go figure).
..Jeff Keegan
seven syllables explain TiVo: kee gan dot org slash ti vo
However, whenever a shortage of labor occurs in the IT market, the government consistently intervenes by importing H-1B workers to fix this shortage. As a result, the growth in wages is damaged. Working conditions (like working 60+ hours per week) do not improve.
Any perceived shortage in the market for IT labor is illusory. If this shortage were real, it would be short-lived, due to government intervention.
By the way, we see the same phenomenon in the market for unskilled labor: e.g. picking vegetables and fruits. The government fixes this shortage by allowing illegal aliens to flood this market for unskilled labor. As a result, wages (hovering around $5.00 per hour for fruit-picking in Southern California) never rise. Working conditions (like standing for more than 9 hours per day in the strawberry fields) never improve.
The rub is that politicians do not care about Washington's gross tampering in and bludgeoning of a (relatively) free market like the USA. Washington is eager to fix shortages of labor. However, Washington rarely fixes shortages of jobs by, for example, creating more government jobs. The interests of Washington are not aligned with the hopes and aspirations of middle America.
We should close the American market to (relatively) non-free markets like India, China, and Mexico. Further, the American market should be flung wide open to (relatively) free markets like Eastern/Western Europe, Canada, and Japan. Free trade is good -- only when we are trading with other societies that maintain (relatively) free markets.
Should I study Data Warehousing or E-Commerce?
You should be studying Computer Science...
Ever wonder what happened to all those mainframe or COBOL folks? Knowing about E-commerce, Unix, Windows, Java, XML, or whatever the technology or trend du jour is might be impressive now, but in a few years, come the next thing, where will you be then? These things change at the blink of an eye.
On the other hand, algorithms, computability theory, formal languages, predicate logic, etc. don't.
A solid foundation of the theory will enable you to understand and learn whatever specific language or technology you need for the job, and allow you to be nimble enough to quickly pick up and go with the latest trends as the market changes.
-- Samir Gupta, Ph. D. Head, New Technology Research Group, Nintendo Co. Ltd., Kyoto, Japan.
Should I study Data Warehousing or E-Commerce?
You should learn how to provide vertically integrated e-commerce solutions providing dynamic interaction to customers in synergistic markets.
Knowing how to work a sock puppet also helps.
Take the cheese to sickbay, the doctor should see it as soon as possible - B'Elanna Torres, "Learning Curve"
You know what, this is just some ploy to get the slashdot crowd out of there mothers basement in to some cubicle.
/. continously, not having to bath because I lack interpersonal skills.
:)
Really, I like being stuck in doors, lacking in vitamin D, watching $Some_Sci_Fi, refreshing
* having no job is kinda cool, think of all the free time I have
/. is good for you.
Sixth item from the bottom of the list.
d etail/
check it out
http://www.thinkgeek.com/gadgets/electronic/732d/
every day http://en.wikipedia.org/wiki/Special:Random
"1) How many of these ads are actually real?"
Not many I'm afraid. Too many companies out there are "testing the waters" by posting false job postings and then tracking the number of resumes they receive.
The app's name, "CJ Miner", is strangely appropriate. In the adult web industry, CJ stands for "circle jerk", and refers to ad-laiden websites designed to fool visitors into believing that the site offers free content (pics and videos), when in fact nearly all links actually lead to other CJ sites.
If you're in the UK... http://www.jobstats.co.uk
The chart has an option for "Legacy Systems" which sounds way too general. I mean, isn't everything currently running in production legacy?
Speak truth to power.
Most jobs arent even posted... So while you may get an estimate, its no where accurate enough to stake your future on.
---- Booth was a patriot ----
Too bad -- getting a profile with the same measurement instrument over the entire bubble-bust cycle would have been very valuable for historians of technology and the politics of H-1b visas most particularly.
I suspect what the graphs would have shown was a far faster drop-off in the jobs for languages other than Java than for Java due primarily to the fact that the Indian CS diploma mills were set up largely by guys from Sun and related companies.
That's correcting itself somewhat now that there are so many project failures due to the attempt to run the presentation layer on the server because -- well -- that's where the cheap programmers know how to program stuff.
Seastead this.
I would agree that it's good to have the theoretical background, but without the desire to learn new things, I'd say it's much less useful than following the trend du jour.
I recently read a blog post by a VC who took a different approach to predicting the future employment rates in the U.S. As a VC, he looked at VC investment in the past N months in the U.S., looked at how "early" those investments were, and basically concluded that since there was a lot of investing, these companies will be growing and hiring people. I don't know how big of a drop in the sea of employment this is, but I thought it was an interesting way of analyzing the situation.
Simpy
Surely, you have to be a US citizen before you can join the military?
Nope. Non-citizens may enlist in the U.S. armed services. Think about it: France has a foreign legion; why can't the USA?
He used to link to it in his /. sig but then, about half a year ago, he announced that he wanted to "move on with his life" and would sell his code, site and related goodwill.
I think part of the deal was that he was going to open the source but I never heard any more about it. Anyone know if that happened? Or of any FOSS projects working on graphing employment data?
Nah, the real myth is that a CS course of study doesn't really prepare you for anything, and is irrelevant. As someone who's been in IT over a decade after jumping over from another profession, I've had to work extra hard to keep up with the guys who got all that "irrelevant" theory hammered into them when they were in their 20's. Of course, as you say, there also has to be the desire to learn new things -- but that's not a problem I've seen with CS grads, they're mostly gung-ho about the work. It's the trade school/cert mill guys like me who often fall short when it comes to pursuing our craft to the next level. I'm very lucky to be surrounded by a group of really smart CS people at my job who are always pushing the envelope (and somehow simultaneously managing to raise families and keep their marriages intact...).
This thing is broken...
I checked Phoenix, New York, and So Cal... and the graphs clearly showed New York above Phoenix. I think checked Phoenix and New York, and the graph shows Phoenix above New York! What gives?
So, yes, CMI has (relatively) non-free markets, and the West has (relatively) free markets. What seems to be the problem?
When the masses in CMI fix their economies, they will have enough jobs. Then and only then should the USA open its market to CMI.
I'm a CS major, and frankly, it's disgusting to see how many people can get a degree and not know how to program at all.
That being said, a language is nothing more than a way to describe a concept. Ask the "trend du jour" people about programming concepts, and you'll probably get a bunch of software engineering babble in the reply. (an experiment of this would be interesting, given that the person wasn't mislead)
My basic belief in learning computer science is to learn how NOT to be a code-monkey. Any idiot with minor interest in the topic of languages or databases can become a code-monkey.
I think Dijkstra was wrong about the cruelty of computer science. The true creulty is that we teach students more than the watered down industry will ever demand. Kind of like putting a professional athlete in a little-league team.
Why is this? One thought that has crossed my mind is something that a prof of mine, who used to work at IBM, once said "Back in the 60s programmers were created, not hired." Because there weren't many programmers at the time.
Now, if people are being trained on the company dime, the employers are going to cheap out. They are going to set a bar of "getting it done" and only demand as much. (we see this today in many parts of the industry)
Steve Jobs once said that "A players hire A players, B players hire C players", where the question was posed "then how do you get B players?" I think that in this case, C players hired people that would become (on average) at best B players.
Over time, these B+C players set the industry standards, both in hiring and development. (for example, if you are a boss and only know COBOL, are you going to start projects that aren't COBOL, with the loss of job security as one consequence, and that the employees currently only know COBOL?)
Which leads us to today and the demand for the "trend du jour" which is just an extension. Where programmers have been forced to ride the wave for decades to maintain bare employability. Thus, the market asks for it, people looking to do the bare minimum supply it, and a de facto standard in the common language is created.
What I'm saying is that if one chooses to enter this industry on the "trend du jour", they better be willing to have to learn the latest fads well into their fifties.
As for what the next trend is, I've heard that the best way to gauge that is to go into any CS department. What they are doing is what you will most likely be doing in 10 years.
Am I open minded towards open source, or closed minded towards closed source?
Or you can use indeed.com, which lets you search all jobs within the last 30 days from almost a thousand job sites (including computerjobs.com).
You don't even have to visit the site to check for new jobs -- it has RSS feeds and email alerts for new jobs that match your search criteria.
Or if you're really ambitious, use their free XML API and do whatever you want with the data.
Whoa! What's with the BSA ad I see associated with this article?
I think the picture speaks for itself :-(
Is that it does not offer predictions on future jobs. The stock market has software logging the performance of thousands of different stocks over several year periods. However, that gives no insight to how the stock WILL perform.
This is the exact same situation, this software logs the jobs available, but gives no insight to future availablity. This type of software should only be used as a rough picture of how the sector is performing now and not how it will perform 3 months from now.
I would hate for somebody to spend a month learning e-commerece and find all of a sudden, the availability of the job is on a downward spiral.
And btw, "free market" is just an idealized abstraction - with the risk of burning some karma I'll say it's pretty crude one, actually.
The notion of "free market" was developed during the Enlightenment or "Age or Reason", when Europe started to attribute everything to science and reason (after the Middle Ages attributed everything to the will of God). As a consequence, it is based on the fact that players in the market make their decisions rationally, when, in fact, they very clearly do not. A free market also has failure points like monopolies, cartels, or the fact that it totally ignores environmental issues such as pollution.
The US is probably the closest country to a free market, however, it still does things like subsidizing agriculture products, and even steel a couple of years ago. As for Europe - they're actually much "worse" in not following the free market dogma.
As for your suggestions. 1. Saying "let's prevent people from getting H1s and work within the States" is clearly promoting isolationism and limitting a person's rights instead of more freedom. 2. yeah, let's just close China off ... . You're probably ignorant of the fact that the US has been 0wn3d by them. They're basically subsidizing America's unbelievably-high debt. We can't really upset them too much.
Finally, I'll say that I'm not against economic freedoms. What I'm against is using the "free market" dogma like this.
The Raven
Goddamnit, that's why the last two months of resume-filing has felt like pissing into the wind. At least I have my medium-yecchy helpdesking job, for now.
So how do I find people who are actually looking to hire?
--grendel drago
Laws do not persuade just because they threaten. --Seneca
They'll probably start by scoring all the first posts....
"Those who have never entered upon scientific pursuits know not a tithe of the poetry by which they are surrounded."
Apparently the graph + legend is a fixed size. I tried lining up all of the different Skills to see how they compared and the size of the legend squished the graph vertically, causing it to be unreadable. And I hadn't even started to add the Locations. So this graph is useless if you want to compare more than 10 skills/locations at once.
"There are more important things than stopping terrorism. Upholding the Constitution is one of them." - Ars Forumer.
My script would poll the site every hour or so and notify me, if anything not seen before appeared.
In Soviet Washington the swamp drains you.
SkillsMarket is apparently still in business, and Hilton is still trying to sell it.
There is another, similar site, whose URL I cannot remember right now.
A lawyer & digital forensics examiner. Also an expert on open source software (OSS).
If I had mod points, I'd mod you up. Interesting post.
I'm sorry, but basing the "health of the IT employment market" on the number of job ads is flawed.
Employment consultants are somewhere just below Real estate agents and car salesmen, where their job is not just making an individual sale(read commission), but portraying a positive and bouyant market, while continuing the industry's success.
By glutting the market with jobs ads that many would not stand up to audit, achieves several things:
- A percieved shortage means that commissions remain higher.
- Give the business community the belief that things are buoyant and its OK to pursue that IT BPR at last.
- Keeps the IT market turning over, rather than IT workers staying where they are because they are insecure, because they believe the market is flat.
In post Patriot Act America, the library books scan you.
Hey why go do a job. Send me some cash at moron@pleasetakemymoney.com and I'll tell you how to work from home making 85K and be off by 2:30PM every day!
Until monster.com and computerjobs.com and all the rest can get rid of these ads they risk being pretty useless.
They can take mah job, but they can't take away mah Bible!
It seems there are some minor issues with the scaling of the graph, for instance try viewing the So. California alone and then So. California and D.C. Metro; look at the max on So.Cal. in both cases, they are shifted by about 300 according to the scale. Maybe it needs some sort of calibration, I guess?
You're obviously taking user interface design at an American college because the amount of pure torture in checking off 20 checkboxes by hand and reading a tiny graph 1/10th the size of the page can only be explained by the sheer dominance in inferiority of that university system.
Now if you went to an Indian college you would have allowed us to compare all 20 options at once on a readable graph without having to kill ourselves checking boxes. If you can't make it work, outsource it.
Now one thing you learn after college a lot more than you learn in college is exactly how to differentiate between jobs. The real world isn't defined as much by the type of programming you do as much as the scope of your responsibility.
Resume readers don't care if you're a Windows programmer, a UNIX programmer, a hardware designer, or a secretary. They want to see if you're a programmer, project lead, project manager, marketing manager, director, etc.
Things like Google, open source, wiki have leveled the playing field to where it doesn't matter if you study hardware, windows, AS/400, or UNIX. These things can all be learned by anyone at any time. In modern companies the skills at any given level of responsibility are being learned on demand as they're needed. Hardware designers one day are being used as UNIX programmers the next day.
Todays differentiation is in how much responsibility you're capable of having. Most resumes are being divided into management, sales and programming and as far as we can tell from the 36 checkboxes, management is the place to be.
Personally, this submission to Slashdot just looks like a cheap way of advertising a lame job board. If there was a real intent to track the IT market, all one would have to do is go to dice.com, and use the bots that you get for FREE there.
Dice.com is bigger, and nationwide. So you could really track how the IT market is doing over time.
Dice is also the only board which takes consultants seriously. That is, it offers specialized selections to make it easy to track consulting-only gigs, so that you don't have to wade through a bunch of full-time jobs just to find the ones which are for consultants.
Plus Dice doesn't charge you $50 to post your Resume, like this site does.
So, IMHO, this Slashdot submission is just a scam to get free advertising for yet-another-lame job board.
The best way to predict the future is to create it. - Peter Drucker.
Goddammit MOD PARENT UP!!! It's been a while since I read something as true and as insightful on slashdot. And modders beware particularly of those who disagree with him in long posts; writing too much does *not* equal being insightful - I say this because I have just seen a long post of someone who disagreed with him modded insightful though on reading it I had the certain feeling the modder either was an idiot or did *not* bother read that reply and examine its content for vailidity.
And for anyone who thinks the US is a free market I'd say you're deluded! You need to read the book "protecting capitalism from capitalists" to understand the concept of "incumbency" and how big business and the rich in the US use their influence upon government to keep things in their favor and oppose the freedom of the market. http://www.savingcapitalism.com/
Jobstats.co.uk has been doing this for years, and aggregating counts of listings from multiple sites.
My Karma: ran over your Dogma
StrawberryFrog
yes is has been done a few hundred, but not a 1000 job sites http://blog.indeed.com/2005/02/03/window-on-the-jo b-market/
http://devnulled.com/content/2005/01/an-evaluation -of-the-current-technology-job-market-updated/
Job Postings Per Capita
http://www.indeed.com/jobtrends.jsp
Can't wait to see what Google does for a job site aggregator
(1) buy indeed.com
(2) put up Google Ads (just like gmail)
(3) ahhh, yes white collar money laundering!
While the government raised the number of H1B visas in response to the Internet bubble (belated as usual), the H1B numbers have since shrunk back to pre-1995 levels. All H1B visas for 2005 were exhausted in a single day last October. According to the parent, since there is a cap on H1Bs we should be seeing the raising wages and improved working conditions. I can't wait...
I got kinda bored reading job adverts everyday and decided to write something to manage the process:
http://www.jobtrawler.co.uk/
A great deal of the jobs in Dallas are for specific technologies; Websphere, Java, Oracle, etc.
You won't have job satisfaction **even as a Senior Project Manager** if you are a 'such-and-such guy' being shoehorned into another discipline.
If the focus is on your station, then it's all about the $$$, not the work - which is what a lot of us are in this for.
Here is a nice web site for posting jobs : http://e-jobs.com/