LinkedIn Is Open Sourcing Their Testing Frameworks (github.io)

destinyland writes: LinkedIn is open sourcing their testing frameworks, and sharing details of their revamped development process after their latest app required a year and over 250 engineers. Their new paradigm? "Release three times per day, with no more than three hours between when code is committed and when that code is available to members," according to a senior engineer on LinkedIn's blog. This requires a three-hour pipeline where everything is automated, from committing code to releasing it into production, along with automated analyses and testing. "Holding ourselves to this constraint ensures we won't revert to using manual validation to certify our releases."

Comments

Just remember

[Ol Olsoc]

Share your email login credentials with them. Much appreciated.

What is amazing is how many people do just that.

Re:Just remember

[sonamchauhan]

hmmm... did ethics testing get automated here?

{ assert true : "Danger Will Robinson, Danger! Unethical program."; }

Re:Just remember

[Ol Olsoc]

hmmm... did ethics testing get automated here?

{ assert true : "Danger Will Robinson, Danger! Unethical program."; }

When I was originally invited to join Linkedin, it was through an invite from a colleague. I went through the steps, then figuratively shit myself. They just outright asked for the password. The putative reason was to let them data mine my address file.

But I kinda wanted to keep my job at the time. Ethical? Hell no.

Linked in Open sources

[invictusvoyd]

Their Spam framework.

Re:Linked in Open sources

[Ol Olsoc]

Their Spam framework.

Trying to make a "six degrees of LinkedIn."

Arbitrary rules

[Anonymous Coward]

Why 3 hours? Why not 2? or 4? Why not manual? Bugs are by their nature UNanticipated and automated testing tools only test anticipated bugs.

"Holding ourselves to this constraint ensures we won't revert to using manual validation to certify our releases."

Why? It seems to defy logic. The first time you do user testing is when real world users are using it. That's moronic.

Re:Arbitrary rules

[Dahamma]

Because they are a fucked company, they know it, and they need to do something to get more publicity.

Re:Arbitrary rules

[Hognoxious]

# Because THREE is a magic number ...
something something trinity (or is it trilogy) /#

Re: Arbitrary rules

[AngryLibertarian]

It only seems moronic if you don't know how it's done and the only training you have is how to write code. Study design pattern for testing and deployment. Jez Humble is a good source.

Re:Arbitrary rules

[KGIII]

Dunno? But, it seems to me, more and more of these things that I read about various companies open-sourcing their material makes me think that they're just looking for someone to fix it for them for free. Maybe someone should suggest they be a bit more flexible with QA than to apply arbitrary time limits? Some things just can't be fixed in the code.

Re:Senior Engineer?

[rednip]

I think that you lost the war for 'what engineer means' when 'train engineer' became common lexicon a long time ago. Perhaps you could take the better fight that 'Software Architects aren't really Architects'.

Re: How do they handle database changes?

[Anonymous Coward]

It doesn't matter. All your base are belong to us.

The better to spam you with!

[brettw]

All this efficient check in and automated testing, just so they can send spam to every email they get their hands on.

My employer specifically marks linked in spam as "NOT SPAM". So custom filters it is.

"Open Sourcing Testing Frameworks"?

[Irate Engineer]

My buzzspeak is a little rusty, but this sounds suspiciously like "Beat monkeys to code faster, send code out the door without testing, and just let the users figure it out". Did something get lost in translation?

Re:"Open Sourcing Testing Frameworks"?

[houstonbofh]

My buzzspeak is a little rusty, but this sounds suspiciously like "Beat monkeys to code faster, send code out the door without testing, and just let the users figure it out". Did something get lost in translation?

From the bugs in groups, and the vast drop in traffic to the same, I think you hit the nail on the head! They hit the nail with their head...

Re:"Open Sourcing Testing Frameworks"?

[tlambert]

My buzzspeak is a little rusty, but this sounds suspiciously like "Beat monkeys to code faster, send code out the door without testing, and just let the users figure it out". Did something get lost in translation?

Yeah.

They deploy to internal users -- LinkedIn employees -- via Enterprise Enrollment for iOS devices, and via Google Play’s alpha testing functionality and the developer console API. So it's not like they are sending the code without testing, and it appears that at any time, an internal user can torpedo a particular release candidate, so that what they end up with at the end of the week might be the same thing they had at the end of last week.

This still has a bunch of holes, just as any continuous integration process does, but at least by staging a 3 hour limit on things, they avoid falling into the trap that Google and Apple have leapt into willingly, where "it takes as long as it takes" to do integration acceptance testing.

In particular, both Google and Apple engage in reactionary testing paradigms. They build tests based on bugs they've previously encountered (because they are in fact able to test for those, since they've been well characterized during the process of being fixed), and stack them on top of all the previous tests.

This results in an ever-growing set of tests that test for regressions, but which do not test for correct behaviour of the products, overall. Instead, it lets you know when you've encountered a bug which you have previously encountered.

Further, since the testing is unbounded (and in fact, will stop the build process at Apple or Google by throwing a stick into the bicycle wheel of progress -- an event which is called "closing the tree", necessitates the existence of a "tree sheriff" role to deal with automated closures of the tree, and fails to automate the back-out of code leading to failure -- it leads to flaky tests.

The flaky tests come about because there's a human involved in making the decision, and in structuring the situation that way, with an unbounded closure interval -- practically bounded only by people showing up in your office (Apple) or open plan desk/cubicle (Google) and breathing down your neck -- there's a strong encouragement toward flaky tests, and there's a strong tendency do disempower the "tree sheriff" in favor of getting the expensive engineers back to work. And because of this, it's very hard to deep dive into flaky tests.

So we are left with the situation that flaky tests are both tolerated (as long as they are not "too flaky" -- or in Google-speak -- as long as they do not exhibit "too much 'jank' when run"), and the flaky tests do not get cleaned out eventually because the set of tests that you run are not pruned occasionally to run in bounded time, and if you try to do a deep dive to actually resolve the flakiness, you get the disapproval of your peers.

So a bounded time is a good thing, if you insist on having continuous integration, but it doesn't necessarily fix the other problems with continuous integration, and it doesn't fix the fact that most testing in organizations where engineers are involved in "agile methods", such as "Scrum Runs", where the primary goal is code base stabilization, rather than forward progress on the product for 2/3rds or more of the engineers time... well, combine that with an iterative process and it's a recipe for engineers who don't know about testing writing a lot of useless tests, because you are moving the goalposts on them iteratively anyway.

So they've done something relatively innovative, but it's probably not innovative enough. And it's probably an interesting experiment, but I would be wildly surprised if other organizations adopted their framework.

P.S.: The other problem with iterative models, particularly constant integration iterative models, is that you tend to only ever do evolutionary things, and never revolutionary things. There's not always a drunkards walk that gets you "from where we are" to "someplace new and fantastic". There are saddle points, and once you are in a saddle point, an iterative model will not get you out.

Re: "Open Sourcing Testing Frameworks"?

[Entrope]

So how do you "test for correct behaviour of the products, overall"? It's not that hard to spell out -- and even automate -- the cases that a person expects to occur. The problem is accurately modeling the state space of the program and then figuring out which walks through it are incorrectly permitted. Even the most sophisticated known approaches fall down for the first part of that problem due to the vast size of an interesting program's state space.

Re: "Open Sourcing Testing Frameworks"?

[AngryLibertarian]

Access to a tool is a small portion of being able to use it effectively. I can give you a free table saw but you'll just cut your hand off if you don't learn how to use it.

Re: "Open Sourcing Testing Frameworks"?

[tlambert]

So how do you "test for correct behaviour of the products, overall"? It's not that hard to spell out -- and even automate -- the cases that a person expects to occur. The problem is accurately modeling the state space of the program and then figuring out which walks through it are incorrectly permitted. Even the most sophisticated known approaches fall down for the first part of that problem due to the vast size of an interesting program's state space.

Personally? If money were no object, I'd use software that cost thousands per engineer in about 1985 or so, but which is now sold as "McCabe IQ". By 1990 you could get three floating licenses for "only" $45,000 (not sure on the current precise price; it appears to *start* at $2,000.00/one-time/user, which generally means that a useful configuration would be more hefty in both price and capability than the "starter" version).

I'd use it to ensure that the requirements document is implemented by the software, and then do a branch path analysis on the resulting code to make sure that the test cases that get generated cover all possible code paths.

Given the substantial costs associated with the tools, most software engineers today have no exposure to *real* CASE tools (but if you care, you can obtain a 30 day free trial from them these days).

Obviously, it also possible to do this type of code coverage testing manually, but most test engineers are trained to do mostly ad hoc testing these days -- assuming you have actual test engineers on your project at all.

If you want a review of McCabe IQ, here's one from ~1990: http://stareast.techwelldev.co...

And here is a more recent company overview: http://www.bloomberg.com/resea...

Re:Senior Engineer?

[Fly Swatter]

At least per wikipedia, US train engineers must renew certification every few years. So even they must be certified to obtain title of 'engineer'.

3 hours is waaaay too long

[JustAnotherOldGuy]

Oh my heavens, 3 hours is waaaay too long in this fast-paced, ever-changing world.

Why not just do an automatic commit with every keystroke, like the Windows 10 telemetry does?

Neglecting to care

[WaffleMonster]

I would rather eat rats than join LinkedIn.

250 developers working a project for a year at any other company would cause me to be curious about what that project is. In this case I have no interest in ever knowing.

Releasing three times a day with three hour _ceiling_ to "ship it" is only possible while harboring extreme levels of disregard and contempt for your users. Refreshing to see LinkedIn's corporate philosophy so well represented in everything they do.

Re:Neglecting to care

[Dahamma]

Seriously, yes. What the HELL do they do with 250 engineers that no one has given a shit about (beyond the investors who watched their stock lose almost 50% in a single day) in the last couple years?

I won't even click on random search links to their site (let alone use it for business networking) once I realized how all of your "private" data is basically negotiable to anyone who wants to pay more than you to get access to it.

Re:Neglecting to care

[JaredOfEuropa]

In my last position I needed to find certain professionals in other companies from time to time, and "cold call" them (no, nothing to do with marketing or recruitment). LinkedIn actually made that very easy, certainly easier than going in through the front door and the receptionist, and no one minded being contacted in that way. Sometimes I found a mutual connection to introduce me to the other person. Similarly, several people from ther businesses found and contacted me through LinkedIn. It's a good tool for business networking if you treat it as a service for just that, and limit your connections to people whom you actually know.

I have the same mixed feelings about LinkedIn as I have for Apple: I don't like their corporate policy but I do use their products because I find them very useful. As for them selling on the data: all of my data on their service is a matter of public record anyway. And that's how you should treat any social network: everything you put on there and (not unimportantly) what you do in there is public, mined and/or sold.

Re: Neglecting to care

[AngryLibertarian]

My suggestion to you is to study your craft. :)

Re:Neglecting to care

[BarbaraHudson]

It's a good tool for business networking if you treat it as a service for just that, and limit your connections to people whom you actually know.

If you already know these people, why do you need linkdink?

Re:Neglecting to care

[JaredOfEuropa]

For people whom I know but don't have an email address for (or their address might have changed)
To rediscover people I know but forgot about (plenty of those). I could ask for and save businesscards from every person I meet but LinkedIn is just that much handier.
To keep track of people; I don't have contact with every single person in my network on a regular basis, and it's handy to know if they change jobs or switch companies. That's one of the good things of social networks: you don't have to keep your contact list up to date, your contacts will do that for you.

Re:Neglecting to care

[BarbaraHudson]

There's this really neat thing called a "search engine". If they're a professional, they'll be searchable. And if you're using linkdink to find/get friends, you're doing it wrong. It's spelled Facebook.

Re:Neglecting to care

[angel'o'sphere]

Sorry, Barbara, but at age of nearly 50 I certainly don't use facebook to find friends. How actually should that work? (I only have friends on FB that I know in RL ... like it should be)

A professional social site like LinkedIn or Xing is much better suited for that, or I use a true dating site.

Re:Neglecting to care

[BarbaraHudson]

I only have friends on FB that I know in RL ... like it should be

Why? That would be like saying I only have friends on slashdot who I know in RL ... like it should be. There are plenty of people here I think would make great RL friends.

Re:Neglecting to care

[Dahamma]

There's this really neat thing called a "search engine". If they're a professional, they'll be searchable.

I almost wanted to agree with your pervious comment, but this one was silly.

As much as I can't stand Linkedin, if you are looking for a "search engine" optimized for professionals, there is nothing better. In fact often if you search Google for someone in a professional context Linkedin is the first result that comes up (which I try to make a point of not clicking on).

Just because you dislike something, doesn't mean it's not the best at what it does. Maybe not the right solution for you or me due to all of its other flaws, but definitely the most efficient way to "search" for professionals in many industries. Which explains both it's huge successes (it's effective) and recent financial problems (it's shady as hell in some ways, and people are fleeing it).

Re:Neglecting to care

[BarbaraHudson]

I was pointing out the fact that you do not need LinkedIn to find a particular person. And given that many professionals aren't on linkedin, you're immediately limiting your chances of success if that's what you use. Why start with one hand tied behind your back, especially when you can get information from a general search that you won't get from a carefully massaged profile on linkedin?

Re:Neglecting to care

[BarbaraHudson]

Well, you're sure welcome to be the first. I'll even post the video on youtube. :-)

Does their latest app still take all my contacts?

[Anonymous Coward]

Does their latest app still take all my contacts? If so then LinkedIn can go fuck themselves.

Open sourcing or outsourcing?

[misnohmer]

Sounds more like they are "outsourcing QA to the end users".

Re:testing frameworks

[Dahamma]

Once my realtor endorsed me for embedded systems development I gave up and deleted my account (if that's even possible, who knows).

Verification and validation

[DeBaas]

First of all, thank you Linkedin for open sourcing this! Always good to share.

First, three hours is not enough time to conduct any manual testing steps, so holding ourselves to this constraint ensures we won’t revert to using manual validation to certify our releases.

I've been in testing for some time and have been taught to make a distinction between verification and validation.
Verification is checking if the software works according to specs. Validation means: does it actually work for us. By defintion that means that you can automate verification but not validation.
Is that just semantics? Not for testers. In the test community there currently is a big debate going on checking vs testing.
See i.e. Michael Boltons blog. . Checking can be done automated. Real validation in my opinion can not.

What I am curious about is the 'to production each three hours' That sounds great, but although I don't use a linkedin app on my phone, I am still pretty sure users don't get their app update three times a day.. With such rapid deployment, I suspect it takes multiple deploments before it adds up to a significant increase in usable functionality.

Many small releases means in general that mistakes are also small and quickly fixed. I am actually in favour of them. But it is not a full garantuee that one of these small releases will not break something badly that would have been found by even a limited manual test. The chance that that happens may be much lower with small releases but it still exists and the impact is still high.

Automated tests can perform a huge amount of checks quickly. Humans can't beat that. But they can also overlook the blatanly obvious. I would hope they would have manual testing at least prior to releasing new functionality. To find these things, but also to do some validation by the definition above.
Else I suspect it may work well for them for quite some time but it may bite them badly at one point as well.

 

Re:Verification and validation

[AngryLibertarian]

Layers and layers. Many small batches do not prevent defects, they make them faster to find and correct. You are correct that many small releases are required to complete a feature, but that means early feedback from E2E test and requestor testing if that was the right future anyway so that you can stop working on a bad idea. Just because a feature is requested, it does not follow that it should be blindly implemented.

Re:Senior Engineer?

[angel'o'sphere]

Hm ... I guess those guys think different: https://www.sei.cmu.edu/cmmi/

Also, what is so hard to grasp that there is no "engineer"?

There are "mechanical engineers", "electrical engineers" and others and in the end also "software engineers", I for my part actually studied "software engineering" and "computer science" ... funnily I should have worded it different, I studied "computer science" and "software engineering" is a important but depending on what else you study: a small sub section of CS.

I don't care what kind of engineer you are, that you seem so pissed, in real live I'm half an "Requirements Engineer", wow ... another engineer ...

My sister endorsed me for something.

[tlambert]

My sister endorsed me for something.

I made her delete the endorsement, on penalty of being delisted as a connection on LinkedIn if she did not do so, since there was no way in hell that she could legitimately claim domain knowledge for the endorsement, or lacking that, experience in an organization we were both in, justifying the endorsement on the basis of my past work.

This is the problem with associating with family members on professional, as opposed to things like Facebook, social networks: they think they are helping you, when really they are not.

While there are several organizations where we both worked simultaneously, they were all in the distant path. If she wanted to endorse me as an electrician or a telephone installer, or a small appliance repair person: yes, she has that domain knowledge. But that's not where I typically work these days, and it's not right for people to comment on things they don't know that way.

Probably, you should have just given your realtor the same speech I gave my sister, and in the limit, asked them to retract or to lose their connection to you, rather than just deleting your account.

Re:My sister endorsed me for something.

[tlambert]

Anyone who is going to be so fucking stupid as to think worse of you simply for a family member giving you a silly endorsement on LinkedIn is a pretentious dumbshit anyway.

Or they just are going to take everything you say with a grain of salt, if it's pretty clear you are getting your family members to inflate your endorsement numbers. That's not being a pretentious dumbshit, that's simply not being a dumbshit, period. Either I can trust the numbers, because I can trust the people endorsing you, or I can't trust the numbers.

I'm pretty sure that if you're applying for a position as biologist, it's statistically improbable that all the members of your immediate and extended family are also biologists, and therefore qualified to judge your ability, or have an opinion. In which case, I will scale my opinion of your ability (down) accordingly.

Re:My sister endorsed me for something.

[KGIII]

Yeah, I don't know how the site works - having never needed to make use of it, but I do know that I'd look at *who* was endorsing whom and what they had for qualifications to assert an opinion on the abilities. I may be able to say that my mechanic, barber, or housekeeper do fine work but, no matter how much I may like you as a person, I can't not say that you're a quality programmer, chemist, biologist, or airplane pilot.

So, were I to use the site, I'd weight opinions and recommendations carefully. Lots of people don't seem inclined to do that sort of thing - with other things, at least. There's an adage about considering the source of comments, opinions, and such. It's a valid adage and a good thing to do. The number of endorsements is insignificant compared to the quality of those who are doing the endorsing. If they've got 39 people saying they're excellent and none of those people are actually in the field but they have 1 person who's saying they're not so very good but they are a recognized expert in the field then you can bet that I'm going to weigh that one endorsement much heavier than I will the endorsements from the 39 people.

It's possible that they're a great person but still suck.

Re:My sister endorsed me for something.

[Dahamma]

Sounds like you are blaming your sister for her endorsement, when obviously it's Linkedin that should be blamed. I have no interest in giving speeches or threats to friends over a shitty service like this.

The reason so many of these strange endorsements happen is that Linkedin makes it as confusing as possible for people. She probably saw something like "do you know X, and think he does great work? Would you like to endorse him?" - basically making you feel bad for not doing it without decent context. Who knows, they may have improved it more recently as it's been one of the major complaints - but I wouldn't know since I haven't used it for years...

Anyway, that's not the only reason I quit Linkedin. The biggest one is that the more you pay for their services, the more "semi private" information they will give you about other people. ie. unlike Facebook, etc, if you pay enough to Linkedin you can get access to people who aren't in your network. Not only that, if you pay more you can see who has looked at your page, etc. I know Linkedin has added some (buried) settings to limit that a bit, but in the end their business model is built on this so clearly they do NOT have their (non paying) users' best interests in mind.

Re:testing frameworks

[JaredOfEuropa]

Meh. Some endorsements I got on LinkedIn make sense, others don't. The more useful ones are from people who took the trouble to write a short recommendation for me instead of treating endorsements as "like" buttons. And this has been my experience with my yearly appraisals as well, back when I was an employee. Most managers just tick a few boxes while only a few of them take it seriously.

Re:How do they handle database changes?

[Hognoxious]

They use MongoDB. You don't need to worry about database changes, because it's webscale.

puff piece

[Hognoxious]

The link doesn't go to anything specific, which leaves me puzzled as to whether this is a framework as in rails or something sort of like a procedure but more conceptual.

The content is buzzword laden twaddle. I reckon the whole thing is just a puff piece.

Re: puff piece

[AngryLibertarian]

Yep, a link to something more descriptive would be more useful.

Re: Senior Engineer?

[Entrope]

I think you're mostly agreeing with the OP. The SEI will certainly tell you that most software development is not done with enough rigor to be called engineering. In large part, that's because most software people do not know how (or do not bother) to elicit and manage requirements well -- so cheers to you for doing that. (Good verification skills and planning for traceability are the other places that typical software projects fall short, IMO.)

Re: How do they handle database changes?

[AngryLibertarian]

Evolutionary database change takes planing. Rapid deploy doesn't mean there's no thought given to planning outcomes.

250 Engineers?

[110010001000]

There is no project on Earth that requires 250 developers. And yeah, the idea of continuos releases and automated testing if very common in the industry. That is what git/jenkins/buildbot does best.

Re:Senior Engineer?

[BarbaraHudson]

Any place that still puts Grady Booch on parade is "process over product", same as UML was, which is why UML failed so spectacularly in not being the magic bullet that everyone said it would be.

Re:Does their latest app still take all my contact

[BarbaraHudson]

Does their latest app still take all my contacts? If so then LinkedIn can go fuck themselves.

Too late - they've already done a double plus fine job of fscking themselves. Then again, considering how many of the users are just engaging in circle jerks trying to get recommendations ...

Re:I wish linkedin would die already

[BarbaraHudson]

Don't worry - the magic words "We're open-sourcing $X" means they're looking to cut staff. This is what most businesses without a clue do, not realizing that in fact they will need to hire more staff to deal with it.

Re:Senior Engineer?

[angel'o'sphere]

Hm, is your country regarding "marketing" or "advertising" really so bad?

Stuff like this which is why XYZ failed so spectacularly in not being the magic bullet that everyone said it would be. I likely have read on /. a 1000 times.

However I never experienced that something was marketing-ed to me in RL in such a way nor did I "market" e.g. UML when I was still an UML coach in that way.

BTW: what is wrong with UML in your opinion? I use it since roughly 20 years, even when it was not called UML but OMT and Use Cases ;D

If anything can be blamed is that basically all UML tools are shit ... but that is partly because of the languages they need to support. When OMT/Booch Method and other stuff was born, "The Gurus" thought SmallTalk and perhaps Lisp would be the language of choice. With stuff like C++ it got pretty ugly ... with C#/Java it is finally a bit more "straight forward" again but now people are doing DSLs instead of graphical modeling.

so much engineering for such a small company

[AnAlchemist]

Is it just me, or does LinkedIn do an amazing amount of engineering for a website that's basically a simplified Facebook + job board?

Do they really need this amount of engineering? Or are they doing it just for the sake of doing it?

Re:Senior Engineer?

[BarbaraHudson]

UML is one further abstraction from the code - pseudo-code is far, far better. The problem is that people can't "design" systems just by drawing pictures. If you need to draw pictures, use a flow chart. Simple is always better.

Re:testing frameworks

[Dahamma]

Meh. Some endorsements I got on LinkedIn make sense, others don't./quote

Which makes it sound pretty "random" - ie. totally useless as a metric for hiring decisions.

Re:I wish linkedin would die already

[BarbaraHudson]

Don't worry - the magic words "We're open-sourcing $X" means they're looking to cut staff. This is what most businesses without a clue do, not realizing that in fact they will need to hire more staff to deal with it.

Interesting, could you please elaborate? What does open-sourcing a product have to do with cutting staff? Do you mean they will rely on the formation of a community of volunteers to maintain their product and let their in-house developers go?! That is pretty insane.

No wonder you post anonymously - you must be REALLY new here. Whether it's insane or not is irrelevant - this is management decisions we're talking about, logic is optional.