Deserialization Issues Also Affect Ruby -- Not Just Java, PHP, and .NET (zdnet.com)

An anonymous reader writes: The Ruby programming language is impacted by a similar "deserialization issue" that has affected and wreaked havoc in the Java ecosystem in 2016; an issue that later also proved to be a problem for .NET and PHP applications as well. Researchers published proof-of-concept code this week showing how to exploit serialization/deserialization operations supported by the built-in features of the Ruby programming language itself.

"Versions 2.0 to 2.5 are affected," researchers said. "There is a lot of opportunity for future work including having the technique cover Ruby versions 1.8 and 1.9 as well as covering instances where the Ruby process is invoked with the command line argument --disable-all," the elttam team added. "Alternate Ruby implementations such as JRuby and Rubinius could also be investigated."

The deserialization issues can be used for remote code execution and taking over vulnerable servers. While .NET and PHP were affected, it was Java until now that has faced the biggest issues with deserialization, earlier this year, Oracle announcing it was dropping deserialization support from the Java language's standard package.

Comments

Deserialization

[Anonymous Coward]

Sun really did a disservice with their serialization API. Developers think serialization is magic. I worked with several clients who needed custom serialization/deserialization. They usually believed that custom deserialization had to use the Java APIs but that is untrue. A good development framework can handle weak references, transience, etc. most freeware serialization frameworks are not sophisticated or reliable.

Re:Deserialization

[gweihir]

Most secure coding these days still boils down to "you have to know what you are doing". I don't see that changing anytime soon. Brooks famous statement "There is no silver bullet" still applies and will continue to apply for a long, long time.

Is that anything like getting your hair wet?

[Anonymous Coward]

Because, if so, that's a big deal. Very big. The biggest.

Dropping support?!?

[Anonymous Coward]

XML was supposed to fix all this!

And we all know, XML is like violence: if it doesn't fix the problem you're not using enough of it.

Re: Dropping support?!?

[Anonymous Coward]

Still waiting for a decent soltion from Oracle

Re:Dropping support?!?

[gweihir]

If XML has been designed by people that actually understand security, if _could_ have fixed the issue. But all the security-ignorant masses of developers ever want is features, features and more features. They would probably have rejected an actually secure XML.

Re: Dropping support?!?

[phantomfive]

The problem has nothing to do with XML. It is a problem with serialization. Even if you use json or yaml or whatever, there are still security issues.

That's what you get for being lazy.

[Qbertino]

Serialisation and deserialisation happens when developers get lazy and/or the original architects of the system designed a shitty appmodel. Or none at all. You see this nice and clearly in PHP CMSes such as Expression Engine or WordPress.

It goes like this:
Check out the model, see bunch of crap, think: "Oh I know, I'll just serialize my stuff and dump it into a single field." Newer stuff in WP is full of this and it doesn't help that this is tacked on to a baaad application model with some really shitty DBAL mechanisms that quickly grow to 2-digit amounts of SQL statements being executed per API call and an ERD designed on crack.

The truth of the matter is: If you don't take total control of your data every step of the way you are bound to be screwed when an exploit like this crops up. Simply serializing is the exact opposite of taking control. And taking control is basically impossible if you don't know how to design your app or its DB.

Whenever I see serialized data lying around in persistence, I know that someone further up didn't do his job.

My 2 eurocents.

Re:That's what you get for being lazy.

[Anonymous Coward]

Blame lazy toolkit and language providers!

Maybe COBOL had it right??

Re:That's what you get for being lazy.

[Anonymous Coward]

And maybe the elders of COBOL were right. I mean, in regard to cylons I mean.

Re:That's what you get for being lazy.

[gweihir]

I don't think it is laziness. I think it is plain incompetence. These people cannot really code and hence they need all these features to get anything to work at all. Of course, they also do not know how things actually work under the hood and they have zero understanding of security, nicely leading to the mess we can observe almost everywhere.

But then, if you get competent coders, you may actually have to pay them like the qualified engineers they are. "Management" cannot have that. A coder may actually earn _more_ than a manager with this! The sky would be falling!

Re: That's what you get for being lazy.

[Anonymous Coward]

As long as they are black and trans, who cares?

Re:That's what you get for being lazy.

[Luthair]

I've been attempting to the write this and not sound like a jerk.... but serialization simply means translating whats in memory into a format that can be stored. Even the scenario you're complaining about isn't necessarily "bad", it sounds like they're using it as an alternative to disk storage and as long as they aren't running queries on the contents of the field that isn't a problem.

The issue that Java (and presumably Ruby, though I don't care enough about Ruby to check) is that it turned out to be possible to craft serialized objects that simply deserializing would cause code execution. In the case of Java most development had long since switched to using other formats instead of native binary serialization before the vulnerabilities were discovered but as there are a ton of legacy applications and frameworks people still had problems.

Re:That's what you get for being lazy.

[iggymanz]

nonsense

moving a program from one place to another can face identical issues.

the path is secure, or it is not secure.

serialization irrelevant.

Re:That's what you get for being lazy.

[Anonymous Coward]

I have been ridiculed by co-workers for writing my own serializes and deserializers, when the .net-provided ones worked great with only two lines of code.

The .net serializes burned us when we needed to move the class definitions to a different namespace. Not only did mine not have this problem, but the end results were a mere tenth the size of the .net serialized objects, with no compression. And, of course, mine didn't have any code execution exploits.

Re:That's what you get for being lazy.

[iggymanz]

wrong

serialization of an object is very useful.

the path it moves on must be secure. if it is not secure, then you can have the same issues moving information or code by other means.

serialization not the problem if the path is compromised.

Re:That's what you get for being lazy.

[sphealey]

Most of what I have seen is (1) lack of understanding of what databases are, what they were designed to do, and what they are capable of (2) fear of using databases based on a shared cultural non-understanding stemming from 1 (3) broken workaround after broken workaround to allow the architect and developer... not to use databases.

Re:That's what you get for being lazy.

[Actually, I do RTFA]

Using serialization operations (properly to a non-binary format) makes things more resilient to changes. You can roll your own, but then again, that seems more likely to be prone to errors. NIH-syndrome is a bitch.

Re: That's what you get for being lazy.

[phantomfive]

If you think native binary serialization is the problem then you are ignorant (either that or you expressed yourself in a way I didn't understand). It doesn't matter if you use binary, or XML, or Jason or yaml, these vulnerabilities are going to be there. It's not just Java either, it's basically any language that does deserialization automatically on untrusted data. The reason is simple: untrusted data needs to be sanitized and treated as dangerous every step of the way. Sanitizing can't be done in an automated way without knowing how the data is going to be used. For example, something that is going into a database needs to be sanitized in a different way than something going into an excel form, and differently than something going onto a web page. Furthermore, you need to always treat untrusted data as untrusted, because when your application grows and it gets used I'm a context you didn't originally expect, you will need to purify it again (this is a well know excel exploit).

Re:That's what you get for being lazy.

[angel'o'sphere]

Whenever I see serialized data lying around in persistence, I know that someone further up didn't do his job.
You mean inside of a database?

There is nothing wrong with serialization. Use it when it is appropriated, don't use it if not.

My 2 eurocents.

Luckily a bit more worth than 2 US cents :D but your opinion: nope.

Re:That's what you get for being lazy.

[angel'o'sphere]

have zero understanding of security
Unless the programmer of the deserialization code did not plant an easter egg, aka a trojan, into the deserialization code: there is no security issue at all!

WTF ... what is next? An SQL select from a database is a security issue?

Re:That's what you get for being lazy.

[angel'o'sphere]

is that it turned out to be possible to craft serialized objects that simply deserializing would cause code execution.
Actually it does not. Unless the writer of the relevant classes deliberately put some special "deserialize()" methods into the classes on the server. Exploit from outside is completely impossible unless a programmer deliberately put in a back door.

Re: That's what you get for being lazy.

[Anonymous Coward]

if you use binary, or XML, or Jason or yaml, these vulnerabilities are going to be there

Have to disagree with you here. If you're using XML or JSON or yaml you are simply translating the state of some artifact into a textual representation of that artifact. That text can do absolutely nothing without a piece of executable code to interpret it. JSON has always seemed a particular danger to me as it is designed to allow easy evaluation directly from JSON string and it will actually invoke function calls if the string contains those instructions. But again, it is your application code that is calling the evaluate function.

As you say, data going into a database needs to be sanitized. If you blindly take the textual data coming in from deserialization and simply use it in an SQL statement, it might be instructions the database will execute. That is not a vulnerability of the format, that is a vulnerability of the consumer of the format, meaning it is the fault of the application developer, not the serialization format.

Java's binary serialization was different in that it worked more like JSON and could be executed. But because of the many issues serialized objects had, such as dealing with different versions of the JVM, few people made use of it. Even some early attempts at actor models where executable code could be sent from one JVM space to another didn't really take off. But taking a Java object tree and serializing that to XML does not introduce a vulnerability because what is being saved is the textual representation of the state of the object, not class files, not object code, just UTF-8 text strings or hexadecimal versions of bytes.

Unless/until your application executes that content without proper controls, there isn't a vulnerability. And the first rule of any defensive programming is never trust input data, always cleanse and/or encode, but never execute blindly.

Re:That's what you get for being lazy.

[Anonymous Coward]

... but serialization simply means translating whats in memory into a format that can be stored.

Stuff in memory is stored . . . in memory. Hence it is already stored. Use the same format on disk, and no "serialization" will be needed. An object is an aggregate of simpler types (ints and floats of various sizes) and can all be stored as such.

Or you can go the hard way and "transform" stuff whenever it moves around. Watch your app getting outperformed by those that doesn't. 1000GB could be 1000 gigabyte-sized objects or a million megabyte-sized objects or aa billion kilobyte-sized objects. With no transformations, all cases needs 1000s to move across a 10Gbps link. Copying a file takes the same time, no matter if it is one big object or a billion small objects inside. The same applies to loading it into memory - it happens at disk speed if you don't have this artifical need to mess with every object as it comes in. Pull in a large array with a single read() issued to the os.

Re:That's what you get for being lazy.

[TechyImmigrant]

Most of what I have seen is (1) lack of understanding of what databases are, what they were designed to do, and what they are capable of (2) fear of using databases based on a shared cultural non-understanding stemming from 1 (3) broken workaround after broken workaround to allow the architect and developer... not to use databases.

That's because databases are heavy lifting for light problems. Avoidance is rational.

It is only a problem if

[Anonymous Coward]

It is only a problem if you are careless. I always checksum the serialized data, then encrypt it and prepend the checksum, then checksum the result, and if any of it doesn't check out, I don't even bother.

Nothing's wrong with serialization and deserialization, it is good old incompetence that corrupts everything.

Re:It is only a problem if

[Anonymous Coward]

Didn't word it property. There is "data" which becomes "checksum of data + data" which is then encrypted and becomes "checksum of encrypted data + encrypted data". If the latter/first checksum doesn't check out, there was a read or transmission error. That can still be meddled with tho. Which is why the former/second checksum is encrypted with the data. Good luck tampering with this one. A scheme that has never failed me.

Re:It is only a problem if

[gweihir]

The problem with deserialization is that it gives tools to fools and makes them more dangerous fools as a consequence.

In the hands of an expert, it is just a feature that is nice to have. But most people writing software are not experts.

Re:It is only a problem if

[Anonymous Coward]

Since we're talking about exploits, how does that stop deliberately created bad data? Are you relying on the obscurity of which type of encryption and checksums you are using?

Re:It is only a problem if

[Anonymous Coward]

I think he wrote about some sort of complex Cookie. He serializes session state into cookie, adds some sort of Hash MAC and sends it to the client. The HMAC will protect the serialized data against modification.

BUT - that is one narrow use case of serialization and the really dangerous case is when you get serialized data from a client program, which can be modified by the attacker. Then you cannot HMAC the serialized data, because the attacker has it before you have it.

Serialization in Client-Server computing is something very much like Gotos or retroactive parallelization of large chunks of badly understood programs.

Its a shortcut to a proper grammar for your input data and for a strong lexer, parser and semantic checker. A shortcut for a proper CS degree.

Re:It is only a problem if

[Anonymous Coward]

Yeah, my attitude is you have to check any input data for consistency. If it's a graph, or list, or whatever, everything has to point to something, and all the data has to be valid for whatever you're doing with it. No different than any other input. You can't just assume because you calculated it once, that it's just like data that's been continuously inside your program.

Re:It is only a problem if

[HiThere]

You're assuming that you determine the inputs. Actually there are ways around it, but they all involve eliminating lots of kinds of things being serialized or deserialized. YAML is safe. Python's ast.literal_eval is safe. Various others. But they don't allow arbitrary things to be de-serialized. And Ruby also has safe ways to handle this. (YAML is available in Ruby.)

But you need to realize that serialization isn't magic, and validate your imports before executing them. So arbitrary binaries can only be deserialized as byte arrays, or something essentially similar.

Re:It is only a problem if

[angel'o'sphere]

I serialize some objects to a file.

You deserialize them, recreating "similar" objects in memory.

What exactly is the security risk?

Sorry, gweihir: you have no clue

But I answer it for you: there is none. There is no fucking difference if you read a text file I wrote (which is called serialization, too) or a serialized object format.

Developers aren't the only ones

[Anonymous Coward]

"Magic" is how XML was supposed to fix all encoding (and therefore, de/ser) problems.

Of course it didn't. So the same "developers" came up with a new trick: JSON. It's simpler. But it's still a crock. Built on the same premise, in fact.

The problem is a lot deeper, and you and your clients are only scratching the surface. (As am I, but I actually like this sort of bit twiddling.) If you're thinking in frameworks, you're doing it wrong. You actually positively need to pick apart and also properly generate a specific wire format? Then you do it all yourself and all you get is the basic read() and write(). Or perhaps you use buffered input and you get to use getchar(). But again that's all you're getting.

The rest you bring youself, which must include both sides and some scaffolding to run automated testing. Probably including pumping gigabytes of random data through the thing and checking what comes out is the same as what came in.

I know nothing about sun's api, but this isn't (just) on sun's head. It's on everyone who just plain doesn't understand that input and output means data is crossing a border (outside: not your program, so not your jurisdiction, and inside: your program, your responsibility) and you need to be careful about what comes in and what goes out. "XML" is a prime example of the "oh! I know! We'll cook up a framework!" non-thinking that inevitably leads to fscking things up royally. But it's a very popular pastime with certian... people with a habit of "magical framework thinking". In kids it's cute. In webmonkeys, not so much.

Then again, this is a rather specialist topic. You may find you really do need specialists to do it and not let the rest of the code monkeys near the innards of the de/ser apparatus. The problem, of course, is that you can't have a "universal" de/ser thingamajig (witness all the RPC things that invariably have their own), and you shouldn't rely on your compiler writers to provide it to you. Not even google: As cute as "packet buffers" is, it still sucks in very google-y interesting ways.

Re: Developers aren't the only ones

[Anonymous Coward]

I also run into quite a few people saying we should use a standard tool for, for instance, serialization, where I wrote my own.

I am saying: well you will probably end up many more hours fighting your magic tool, shoehorning it into our framework, not doing quite the right thing, than debugging a few hundred line of code optimized to our needs.

Often you just pick some standard tool, but often the task is so small that it is more efficient to write your own and stay in control.

If you want your system to communicate with other, unknown, people's systems either choose a complete standard protocol, or make such a simple protocol, that any software guy can implement it. Please, don't build on top of some 3rd party serialization library: you have no clue if those other system can use that library for technical or legal reasons.

Seriously?

[Anonymous Coward]

How is this news?

If you don't verify your data and rely on built in functions to automagically do it for you, you're going to have a bad time.

Re:Seriously?

[gweihir]

The news here is that we have _still_ not managed to assure any level of real skill in the people writing software. This cannot continue.

Re:Seriously?

[Anonymous Coward]

The news is that Ruby itself, without any additional third-party libraries, exposes you to attacks.

Re:Seriously?

[Narcocide]

The news is that Ruby itself, without any additional third-party libraries, exposes you to attacks that it was designed to prevent, which were fixed in PHP years ago. But importantly, back then even when it was known to be unsafe, PHP warned against using it in unsafe ways. Ruby on the other hand said "don't worry we got your back" but apparently actually didn't.

Re:Seriously?

[Tyger-ZA]

The news here is that we have _still_ not managed to assure any level of real skill in the people writing software. This cannot continue.

Correct, but to elaborate: The news here is that some framework vendors have provided a feature to unsuspecting developers of varying levels of ability, these unsuspecting developers are using the feature that can be reasonably expected to have been tested thoroughly by said vendor. Meanwhile, some of these vendors have written the same feature badly, and these unsuspecting developers who definitely don't have the time to code review the entire framework, and likely don't have access to the source and/or don't know what sort of problems to be looking for anyway, are blindly trusting the vendor to provide quality code.

Part of the problem is having several vendors all competing for developer mind share, providing their own abstractions on top of how_a_computer_actually_works, and also providing lock in so that the developer finds it hard or impossible to get rid of the framework, while another part of the problem is that some developers depend on their framework training wheels and never learn how to work without it (web development is especially rife with this)

It boils down to

[jabberw0k]

• Data read from a file or network is only as secure as that file or network.
• There are reasons why the Unix Way abolished binary data files wherever possible in favor of plain text files; read The Art of UNIX Programming by ESRaymond.

Re:It boils down to

[Anonymous Coward]

Actually it is not a matter of binary vs. ASCII text.

In both approaches you need a syntax, a grammar and a semantic checker. All must be well-defined. Then you have a chance at being secure. Also, to do this properly, you need someone with a CS degree. Self-taught folks with an inflated ego are not the right people.

I guess we will have this in the year 2500 or so. There are no consequences for shitty security on the exec level. No execs jailed. The worst they get is a golden parachute.

Re:It boils down to

[Anonymous Coward]

CS is not a religion. I've known people with CS degrees who were dummer'n rocks when it came to making things work.

Self-taught folks with an inflated ego are not the right people.

As opposed to an AC on /.

Re:It boils down to

[Eravnrekaree]

Binary formats can be perfectly safe, it takes a competent person to write the processors for them. They are not more difficult to write than one for a text based format, so its not like text based formats are safer. They can save some CPU cycles since using length field demarcation rather than demarcation characters, you dont have to inspect every single character.

Re:It boils down to

[Cesare Ferrari]

Agreed. The simplest binary formats are very simple to store and retrieve, if you think about it, you just create a struct with non-pointer/reference types, and write your fields. Serialisation is a write of the memory to a file, deserialisation is a read. All in all, it'll be 10 lines of blindingly obvious code without any chance of programming errors, buffer overruns etc.

However, and this is a big however, it's completely inflexible, and has a number of massive downsides. It's going to be endian specific, it can't represent complex relationships between objects, it's not going to have any versioning support, the list goes on.

Once you start trying to address the above issues, you end up realising the problem is a bit larger than you thought, and end up re-creating something like google's Protocol Buffers, or Real Logic's Simple Binary Encoding, so you are better off using an established framework which then involves understanding how these work. At this point, you've kind of added back the complexity of the text based serialisation/deserialisation approach, so API wise it's not a great win. Performance wise though, something like SBE is leaps ahead of native serialisation if you don't need the flexibility that native serialisation supports (with it's polymorphism etc). Do check out SBE if you are wanting a lightish and performant mechanism, it's worked well for me (https://github.com/real-logic/simple-binary-encoding)

Seem to have gone a little away from the original point, apologies.

Re:It boils down to

[Anonymous Coward]

This isn't a text versus binary issue. This is an "untrusted data being passed off to a magic method" issue.

While the actual example uses the Marshal class to instantiate objects, I'll bet you could pull off something similar using YAML, which is a text-based object serialization format. I don't know the exact deserialization process used when loading objects via YAML (and their documentation helpfully doesn't explain at all), so the specific method they used may not work, but it seems quite possible.

The problem is that magic serialization schemes will always be vulnerable, because they're "magic:" the resulting data is opaque, and the deserialization process is usually a one-shot "load the entire object graph" method. Because you can't tell these deserializers what you're expecting (because they work on everything) you can't tell them not to load a malicious object. While a text format may make scanning the object graph for malicious data ahead of time easier, an even better solution would be to not use magic serialization at all.

No

[Anonymous Coward]

Most developers are surrounded by non-IT people who consider them geniuses. So the "genius"of course thinks he is also a security expert. After all, he knows how to devise a strong password, what else could be hard ?

Re:No

[epyT-R]

Most of the developers I've met are clueless about systems and security..and the systems people I've met know very little about software design. The ones who have at least some experience in both tend to be the brightest at learning more about either.

Re:No

[Anonymous Coward]

I know both, but for years selling that combination was nigh impossible.

Now, "devops" is a thing but I lack the requisite RECENT ten years of experience in five years old "technologies" to be considered and have holes in my employment history big enough to guarantee never to get hired again.

Thanks, HR. Yeah, "we" have issues with getting the right person employed in the right position. Me, I stopped trying and decided my life goal is to hang around on the dole until I die, writing bitter comments on slashdot. I call it aiming for attainable goals. The upshot is that I do meet those goals. What more could I possibly want?

Liberats

[Anonymous Coward]

...think you should never limit anything by formal rules. Every social scientist should be able to program the control units for your ABS brake.

If you die, your wife and 14 year old son can avenge you in the wonderful world of Liberats.

Re:Liberats

[gweihir]

And there you nicely show that you are part of the problem, because you completely misidentified the issue.

Programming is about automation

[lucasnate1]

This has nothing do with serialization, which is just as much as serialization when written by hand. This has something to do with trusting inputs. One should not trust input that the user can control. If one only saves internal records to disk, automated serialization is welcome.

And in general, it would be good if programmers stop hating processes for automating code. The whole point of coding, and programming, is automating things. If you insist on doing things by hand, why not try your hands at doing construction work or something that actually CAN'T be automated?

Re:The Donald

[Anonymous Coward]

Even leftists voted for Trump, because Hillary stood only for two things : identity politics and war.
I say this as a Euro wimp, redder than the USSR flag and as we pick up degeneracy a decade or less after the US does : before the cunt-in-chief, I didn't think anti-white racism was a real thing. That got worse after the election.
Let's remember what the cunt-in-chief shares with Obama : responsibility for the deaths in Libya, Syria and Yemen, as well as the migration crises and the bombings and attacks in Europe. But no she has moral superiority because vote for me I have a cunt. The equivalent would have been to see Hitler's small dick and declare him a saint and celebrate him because it's proof he was a victim.
Also, US presidents always make sure to kill one million, maybe two millions max (or three in earlier times) because if they get to six millions some people will notice and then someone will lose their ultra supreme six million victim status.