Long time ago, but by far the worst for me was when I inherited some code that a previous programmer had done. Every variable was a breakfast item. So if biscuit>bacon then scrambledeggs=10. Shit like that. It was a nightmare and luckily I only had to deal with it infrequently.

One time, I had to request firewall access for a machine we were deploying to, and they had an Excel sheet to fill in your request. Not great, I figured, but whatever.

Then I asked who to send the Excel file to and they told me to open a pull request against a Git repo.
And then, with full pride, the guy tells me that they have an Ansible script, which reads the Excel files during deployment and rolls out the firewall rules as specified.

In effect, this meant:

1. Of course, I had specified the values in the wrong format. It was just plaintext fields in that Excel, with no hint as to how to format them.
2. We did have to go back and forth a few times, because their deployment would fail from the wrong format.
3. Every time I changed something, they had to check that I’m not giving myself overly broad access. And because it’s an Excel, they can’t really look at the diff. Every time, they have to open it and then maybe use the Excel version history to know what changed? I have no idea how they actually made that workable.

Yeah, the whole time I was thinking, please just let me edit an Ansible inventory file instead. I get that they have non-technical users, but believe it or not, it does not actually make it simpler, if you expose the same technical fields in a spreadsheet and then still use a pull request workflow and everything…

I found code that calculated a single column in an HTML table. It was “last record created on”.

The algorithm was basically:

foreach account group
  foreach account in each account group
    foreach record in account.records
      if record.date > maxdate
        max = maxdate

It basically loaded every database record (the basic unit of record in this DATA COLLECTION SYSTEM) to find the newest one.

Customers couldn’t understand why the page took a minute to load.

It was easily replaced with a SQL query to get the max and it dropped down to a few ms.

The code was so hilariously stupid I left it commented out in the code so future developers could understand who built what they are maintaining.

I’ve had legacy systems that would encrypt user passwords, but also save the password confirmation field in plain text. There was a multitenent application that would allow front end clients to query across any table for any tenant, if you knew how to change a header. Oh and an API I discovered that would validate using “contains” for a pre-shared secret key. Basically if the secret key was “azh+37ukg”, you could send any single individual character like “z” and it would accept the request.

Shits focked out here, mate.

Java webapp. Customer facing. E-commerce application, so in PCI scope and dealt with credit card info and such.

There was one specific cookie that stored some site-wide preference for the customer. (Why not just put that preference in the database associated with the user? Because that would make too much sense is why.)

But the way they encoded the data to go into the cookie? Take the data, use the Java serialization framework (which is like Python’s “Pickle” or Go’s “Gob”) to turn that into a string. But that string has binary data in it and raw binary data is kindof weird to put in a cookie, so you base64 encode the result. (The base64 encoding was the only sane step in the whole process.) Then you do the reverse when you receive the cookie back from the browser. (And no, there was no signature check or anything.)

The thing about the Java serialization framework, though is that decoding back into Java objects runs arbitrary object constructors and such. As in, arbitrary code execution. And there’s no checking in the deserialization part of the Java serialization framework until your code tries to cast the object to whatever type you’re expecting. And by that point, the arbitrary code execution has already happened. In short, this left a gaping vulnerability that could easily have been used to extremely ill effect, like a payment information breach or some such.

So all a malicious user had to do to run arbitrary code on our application server was serialize something, base64 encode it, and then send it to our servers as a cookie value. (Insert nail biting here.)

When we found out that there was a severe vulnerability, I got the task of closing the hole. But the existing cookies had to continue to be honored. The boss wasn’t ok with just not honoring the old cookies and developing a new cookie format that didn’t involve the Java serialization framework.

So I went and learned enough about the internal workings of how the Java serialization framework turned a Java value into a binary blob to write custom code that worked for only the subset of the Java serialization format that we absolutely needed for this use case and no more. And my custom code did not allow for arbitrary code execution. It was weird and gross and I made sure to leave a great big comment talking about why we’d do such a thing. But it closed the vulnerability while still honoring all the existing cookies, making it so that customers didn’t lose the preference they’d set. I was proud of it, even though it was weird and gross.

The value that was serialized to put into the cookie? A single Java int. Not a big POJO of any sort. Just a single solitary integer. They could just as well have “serialized” it using base-10 rather than using the Java serialization framework plus base64.

A program that HR had built so that all employees could they their payment receipts online

The username was the companies’ email address, the password was a government personal id code that you can lookup online, a don’t change, and you can’t update the password to something else.

So I told the director of HR this was a bad idea. She told me I was overreacting until I showed her her own receipt, then she finally understood that this is a really fucking bad idea.

Okay, so now she out me in charge of debugging that program.

So I setup a meeting with the director of the company they hired, he came by with the developer: a 21 yo girl who I think hadn’t finished college yet. Great start! Apparently it was her idea to do the authentication like that so that explains a few things.

So we dive in to the code.

First of all, the “passwords” were stored in blank, no hashing, no encryption, nothing. That wasn’t the worst.

For the authentication she made a single query to check if the user email existed. Of that was true, then step two was a second query to see if the password existed. If that were true, the email had been authenticated.

So let’s say, hypothetically, that they had actual passwords that people could change… I could still login with the email from anyone, and then use MY OWN password to authenticate.

This just blew my mind so hard that I don’t think I ever fully recovered, I still need treatment. The stupidity hurts

Oh boy, this one was a doozy…

Was working at a very big company named after a rainforest on smart home products with integrations for a certain home assistant…

New feature was being built that integrates the aforementioned home assistant with customer’s printers so they can ask the assistant to print stuff for them.

The initial design lands from our partner team with a Java backend service fairly nicely integrated with some CUPS libraries for generating the final document to be sent to the customer’s printer. All good.

They are about to launch when… uh oh… the legal team notices an AGPL licensed package in one of the CUPS library’s dependencies that was absolutely required for the document format needed by the project and the launch is cancelled.

So the team goes off in a panic looking for alternatives to this library and can’t find any replacements. After a month or two they come back with their solution…

Instead of converting the document directly in the backend service with the linked CUPS library (as AGPL is a “forbidden license” at this company) the backend uploads the initial document to an S3 bucket, then builds a CUPS document conversion bash shell script using some random Java library, the shell script is then sent (raw) to a random blank AWS host that comes prepackaged with CUPS binaries installed (these hosts were not automated with CI/CD / auto updates as was usually mandated by company practice because updating them might remove the CUPS binaries, so they required a ton of manual maintenance over the service’s lifetime…), the bash shell script is then executed on that “clean” host, downloading the document from S3, converting it via the CUPS command line binary, then reuploading it to another S3 bucket where the Java backend picks it up and continues the process of working the document through the whole backend pipeline of various services until it got to the customer’s printer.

This seemed to satisfy the legal team at the very least, and I have no doubt is probably still in production today…

The kicker though? After all those months of dev work from a whole team (likely all on 6 figure salaries), and all the time spent by various engineers including myself on maintenance and upkeep on that solution after it was transferred to us?

An alternative, completely unrestricted corporate license was available for the package in question for about $100 per year so long as you negotiated it with the maintainers.

But that was a completely unacceptable and avoidable cost according to upper management…

There was a website where users could request something or other, like a PDF report. Users had a limited number of tokens per month.

The client would make a call to the backend and say how many tokens it was spending. The backend would then update their total, make the PDF, and send it.

Except this is stupid. First of all, if you told it you were spending -1 tokens, it would happily accept this and give you a free token along with your report.

Second of all, why is the client sending that at all? The client should just ask and the backend should figure out if they have enough credit or not.

Our CFO’s social security number, contact info, and just about everything you’d need to impersonate them inside a random shell script that was being passed around like drugs at a party for anyone to use. Oh and it had an API key to our payments processor hard coded into it.

That was the tip of the iceberg of how bad the systems were at the company. All of these are from the same company:

• A fintech based company with no billing team
• An event system that didn’t event
• A permissions system that didn’t administer permissions
• A local cache for authentication sessions. Which means that requests would intermittently fail auth because the session was only on one replica. If you hit any of the other ones, you’d get an unauthenticated error
• A metrics collection system that silently lost 90% of it’s data
• Constant outages due to poorly designed and implemented systems (and lack of metrics… hmmm)
• Everything when I joined was a single gigantic monolith that was so poorly implemented they had to run at least 3 different versions of it in different modes to serve different use cases (why the fuck did you make it a monolith then?!)
• The subscriptions system was something like 20 or 30 database tables. And they were polymorphic. No one could touch the system without it breaking or that person declaring failure, which leads me to …
• A database schema with over 350 tables, many of which were join tables that should have been on the original table (fuck you scala/java for the limitations to the number of fields you can have in a case class). Yes you read that right. Table A joined to table B just to fill in some extra data that was 1:1 with table A. Repeat that a few dozen times
• History tables. Not separate from the original table, but a table that contained the entire history of a given piece of data. The worst example was with those extraneous join tables I just mentioned. If you went and changed a toggle from true to false to true to false, you’d have 4 records in the same table. One for each of those small changes. You’d have to constantly try to figure out what the ‘latest’ version of the data was. Now try joining 5 tables together, all of them in this pattern.
• Scala… I could go on a tirade about how bad scala is but needless to say, how many different error handling mechanisms are there? Scala decided to mix all of them together in a blender and use them all together. Scala is just two white paper languages in a trenchcoat. Never use it in a production system
• A dashboard for “specialists” that was so easy to overwhelm that you could do it by breathing on it due to the LACK of events that it needed
• Passwords stored in plain text (admittedly this was in the systems of the company we acquired while I was there). Doesn’t matter if they were actually <insert algorithm here>, they were visible in a dashboard accessible by employees. Might as well have been plain text
• A payments system that leaked it’s state into a huge part of the rest of the system. The system ended up being bifurcated across two systems, I was brought in to try to clean up some of the mess after only a couple of months. I desperately tried to get some help because I couldn’t do it solo. They ended up giving me the worst engineer I’ve ever worked with in my 15 year career, and I’ve seen some bad engineers. Looking back, I’m reasonably confident he was shoving our codebase into an AI system (before it was approved/secured, so who knows who had access) and not capable of making changes himself. I could make several posts about this system on its own
• I could go on but I’ll cut it off there

I was told about a bug in a specific tool. It was being used in production. Apparently we’ve gotten a lot of complaints about it over the years, and they would complain if the site was actively used it always failed.

I couldn’t find it in the development branch in source control.

I asked if this tool was purchased from a third party. My boss, who was not a developer, said no. And he was very sure of that. But he didn’t know where the code was.

I was the developer with the most seniority, and I was there for less than a year at this point.

I looked again. I finally found it… In an unapproved pull request from a few years prior.

The meat of this tool basically took information to make an order and create an order in the system.

Any time we needed to insert a record, it would find the highest Id in the table, increment 1, and insert the new record, id and all. It did this for every entity that needed to be inserted. Address, customer… Everything.

Worse, very little validation was done. Want to order something but it’s out of stock? No problem, this tool just pushed it right through.

Want to ship something using a shipping method that doesn’t exist? You got it.

Want to ship something to an address that doesn’t exist? Sounds like the warehouse’s problem.

Knowing about the level of knowledge here, you know that there were no transactions. All sorts of unused records were left in the database when there was an error. The users would get a generic error and try again several times, too.

The worst part was, we have an already existing function that would take order information and try to make an order. And it would give you actionable errors!

XML-DOM page templates stored in a database, line by line.

So rendering a page started with:

select * from pages

where page_id = ‘index’

order by line_number asc;

Each line of XML from each record was appended into a single string. This string was then XSLT transformed to HTML, for every page load.

The architect sending a pointer over an API, in hexadecimal string format. char *c = “71E4F33B” just cast it on the right structure bro.

Just to add, we only did C/C++, on windows mfc, in a monolithic software.

I spent quite some time assuring myself that I was not the insane person before bringing it up with him.

I’ll consider myself lucky that the worst I’ve had to deal with was a 8K LOC C file that implemented image processing for a cancer detection algorithm. Nothing terribly tricky but just poorly organized. Almost no documentation at all. The only test was running this code against a data set of patient images and eyeballing the output. No version control other than cloning the project onto their NAS and naming it “v2” etc.

Research code can be really scary.

We had some super old code in our company monorepo that was written by someone who became the CTO, there was a comment forbidding people from writing private methods in the code base because “we aren’t babies”. It explained so much about the awful code and why everything was crazy.

A data ingestion service that was processing ~15 billion logs each day that was duplicating each of those logs 2-4 times in memory as a part of the filtering logic. No particular reason nor need to do it. When I profiled the system it was BY FAR the largest hog of CPU and memory.

The engineer who wrote it once argued with me about writing comparisons a == b vs b == a because one was technically more efficient … in a language we weren’t using.