A while back I was developing a workflow scheduler. However the run duration kept coming back negative. The runs kept finishing like 15 seconds before they started. Eventually I figured out why; the company had configured a policy to use their custom NTP server, but it was 16.7 seconds off. So my Windows laptop’s time was in the future, but the WSL VM (which ran the program) used another NTP server and was correct. In the end I just added a bodge that made the program request the time from an external server on startup to calculate an offset.

  • The Bard in GreenA
    link
    fedilink
    English
    arrow-up
    1
    ·
    25 minutes ago

    I’ve told this story before but we were supporting a custom ATS for a customer where we had this perfect storm happen.

    1. We had a bug in it that we sorted out eventually, where sometimes when you fed it input that contained weird characters, it would cause the name of a function or variable to get echoed in the nearby output. So if you had a variable called $debug_page or a function called $send_to_processing, and you input say some Arabic characters (this customer had many job applicants who were native Arabic speakers) you might get output that was the output you wanted + that string so

    Searching... 10 applicants found debug_page or Applicant: Mohammad Al-Saud send_to_processing.

    1. We had this developer who named things in lolcat speak. So a function might be called $ICanHaz($applicant) or $LongStringIsLong($string).

    So we got an email from the customer saying they had a VERY offended job applicant and what had happened. They had clicked the reject button on someone, which triggered the system to send an automatic rejection form letter to the applicant. This applicant had some Arabic character in their last name, which triggered the bug.

    And the reject applicant function had been written by this guy. So that poor applicant got an email that said

    Dear Applicant

    We have reviewed your application and while we thought you have an excellent background unfortunately we decided you are not a good fit for our company at this time… blah blah blah etc…

    YouCanHazNoJobKThanksBai

  • skisnow@lemmy.ca
    link
    fedilink
    English
    arrow-up
    5
    ·
    11 hours ago

    3rd line support for a global IT brand you have heard of, on the kind of security product that people don’t like to mess with too much once it’s installed on their servers.

    Bug had been in the tracker over 3 years when I joined, and had defeated anyone who tried it. Segfault that happened to only one customer, unpredictably, and only once or twice a year, and nobody could reproduce it.

    After a few weeks of tracing the compiled code line-by-line I found a race condition in the teardown of an object. That’s not the silly part.

    I wrote the patch for the version of the code the customer was running and was hailed as a mighty hero and everyone on the bus clapped etc. Then I went to forward-port the patch to the newer versions of the program and… it was already fixed there. Some fucken doofus had either fixed the bug without realizing, or, fixed it and just skipped the standard backporting process that would normally patch all the other versions that were still in support.

  • exdor@programming.dev
    link
    fedilink
    arrow-up
    3
    ·
    11 hours ago

    My system didnt boot for obscure reasons

    Found out something had caused my nix store (packages) to explode in size. Chrooting didnt work as it was filled to the last byte. Deleted some files, then chrooted in and did a cleanup

    • Buddahriffic@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 hour ago

      Or variations of this caused by a makefile not having correct dependencies set up, eg it points to dependent source files but not the headers so header updates get ignored during compiles. Made me add a “make clean” step to the debug process of “change that should have fixed the issue appears to be doing nothing at all” issues, or even just defaulting to rebuilding everything because test times can take hours or even days to get to the issue, so I don’t want to waste that much time only to find out it’s still running the old version.

    • bss03@infosec.pub
      link
      fedilink
      English
      arrow-up
      2
      ·
      3 hours ago

      Yep. Last time I used MS Visual Studio it was still all too eager to run an older binary, and I lot plenty of time debugging a binary that doesn’t reflect the current code. And, even more recently, when testing requires you to remember to run separate routines to upload to a testing server and restart a service, it is really easy to forget to do that when you are bouncing between the code and the browser console to figure out what to change.

  • kopasz7@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    13
    ·
    20 hours ago

    ECU (the computer in your car) only came online on the bus at every 3rd restart. No memory was written, yet the bastard somehow knew it anyway.

    Turns out the transciever’s enable pin was left unconfigured and supposedly some charge accumulated during two startups to set it.

  • MECHASQrL@lemmy.world
    link
    fedilink
    arrow-up
    17
    ·
    23 hours ago

    I can’t quite remember what I was debugging exactly, some constant was different in runtime compared to what was in code. Someone added a leading 0 to a const int to make the number align with others in terms of character count just to make it pretty I guess. That is when I learned about octal literals in C. I think that was the only time I’ve seen one in many years and it was used unintentionally. To save you a search leading 0 in C makes number to be base 8 same as 0x makes it hex aka base 16, so that inconspicuous 01000 is not a 1000 it’s a 512.

    • Dejected Warp Core@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      1 hour ago

      It’s one of the most sneaky legacy foot-guns C (and C-like languages) have. I think we’re decades removed from using octal everywhere, except for Linux filesystem flags (which the letter-equivalents should probably be used instead).

  • shalafi@lemmy.world
    link
    fedilink
    English
    arrow-up
    12
    ·
    22 hours ago

    This was dumb, and it was on me.

    Couldn’t figure out why two obscure software packages were erroring in a very particular way. Thinking there was no way in hell a search would help, I plugged it in Google. There was user on Spiceworks outlining my exact situation and fix. Who is this mad genius?!

    It was me. I fixed it two years previously and went back to note so in the forum.

  • Thebeardedsinglemalt@lemmy.world
    link
    fedilink
    arrow-up
    12
    ·
    22 hours ago

    I fixed a 5 year old issue that the previous guy claimed was “unfixable”. Turns out it was because someone copy-pasted a value from an Outlook email…and it had a single dash in it…which outlook converted to a double dash.

    • Giloron@programming.dev
      link
      fedilink
      arrow-up
      5
      ·
      19 hours ago

      That’s why I was so confused about the AI emdash connection. MS Office has been replacing dash with emdash for 20+ years.

      • thomasloven@lemmy.world
        link
        fedilink
        arrow-up
        4
        ·
        12 hours ago

        I was confused too. But now I think it’s specifically em dashes in the middle of words, isn’t it? Like copy—paste instead of copy-paste?

    • monkeyman512@lemmy.world
      link
      fedilink
      English
      arrow-up
      2
      ·
      22 hours ago

      I have had issues of having the same characters but in Unicode failing. But if I type it manually or paste it into Notepad++ forcing it to ASCII first, it works.

  • ExtraMedicated@lemmy.world
    link
    fedilink
    English
    arrow-up
    30
    ·
    1 day ago

    Internet Explorer didn’t automatically have “console.log” defined until you opened the console window. So there was an error breaking the web page that would disappear when you tried to open the console to see the error. The error was that console.log wasn’t defined. Fucking IE.

  • IphtashuFitz@lemmy.world
    link
    fedilink
    English
    arrow-up
    11
    ·
    1 day ago

    Back in the 90’s I had to debug a printer routine in a reporting tool. It would occasionally crash. Only on Wednesdays. Only on Wednesdays in September. Only on Wednesdays in September after September 9th.

    Code was written in c.

  • mkwt@lemmy.world
    link
    fedilink
    arrow-up
    20
    ·
    1 day ago

    This was on a Fortran project that had no encapsulation and way too many global variables.

    Suddenly, one particular calculation was failing, and not in a floating point rounding error kind of way. After setting up several scenarios to generate various input values, I determined that Fortran’s absolute value function was returning the wrong results.

    I couldn’t understand why, so I went to the Lead Architect. He poked at it for about four minutes and did some searches, and found where someone else had defined a global variable named ABS that was masking the built-in abs function. The function call argument [e.g. abs(3)] was being compiled as an array index on this ABS variable [which was not an array, so yay, undefined behaviour].

  • yaroto98@lemmy.world
    link
    fedilink
    arrow-up
    14
    ·
    1 day ago

    Was told to test a new database migration as they updated from mainframe to a current version of Oracle. First I was to test data integrity. Wrote a python script to pull from the old db and verify that record matched the new. Found sooooo many bugs, sent it to the dev. Half were real bugs, half were bugs with my script. Turns out they didn’t actually want a 1:1 on the new db they were going to change things cuz they could. Lots of back and forth and by the end my python script essentially mirrored the production code.

    Then they wanted me to load test the new system. I wrote a quick script to query N rows from the db with X threads and blast the new service. Didn’t take long to write, presented my finished script and plan to pull in manual testers to also run my script as needed.

    Dev was cocky because they’d setup auto scaling on AWS. Manager was concerned I’d taken time to write a script instead of just spamming the same message over and over with a tool like gatling.

    First run I took down the system very quickly, like within a minute with just my laptop. Note, this was a system that was planned to handle 100s of millions of requests an hour. Dev was impressed, turned out to be a auto scaling misconfiguration. Second time it took a few mins but it died again. Config needed some more tweaks. Third time I took it down because of some queueing issue that raised it’s ugly head. Fourth time I took it down they found a bug in the code. Fifth, Sixth, and Seventh time I needed the manual tester’s help, but the devs were scratching their heads trying to figure out how I was taking them out in 5mins of sustained load. Took two teams of devs scratching their heads and several itterations through bringing the system down and even two separate devs walked through my code to see what on earth I was doing. To top it off it wouldn’t just take down the microservice, it would crash our entire AWS infrastructure for the company. Not just our project but all our aws projects in the building went down.

    They finally found the problem. The silly little problem. Remember that data I pulled from the db and blasted them with? Well, we hit a poision pill situation. One of the records turns out was a manual tester’s favorite to use when running one of his tests. That one record had 30,000 attached sub-records. Everytime that tester ran that test case he just kept adding a sub-record to it. Dude didn’t know he was slowly crafting such a massive data bomb all those years that would take down all our AWS servers.

    They had also fixed the problem with one service crashing the others. And that my friends is why you never skimp on good automated testing. When it went live there was no major issues or outages, though we did let it soak in parallel with prod for weeks to make sure we trusted the results.