From the same group that doesn’t understand joins and thinks nobody uses SQL this is hardly surprising .
Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.
You’re giving this person a lot of credit. It’s probably all in the same table and this idiot is probably doing something like a for-loop over an integer range (the length of the table) where it pulls the entire table down every iteration of the loop, dumps it to a local file, and then uses plain text search or some really bad regex’s to find the data they’re looking for.
Considering that is nearly exactly some of the answers I’ve received during the technical part of interviews for jr data eng, you’re probably not far off.
Shit I’ve seen solutions done up that look like that, fighting the optimiser every step (amongst other things)
I think you’re still giving them too much credit with the for loop and regex and everything. I’m thinking they exported something to Excel, got 60k rows, then tried to add a lookup formula to them. Since you know, they don’t use SQL. I’ve done ridiculous things like that in Excel, and it can get so busy that it slows down your whole computer, which I can imagine someone could interpret as their “hard drive overheating”.
I have to admit I still have some legacy code that does that.
Then I found pandas. Life changed for the better.
Now I have lots if old code that I’ll update, “one day”.
However, even my old code, terrible as it is, does not overheat anything, and can process massively larger sets of data than 60,000 rows without any issue except poor efficiency.
60k isn’t that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?
A TI-86 can query 60k rows without breaking a sweat.
If his hard drive overheated from that, he is doing something very wrong, very unhygienic, or both.
He probably mining crypto on top of running his SQL queries.
There must be more join statements than column names
Seriously - I can parse multiple tables of 5+ million row each… in EXCEL… on a 10 year old desktop and not have the fan even speed up. Even the legacy Access database I work with handles multiple million+ row tables better than that.
Sounds like the kid was running his AI hamsters too hard and they died of exhaustion.
I’d do that if I was given so much stupid access
No, its an external drive, appearently.
deleted by creator
60k of rows is nothing. Fuck, where do you find these “geniuses”?
I do data analysis for a living, I reach out to tech and complain if I can’t open a file with a million+ lines.
There’s only one reason he wants kids and not experienced adults.
He’s attracted to them?
Probably
Easily manipulated, but probably that too.
my hard drive overheated
So, this means they either have a local copy on disk of whatever database they’re querying, or they’re dumping a remote db to disk at some point before/during/after their query, right?
Either way, I have just one question - why?
Edit: found the thread with a more in-depth explanation elsewhere in the thread: https://xcancel.com/DataRepublican/status/1900593377370087648#m
So yeah, she’s apparently toting around an external hard drive with a copy of the “multiple terabytes” large US spending database, running queries against it, then dumping the 60k-row result set to CSV for further processing.
I’m still confused at what point the external drive overheats, even if she is doing all this in a “hot humid” hotel room that she can’t run any fans I guess because her kids were asleep?
But like, all of that just adds more questions, and doesn’t really answer the first one - why?
Have you ever heard of case of overheating hard drives within the last decade?
Plus, 60k is nothing. One of our customers had a database that was over 3M records before it got some maintenance. No issue with overheating lol
I run queries throughout the day that can return 8 million+ rows easily. Granted, it takes few minutes to run, but it has never caused a single issue with overheating even on slim pc’s.
This makes no fucking sense. 60k rows would return in a flash even on shitty hardware. And if it taxes anything, it’s gonna be the ram or cpu- not the hard drive.
In my experience, the only time that I’ve taxed a drive when doing a database query is either when dumping it, or with SQLite’s vacuum, which copies the whole thing.
For a pretty simple search like OP seems to be doing, the indices should have taken care of basically all the heavy lifting.
I literally work with ~750,000 line exports on the daily on my little Lenovo workbook. It gets a little cranky, especially if I have a few of those big ones open, but I have yet to witness my hard drive melting down over it. I’m not doing anything special, and I have the exact same business-economy tier setup 95% of our business uses. While I’m doing this, that little champion is also driving 4 large monitors because I’m actual scum like that. Still no hardware meltdowns after 3 years, but I’ll admit the cat likes how warm it gets.
750k lines is just for the branch specific item preferences table for one of our smaller business streams, too - FORGET what our sales record tables would look like, let alone the whole database! And when we’re talking about the entirety of the social security database, which should contain at least one line each in a table somewhere for most of the hundreds of millions of people currently living in the US, PLUS any historical records for dead people??
Your hard drive melting after 60k lines, plus the attitude that 60k lines is a lot for a major database, speaks to GLARING IT incompetence.
Reminds me of those 90s ads about hackers making your pc explode.
Musk gonna roll up in a wheelchair, “the attempt on my life has left me ketamine addicted and all knowing and powerful.”
I don’t think I’ve seen a brand new computer in the past decade that even had a mechanical hard drive at all unless it was purpose-built for storing multiple terabytes, and 60K rows wouldn’t even take multiple gigabytes.
I have when a misconfigured spark job I was debugging was filling hard drives with tb of error logs and killing the drives.
That was a pretty weird edge case though, and I don’t think the drives were melting, plus this was closer to 10 years ago when SSD write lifetimes were crappy and we bought a bad batch of drives.
I’d much sooner assume that they’re just fucking stupid and talking out of their ass tbh.
Same as Elon when he confidently told off engineers during his takeover of Twitter or gestures broadly at the Mr. Dunning Kruger himself
Wonder if it’s an SQL DB
Elon probably hired confident right wingers whose parents bought and paid their way through prestigious schools. If he hired anyone truly skilled and knowledgeable, they’d call him out on his bullshit. So the people gutting government programs and passing around private data like candy are just confidently incorrect
My one question would be “How?”
What the hell are you doing that your hard drives are overheating? How do you even know it’s overheating as I’m like 90% certain hard drives (except NVMe if we’re being liberal with the meaning of hard drive) don’t even have temperature sensors?
The only conclusion I can come to is that everything he’s saying is just bullshit.
They have temp sensors. But have never heard of a overheating drive.
Hard drives do get hot and need some cooling but not at 60k rows. Its either made up or their computer case is made of thermal cladding
You could query 60,000 rows on a low tier smart phone. Makes no sense at all.
Can we think of any device someone might have that would struggle with 60k? Certainly an ESP32 chip could handle it fine, so most IoT devices would work…
Right? There’s no part of that xeet that makes any real sense coming from a “data engineer.”
Terrifying, really.
Unless the database was designed by someone who only knows of data as that robot from Star Trek, most would be absolutely fine with 60k rows. I wouldn’t be surprised if the machine they’re using caches that much in RAM alone.
dude is 100% talking about ssds. NVME ones at that, he’s just stupid.
Even if it was local, a raspberry pi can handle a query that size.
Edit - honestly, it reeks of a knowledge level that calls the entire PC a “hard drive”.
Unless they actually mean the hard drive, and not the computer. I’ve definitely had a cheap enclosure overheat and drop out on me before when trying to seek the drive a bunch, although it’s more likely the enclosure’s own electronics overheating. Unless their query was rubbish, a simple database scan/search like that should be fast, and not demanding in the slightest. Doubly so if it’s dedicated, and not using some embedded thing like SQLite. A few dozen thousand queries should be basically nothing.
Yeah, no matter what way you disorganize 60,000 rows, the data is still going to read into memory once.
Why? Because they feel the need to have local copies of sensitive financial information because… You know… They are computer security experts.
I’m a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?
Seems like a good excuse to someone who doesn’t know what they’re doing and needs an excuse because why they haven’t completed it yet?
The whole post is complete bs in multiple ways. So weird.
Some interesting facts about excel I learned the hard way.
- It only supports about a million or so rows
- It completely screws up numbers if the column is a number and the number is over 15 digits long.
Not really related to what you said, but I’m still sore about the bad data import that caused me days of work to clean up.
The row limitation seems, to me, like an actually-good thing. Excel is for data where you might conceivably scroll up and down looking at it and 1M is definitely beyond the ability of a human even to just skim looking for something different.
An older version of Excel could only handle 64k rows and I had a client who wanted large amounts of data in Excel format. “Oh sorry, it’s a Microsoft limitation,” I was thrilled to say. “I have no choice but to give you a useful summarization of the data instead of 800k rows (each 1000 columns wide) of raw data.”
Some time ago, I heard a story of CS and Econ professors having lunch together. The Econ professor was excited that Excel was going to release a version that blew out the 64k row limit. The CS professor nearly choked on his lunch.
Dependence on Excel has definitely caused bad papers to be published in the Econ space, and has had real world consequences. There was a paper years ago that stated that once a country’s debt gets above 120% of GDP, its economy goes into a death spiral. It was passed around as established fact by the sorts of politicians who justify austerity. Problem was, nobody could reproduce the results. Then an Econ undergrad asked the original author for their Excel spreadsheet, and they found a coding error in the formulas. Once corrected, the conclusion disappeared.
Diming on her own stupidity without realizing it, like all of them do
This sounds like trying to do stuff in Excel? The computer isn’t overheating but the amount of memory needed is very high which would make it run poorly. They might interpret that as overheating?
It also makes sense if they are on calling the entire computer “the hard drive” like grandma and the fans kicked on.
Yeah, everyone commenting about being able to handle billions of rows easily, which obviously very true if you are worming with sql or similar.
But this is probably some finance kid, investment banker analyst, and only knows how to use Excel.
60,000 rows in excel with formulas, if not done efficiently, can for sure make you computer a little toasty.
Could be a gen 5 nvme drive without adequate cooling. Them bastards can run hot. Especially the early gen 5 drives.
But with only 60k rows would one even have enough time to overheat?
This cannot be real, wtf. This is cartoon levels of ineptitude.
Or sabotage by someone heading out? Please let this be resistance sabotage they haven’t noticed yet.
You guys arent running your software off raspberry pi’s with sdcards from the gas station?
My allowance is 5$ a month!
I think my Pi could process 60k rows without overheating. And the poor thing is dangling behind my bookshelf from its power cord with a fine layer of dust coating every inch of it.
Why would you do this to that poor rPi?
Electronics get what they deserve.
Look, all I’m saying is give Pis a chance.
This shit sounds like when your mom tells you that the Facebook printed out her bank statement on the fax machine. I’m not smart enough to even guess how you did something dumb enough to make that happen.
How bad are you at writing queries? How does your hard drive overheat even under 100% load? Do you have it smothered under a blanket? Did you crack it up and expose it to cheeto dust? What does running a query on your, presumably, remote database even have to do with your harddrive in the first place? Are you trying to copy the entire database locally to a laptop? Do you know how to tie your shoes yet, or are you still on the velcro?
A laptop should easily handle a database of 60,000 rows. I run much bigger databases on my own laptop for development purposes.
Not if it is filled to the brim with 3 TB of porn
Delete Ass Master volume 7 to make room - that one wasn’t any good anyways.
Really went downhill after volume 3. You hope that they’ll get back on course by volume 6 at least, but alas.
I can personally vouch for that not causing a problem either.
This shit sounds like when you’re mom tells you that the Facebook printed out her bank statement on the tax machine.
My dear sweet mother asked me somewhere around 2005-06 “If we can fax paper, why not groceries, or pizza delivery?”
Apparently she had believed, for decades, that fax machines literally transported physical paper over phone lines. She has a college degree, and my family is wealthy.
Do not underestimate the mind boggling technical and scientific ignorance of old people who should know better.
🤣
Maybe
Maybe
They are just making shit up and doing jack shit
I really hope so.
When the only thing that is stopping kids from dismantling your government is an O(N^N) algorithm
Are you telling me there’s a difference between an inner and a cross join?
Cross join is obviously faster, I don’t even have to write “on”
You’re not supposed to place your laptop directly in the lap of your fur suit. Always leave an air gap for ventilation, smh.
Even a gamer knows that ssdd heat up but never to that level, lol.
What kind of cheap temu ssd does he have in his laptop?
He was saving money, you know, to be efficient.
Literally every time someone dismisses Wikipedia, it’s because they believe something crazy that Wikipedia told them is wrong.
I checked conservapedia once, and its actually unhinged. If someone tells you to look at that, or reccommends it, they’re crazy.
Did they ever finish their own bible translation? The one they started because King James was too woke.
YES . its so unhinged . they have an entire page discussing if Obama is actually a Muslim
Flashback to 2014.
heh
“YOU’RE JUST JEALOUS” is such a fucking pussy-ass response, too.
God they picked out the ONE possible thing they could criticize her for, there’s like 3 other things RIGHT NEXT TO THAT
“Software Engineer” was literally right next to it.
She is vehemently against crypto. She runs the newsletter “web3 is going great” - https://www.web3isgoinggreat.com/
deleted by creator
As a reasonably experienced “data guy,” this seems obviously laughable, but the discussion on X is scary. This guy is a savior in the MAGA world.
We can criticize and poke fun all day, but it doesn’t matter much if our message isn’t challenging the mindset of those with other opinions.
How do we make better use of our time to impact outside opinion?
I’ve been told violence isn’t the answer and we shouldn’t just shoot nazis and nazi enablers dead.
The way most people change their mind isn’t based on facts or figures, but emotions. Specifically, in-group belonging. For most people, and this certainly includes me and you some of the time, what our in-group believes is more compelling than an out-groups supposed facts.
They see that guy as someone in their group so they believe him. They see you as a bad outside bad bad bad liar, so nothing you say is likely to get through. (This comic is worth reading on this topic: https://theoatmeal.com/comics/believe )
If you want to change someone’s mind, they have to see you as in-group. Not necessarily the same group as what you’re arguing with. We all belong to many groups. American, new yorker, white guy, middle aged, yankees fan, etc etc there are many such slices. Like how you can’t get a republican to recycle by appealing to environmental concerns (because environmentalists are out-group, so fuck them), but you might be able to get them to recycle via something like “only american ingenuity can turn trash into bridges and tanks!”
This takes a lot of time and effort, and if you don’t get them to stop hanging out with the other group, you won’t make any lasting changes.
So I think you’d need a multi prong approach:
- Get them off bad media. Facebook, fox news, etc. This is reinforcing their bad beliefs. Because they see this stuff as trustworthy in-group, it goes right into the worldview.
- Get them to stop hanging out with their shitty maga-hat friends. This is the social in-group that’s reinforcing bad beliefs.
- Get them to trust you.
- Gently introduce the idea that maybe the extreme right doesn’t have their interests at heart, etc
All of which takes a lot of time and effort, and your opposite number is basically trying to do the same thing. Except they have fox news, trump, and such in their corner.
And, again, I’m told we definitely shouldn’t just shoot extreme right wingers and other nazi sympathizers dead. Nor should we burn their houses down. If we’re an emergency responder, we definitely shouldn’t let them die while thinking to ourselves “they would let so many die. without a thought, their passing deserves no mourning” or similar.
You should definitely nullify if you’re on a jury and someone allegedly did violence to a shitty ceo or red-hat, though, bu that’s getting off topic.
Wow, that was an awesome rabbit hole, thank you for the link.
If you want to change someone’s mind, they have to see you as in-group.
Maybe a less manipulative-sounding way to phrase that might be that we should remind people that we’re all in it together. The far right media and their billionaire buddies have spent the past decade and a half dividing us, and they succeeded. Idk what it would take to unite this country again, but it at least is a little comforting to have a clear problem statement.
I’ve been told violence isn’t the answer
By the very same people asking the question: What are you gonna do about it?
Where ‘it’ is your oppression.
We must make better memes
I’m not even joking, the world runs on memes now. It’s fucking stupid, but we must shitpost to save ourselves
I agree some form of consistent opposition messaging is needed.
The maga world talks in consistent themes and terminology, which creates a psychological advantage. Unfortunately, it’s playground psychology, but if that’s the game being played you need to find a way to win at it.
Compelling point. I just found that arguing with „these kind of people“ (livibg in europe, so no MAGA‘s here but like-minded, conservative fundamentalists etc.) leads to nowhere. It‘s kind of like the covid-conversations. And often I heard „you can‘t make them change their minds, so just let them be“. Still, I think this behaviour leads to isolation and separates us as a people even more.
Long story short: good question. If you found the answer, let me know.
Good lord.
