Recently I was wandering if there is someone or some group preserving , collecting , organizing and publishing all the knowledge of mankind ever created throughout its existence so that if ever mankind faces the 6th mass extinction we don’t have to reinvent the wheel and can have a kick start to our new post apocalyptic civilization .
Wikipedia is a great start. You can download its entirety, roughly 100gb. Most of the basic and advanced human knowledge.
Check out kiwix to get it offline
You can do all of Project Gutenberg too. It’s only about 75gb, surprisingly.
You son of a bitch
Such an insightful commentary on the importance of the social contract and the irreplacibility of the individual. The only way forward is to share our personal experiences and strive for understanding. Once we know each other’s value, we will never surrender our common bonds, disappoint one another, go behind each other’s backs, nor do each other harm.
You’re using it right now.
That is pretty much exactly the goal of the Wikimedia Foundation which runs Wikipedia and its sister projects.
But by now we figured out what wikis can do well and what not. Wikis are suitable for crowdsourcing objective facts about the world (all it takes is one person to add any given fact), they are not a universal remedy for everything, especially not contentious issues or useful instructional materials.
I have made more than 100000 edits to their projects. I don’t participate there anymore. The time when they were a force for good in the world is long past.
Why aren’t they a force for good anymore?
Many reasons most of which you’ll only understand if you pay some attention to what’s going on behind their scenes.
There are reasons why nowadays pretty much everywhere else on the Internet more content is created all the time than on the Wikimedia projects.
The Wikipedias’ “neutral point of view” policy used to mean “we try to treat all sides fairly”, now it means “we are writing an unconditional propaganda organ for the status quo”. The mainstream media that is accepted as “reliable” as Wikipedia sources just isn’t that credible anymore.
Also, when I started editing there, the individual projects were mostly left alone by the WMF. Nowadays the WMF issues intransparent sanctions, up to lifetime bans from all projects, left and right.
I wish someone started an organization with the same goals as the WMF with an actually working system where people could actually enjoy participating.
Be the change you want to see in the world…
Heck most of the hard work is already done for you, since the software that runs Wikipedia is open source.
Many people have tried that before. Wikis just aren’t that appealing anymore. Today’s internet is all about social media.
If there’s anything that is absolutely atrocious as a searchable repository of knowledge, it’s social media.
All of Wikipedia is <256 gb.
All of Wikipedia in English <64 gb.
Then archive.org for multimedia, ~10 peta bytes. Yipes.
Well archive.org has much more videos
Which are part of human knowledge.
True but I still don’t really find it to be a fair comparison, both are great at what they do
According to my ex-, her.
Can confim
Haha you can have her. Good luck!
Please take her back. I’ll even pay you.
Check out this book: https://en.m.wikipedia.org/wiki/The_Knowledge:_How_to_Rebuild_Our_World_from_Scratch. It analyses that precise question in the first chapter. The author argues that even though Wikipedia is probably the closest thing there is, there is a clear lack of practical knowledge that will be essential in the situation that you are describing. Science progress heavily relies on industrial progress, and even if you know how to build something that doesn’t mean that you can do it, as there are other things that are required first.
I think the internet as a whole is going to be the closest we’ll ever come. Capitalism will make sure it’s never even close to complete so it always has something to monetize.
you can read pretty much (except the lost media like those lost in library burnings , film destruction and wars) read any book written by humans since 2500 bce (example Rig Veda the first ved of Hinduism was written even before 2500 and is today said to be 98% at its original state thanks to Indian gurus and saints who passed it on orally and was made into a book only after 8th century) , watch any movie ever released , hear any music ever made after recording was invented .
ofcourse there is a catch that these medias are not freely and publicly available and most you have to pirate in order to consume it thus we need to have a centralised database of these things safely kept somewhere so that we don’t have to reinvent the wheel in case of a catastrophic event .
I wouldn’t say “complete” can even be sufficiently defined in this case. Every functional definition I can think of has a limiting factor.
Let’s try to define knowledge. What kind of information qualifies? We can usually think of important, useful info like physics and medicine. But what about other data, like sports game stats, atmospheric sensor readings, or even something more esoteric, like the location data of every object on earth.
And even if we could have the information of every single thing at any particular time, what about when things change in the next second? And the one afterwards?
Essentially, nothing will ever be “complete”. Thanks for listening to my rant on semantics.
That was a lovely rant on semantics. I thoroughly enjoyed reading it!
I’m surprised no one mentioned projects like libgen and scihub. They are much better than Wikipedia imo.
imo zlib is much better but they keep changing their domain … also sci hub is only for research papers which most people can understand
Well, technically, Library of Babel though that probably isn’t really what you’re looking for.
Besides what other commenters already said, archive.org does a great job.
Thanks
It’s never going to be all knowledge, since a lot of stuff is just lost or never recorded. A ton of stuff (like this thread) are probably low on the priority list for recording as well. But the closest you’d probably get to a full catalog of human knowledge (at last text based) are the huge data sets of nearly all text data on the internet used for training LLMs. I wouldn’t be surprised if there are ones soon that include video and pictures as well, since newer AI models are starting to be able to interpret those too.
I believe this is one of those data sets: https://github.com/yaodongC/awesome-instruction-dataset
Edit: here’s a big data set used for a lot of gpt3 https://commoncrawl.org/
