We’re (a group of friends) building a search engine from scratch to compete with DuckDuckGo. It still needs a name and logo.
Here’s some pictures (results not cherrypicked): https://imgur.com/a/eVeQKWB
Unique traits:
- Written in pure Rust backend, HTML and CSS only on frontend - no JavaScript, PHP, SQL, etc…
- Has a custom database, schema, engine, indexer, parser, and spider
- Extensively themeable with CSS - theme submissions welcome
- Only two crates used - TOML and Rocket (plus Rust’s standard library)
- Homegrown index - not based on Google, Bing, Yandex, Baidu, or anything else
- Pages are statically generated - super fast load times
- If an onion link is available, an “Onion” button appears to the left of the clearnet URL
- Easy to audit - No: JavaScript, WASM, etc… requests can be audited with F12 network tab
- Works over Tor with strictest settings (official Tor hidden service address at the bottom of this post)
- Allows for modifiers: hacker -news +youtube removes all results containing hacker news and only includes results that contain the word “youtube”
- Optional tracker removal from results - on by default h No censorship - results are what they are (exception: underage material)
- No ads in results - if we do ever have ads, they’ll be purely text in the bottom right corner, away from results, no media
- Everything runs in memory, no user queries saved.
- Would make Richard Stallman smile :)
THIS IS A PRE-ALPHA PRODUCT, it will get much MUCH better over the coming months. The dataset in the temporary hidden service linked below does not do our algorithm justice, its there to prove our concept. Please don’t judge the technology until beta.
Onion URL (hosted on my laptop since so many people asked for the link): ht6wt7cs7nbzn53tpcnliig6zrqyfuimoght2pkuyafz5lognv4uvmqd.onion
I don’t know DuckDuckGo, but what’s the purpose of trying to compete with it? This is not a rhetorical question. Is there something wrong with DuckDuckGo, something you feel you can do better, or are you just making a competitor for the principle?
Not OP, but there is value in having competition. DDG is just a bing front-end. The big search engines have a major problem with the quality of results going down, as the internet is SEOd to death. The companies behind these engines don’t seem to be very eager to fix it, they are just hoping to replace them with AI. We’ve also seen how these engines have been turned into ad platforms, which changes the incentives… Instead of ranking quality, they are ranking who pays more.
Taking a different approach to ranking results that isn’t ad driven, that can punish AI generated content and low quantity results would bring a huge value.
DDG is just a bing front-end.
That is wrong. Yes there are licensing the bing search database but it is not the only one they use. They have their own crawler too.
“Only two crates used”. What’s great about reinventing the wheel? A closed source project with big claims trying to reinvent everything from scratch. Nice project 🤣
I got so excited reading this post, but as I read that the project will not be open source, my excitement immediately faded away
They won’t open source it because the rust code is very likely a joke. They are proud of just using two dependencies, don’t know that their “statically generated” stuff is actually called server side rendering and are hosting this stuff on a fuckin laptop.
It’s probably a project that will teach them a lot. But in practice their implementation is worthless to everybody else because they are obviously completely inexperienced.
That said, that project is likely not worthless to them because they will probably learn a ton of stuff why it’s hard to build a search engine.
Would make Richard Stallman smile :)
If this is a closed source project, that statement doesn’t work even as a joke.
However, the screenshots looked good. :)
That comment is there specifically to drive engagement up with all of the people correcting me in the comments.
Lying is a great way to get engagement in the post, and then see your project crash and burn.
I’m only interested in your rant in a few weeks when nobody cares.
Fair!
How does it compare with Kagi?
That Kagi works and that Kagi doesn’t write their own database engine and host the project on a laptop 🤣
For anyone who asks, we will not open source, but we may offer a licensed self-hosted version for a small yearly fee (maybe $10-20).
I’ve open sourced everything I’ve made in the past, and all that’s happened is someone with more money picks up my project, outcompetes me, and drives me out of business. I dont want our hard work going to waste.
It would be a little more work, but you could make your front-end separate and open source, just so people could see you aren’t tracking them while still keeping your indexing, web crawling, ranking, and database, I.e the real hard stuff closed source. I don’t care much, but with the crowd you’re appealing to it might help get more customers.
The frontend is pure HTML and CSS, you can see what its doing with inspect element, all network requests too
we will not open source
Would make Richard Stallman smile :)
pick one.
Our frontend is open, its just HTML and CSS, nothing proprietary client side, which would piss him off.
He’d be pissed off about the BE being proprietary.
Actually, kind of not. FSF has a weird blindspot when it comes to backend stuff. They care about what runs on your computer, e.g. Javascript, but stuff running on a server that you don’t own they don’t really care about. The AGPL had to come from an outside third party, not the FSF themselves. The FSF has been pretty silent when it comes to making licenses to regulate the whole “cloud” space.
It’s one of the big reason why the FSF has been slowly driving away into irrelevancy. The modern computing world is all about servers and data flowing between them. And the FSF is continuing doing licensing like it’s the 1980s. They are so far behind that politics got there first with the GDPR. There is still no “GNU GDPR” that you can slap on your software to give people outside Europe similar rights.
Isn’t that literally what the GPL3 exists to prevent? So called “tivoization”?
You know what would really make stallman smile is if you licensed it under something that requires sharing the source and all modifications.
Also solves the cost problem since you no longer have to support the infrastructure for all users just a small subset. I already host searxng and it’s not great but I host it because I support projects that encourage self hosting and we need more free and open source search options
Also how is your product getting used by another company “going to waste” all your work? Maybe you mean take advantage of without paying? But if that’s the case and money is such a big deal how is advertising off the table? Unless you’re trying to compete with kagi, but you’re facing a huge uphill struggle if you plan to monitize and not be open source/self hostable…
If on the other hand you change your mind about it, I’ll be one of many self hosting fans who will happily deploy and contribute to improve the project in any way possible. (Currently self hosting searxng here)
That said, even if it’s not fully open, really looking forward to seeing the project grow, it sounds like you have your hearts in the right place and we sure do need something better.
the goal is to prevent competition, not promote it.
Not sure I follow your reasoning here… Having commercial forks isn’t necessarily “competition” or if it is it’s not always a bad thing. I suppose you could look at something like say Debian as being in "competition” with Ubuntu, but they both promote the underlying linux kernel and benefit from each other’s improvements, while one is very much for profit while the other is a very well funded non profit organization.
Sounds promising, please keep us posted and good luck!
Sounds promising
Does it? Sounds like the exact opposite.
Pages are statically generated
Can you elaborate on that? To me, statically generated would mean you are pre-rendering a html page for every possible search, which doesnt sound possible? Do you mean that its all server side generated (at the time of search)?
That’s a neat project.
You can be proud of your work 😊But I for one won’t donate to your cause, as the software seems to be closed-source, and I already have DuckDuckGo & Google for my searching needs.
I genuinely believe that the only viable niches for new search engines are environmentally-friendly (e.g. Ecosia) or open-source.
Literally no one will pay for a closed-source search engine.
But I like your tech stack, and your project’s looking good.
One more thing: You claim to be against censorship; how will you combat spam & SEO farming?
deleted by creator
If you’re willing to release the database or at least the scraper as FOSS, I’d be willing to try making it distributed. The way this would work is:
- Users opt-in to storing part of the index on their machine
- Official servers are just relays to a network of these users
- Frontend JS would be necessary to stream the results from the network
I also don’t know the profit model here, but costs would be quite low since you’d only need geographically distributed relays with no storage requirements and minimal processing, though bandwidth would be quite high (I think bandwidth could be reduced with something like WebTorrent). I’m thinking maybe users could pay some nominal fee ($1-2/month), or get the service free by agreeing to host data (to seed, you would provide a few nodes with the full index; these could be removed once enough people sign up).
I’m working on a related project right now (distributed Reddit alternative), but a search engine project may be faster to get off the ground. Updates to the index would use the same network.
Sounds great! Where do I fill in my sensitive data?
Sounds interesting! I saw some other guy post about how you guys wouldn’t pick pro ukrainian content over pro russian and I think that’s the right thing to do. I always found it “interesting” that youtube will always promote the legacy media (in my eyes akin to propaganda) whenever you search for news or current events. Look forward to seeing where this goes and i hope you have an open policy about decisions in the search engine about what you promote vs demote. Who knows what else the other engines are promoting when people search to skew their views.
It still needs a name and logo.
Index. Tell it like it is.
Pointless talking about the code when the main challenges will be ops and infrastructure. What are you doing about those? Linking to an imgur hosted graphic isn’t a good look in that regard.
