How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Here’s what is says on their site:
“Courtesy backoff
Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.”
By the time you get the error you have already crashed the site.
Geez.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
Someone at work vibe coded a replacement for a entire third party SaaS that we were using in a week. Their version is better.
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
> If Anthropic or OpenAI fails, the other wins the US market, no?
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
I agree on the analysis, but not on the conclusion. What makes you think that the conditions you describe are stable grounds for a duopoly where both could be successful at the same time?
If they did then Google would take over, and others would form from their competent engineers. Also, they don’t need to be profitable to succeed, they merely need to hype investors enough to ensure that their losses matter less than their growth projections.
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
The last part is extremely dubious with some recent reveals.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Incentives are crazy powerful.
Please stop. The last thing we need in the world is more greed and selfishness.
Only strong regulation can help.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
I understand if you pay is a different story.
I mean ... "how does this work?"
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
the moment there is a model which can code whole app, we can fairly say software engineering is done.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
You are right that there's plenty of use cases where Opus 5.5 is better.
Not everyone wants Anthropic to succeed.
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.