How do search engines work

Crawling is the process by which search engines discover new and updated web content. This is done by programs called crawlers or spiders. These crawlers continuously scan the internet in search of content and use different techniques to find out how many pages a website has and what type of content it is, ...
And then they download it all and put it in a gigantic database.

When these crawlers visit a website, they not only scan the content but also follow any links on that website (these can be internal or external links). This way, they can build a website map and discover more pages.
Crawlers tend to visit popular websites more frequently than smaller unknown websites. That is why getting a link from a popular website can result in your content getting discovered more rapidly (backlinks).

Now that you know the basics of crawling, it's time to explain 'Crawl budget' (or crawl time). These terms are the number of pages search engines crawl onto a website within a timeframe.
Search engines don't have unlimited resources (and power), so they want to crawl a website as efficiently as possible. That is why they assign a crawl budget (or time) to websites.

If you have a brand new site or blog, you will get assigned a limited crawl budget from the search engines. Which means that they only will crawl your site once every 2 or 3 weeks (this is not a fixed number). When your site gets older and has much more articles, search engines will see your blog is worth crawling and they will assign a higher crawling budget for your site or blog so it will be crawled more often.

The internet has billions of web pages, so you can imagine what a busy job these crawlers have. You, as a blogger, can do a few things to make it a bit easier for these crawlers and make sure that your site is crawled correctly without wasting any crawling budget:
- Use Robots.txt to tell the crawler which pages of your site you don't want being accessed.
- Use sitemap.xml to list all important pages so the crawler knows which pages are new or updated.
Both files are put in the root directory of your website (if you are using WordPress, plugins automatically do this).

Now crawlers have visited your site, it is time for the search engine to index it.
The information (content) found by the crawlers needs to be organized, sorted, and stored so the search algorithm can process it before it can be used as a search result.
This process is called indexing.

Here is where search engine algorithms come into play: they analyze the content of every page. Using natural language processing (NLP) and semantic analysis, search engines know the content of every page.

Search engines don't save a complete webpage (or blog post) into their database. No, it keeps a summary of it, containing the date of creation (or modification), short description, relevant keywords, inbound and outbound links, and some other data needed for their algorithm.
All this data is stored in a gigantic database called the index. This index is stored across different servers (using various database techniques) and serves as a reference for search queries

Okay, maybe ranking is the wrong word for the final phase. A better term would have been 'serving the results'. This is a very complicated process, which consists of 2 parts: analyzing the search intent and finding the appropriate answers. Both are handled by the (famous and mysterious) search algorithms.
These search algorithms have become extremely complex over the years (for example, Google uses more than 200 rules before making a decision).

And here is where SEO (Search Engine Optimization) plays a big role.
You want your blog ranked as high as possible to get more traffic. SEO helps search engines understand your blog better by pointing it in the right direction and telling Google what it is all about. So, in fact SEO is trying to convince the search engines why your blog is the most suitable answer for the user query.

In a next article, I'll write about about some basic SEO techniques you can implement to have your blog blog ranked in Google.
If you want to know more about making passive income with blogging, please visit Passive income with blogging for more tips and tricks.