Crawlers are responsible for inspecting URLs by following both internal and external links to evaluate backlinks. With this update, Semrush claims that the number of crawls has tripled, which also makes it easier to locate relevant content.
A sites with URL parameters that do not affect the content of the page have also stopped being crawled, while the reading ratio of robot.txt files on websites has been increased, in order to offer cleaner data, without duplicate links.
Storage
This has undoubtedly been the aspect that has taken the most work from the developers, since the idea was to completely eliminate temporary storage. To do this, a complete rewrite of the tool's benin business email database structure was carried out, meaning that a new one was practically created from scratch. The result was a limitless scalable system that updates information automatically.
To achieve this goal, the Semrush engineering team incorporated more than 500 servers in total , 287 TB of RAM and 16,128 CPU cores into the system , which allow information to be filtered and reports to be generated at incredible speed, in order to have them available immediately.
By the way, we have spoken with Semrush and they have offered our readers an extension of the free trial from 7 to 14 days, through the following link.