at the same time the author also found an interesting phenomenon: the love of spiders in Shanghai for each page crawling generally only 2 times, while the first is always the right spider
for the above phenomenon the author explained: love Shanghai was originally the content on the website of the spider crawling should be 220.181.108.*, love Shanghai in crawling each new page are provided with the right spider, and then extract the contents and returns the database comparison, the decision of second of your site spider climb IP. That is to say, the so-called "220.181.108.* over the next day included for second times IP.
in the last "Shanghai Dragon: origin Website Web log play in the role of" Shanghai dragon has been introduced briefly to see a simple web site log in Shanghai Longfeng work, mentioned the importance of log. In fact, in my opinion, web log of each record is rather than to trace, is a mirror of the appropriate. The last time I mentioned, a very important point, but also a lot of people misunderstanding is easy to fall into the love of spiders in Shanghai is the first to enter your website to view the content quality rather than to know content quality and then decide what to see IP. Many readers of this very disdain, today I, took out the evidence for you to see.
second chain guide. >
most of the time we may take the role of Web log or ignore the myth, in fact, in my opinion, the main role is the role of web log.
Please look at the
on the content of the guide. When our web page 123.125.71.* IP we should be vigilant. We should for the first time to see this page, the page content is copied, or whether the contents of the spam. For the 123.125.68 and 220.181.51.* IP, we should understand it as a love Shanghai spider on your web site, warning. For these cases we should be the first time to the corresponding page for rectification. Most of the time when we visited the site in the spider will return 301304403404 code, we have to deal with in a timely manner.
circle above a red pen, the same page within a day crawling two times, two IP respectively is 18.104.22.168 and 22.214.171.124 can be very easy to see that the 2 IP is a right spider, a spider is spam. On the same day to crawl your site, for those online said the next day included pro, I will tell you, I love Shanghai in the input 贵族宝贝idilon贵族宝贝/products_show.asp today? Id=348, this page has not been included? Please say the next day included