javaee论坛

超级管理员

97

帖子

126

回复

10438

积分

楼主
发表于 2019-08-27 12:31:21 | 查看: 41 | 回复: 0


最近整理了一下各平台爬虫的信息,记录一下(有异议的地方请指正)。有些个人的恶意IP并没统计。以下整理的爬虫IP都是根据对应的User-Agent排序的,所以可能统一号段的IP分属不同UA(当然UA也是可以随意伪造的),加上网上搜集的一些但是UA不确定所以都放到每个分组的最后边。


百度爬虫IP列表


爬虫Agent:Mozilla/5.0 (Windows NT 6.1; WOW64; Baidu Transcoder) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11

61.135.169.75

61.135.169.55

61.135.169.22

220.181.51.40

220.181.51.39

111.13.13.74

111.13.13.73

111.13.13.72

111.13.13.71

111.13.13.6

111.13.13.5

111.13.13.4

111.13.13.3

101.254.184.206

1.95.9.244

爬虫Agent:Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com)

59.50.71.83

211.89.227.50

211.89.227.16

211.89.227.15

183.224.87.36

124.119.30.90

117.25.173.119

116.113.28.179

爬虫Agent:Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

61.191.191.87

61.184.241.206

61.183.41.95

59.49.46.165

58.222.20.226

58.218.204.136

45.113.253.53

27.151.30.176

27.151.28.24

222.216.28.45

222.186.3.229

220.181.51.81

220.181.51.76

220.181.51.75

220.181.51.74

220.181.51.110

220.181.51.109

220.181.51.107

220.181.108.97

220.181.108.96

220.181.108.87

220.181.108.86

220.181.108.84

220.181.108.83

220.181.108.81

220.181.108.80

220.181.108.79

220.181.108.78

220.181.108.77

220.181.108.76

220.181.108.75

220.181.108.186

220.181.108.184

220.181.108.181

220.181.108.179

220.181.108.178

220.181.108.177

220.181.108.175

220.181.108.174

220.181.108.173

220.181.108.172

220.181.108.171

220.181.108.170

220.181.108.168

220.181.108.166

220.181.108.165

220.181.108.161

220.181.108.158

220.181.108.157

220.181.108.156

220.181.108.155

220.181.108.153

220.181.108.152

220.181.108.151

220.181.108.150

220.181.108.149

220.181.108.147

220.181.108.146

220.181.108.145

220.181.108.144

220.181.108.143

220.181.108.142

220.181.108.141

220.181.108.140

220.181.108.139

220.181.108.123

220.181.108.122

220.181.108.120

220.181.108.119

220.181.108.117

220.181.108.116

220.181.108.115

220.181.108.114

220.181.108.113

220.181.108.112

220.181.108.111

220.181.108.110

220.181.108.109

220.181.108.108

220.181.108.107

220.181.108.106

220.181.108.105

220.181.108.104

220.181.108.103

220.181.108.101

220.181.108.100

219.153.68.137

219.138.135.236

202.46.52.201

202.46.51.156

202.46.51.132

183.61.171.152

183.61.171.137

183.2.242.150

180.97.35.165

180.76.15.9

180.76.15.8

180.76.15.7

180.76.15.6

180.76.15.5

180.76.15.34

180.76.15.33

180.76.15.32

180.76.15.31

180.76.15.30

180.76.15.29

180.76.15.28

180.76.15.26

180.76.15.24

180.76.15.23

180.76.15.22

180.76.15.21

180.76.15.19

180.76.15.18

180.76.15.17

180.76.15.163

180.76.15.162

180.76.15.161

180.76.15.158

180.76.15.157

180.76.15.156

180.76.15.155

180.76.15.154

180.76.15.153

180.76.15.152

180.76.15.151

180.76.15.150

180.76.15.15

180.76.15.149

180.76.15.148

180.76.15.147

180.76.15.145

180.76.15.144

180.76.15.143

180.76.15.142

180.76.15.141

180.76.15.14

180.76.15.138

180.76.15.137

180.76.15.136

180.76.15.13

180.76.15.12

180.76.15.11

180.76.15.10

14.29.49.206

124.237.78.217

124.232.151.230

124.228.91.179

123.125.71.98

123.125.71.94

123.125.71.92

123.125.71.91

123.125.71.90

123.125.71.89

123.125.71.87

123.125.71.86

123.125.71.85

123.125.71.84

123.125.71.83

123.125.71.82

123.125.71.81

123.125.71.78

123.125.71.74

123.125.71.73

123.125.71.70

123.125.71.69

123.125.71.60

123.125.71.58

123.125.71.57

123.125.71.56

123.125.71.55

123.125.71.53

123.125.71.52

123.125.71.50

123.125.71.49

123.125.71.47

123.125.71.45

123.125.71.44

123.125.71.43

123.125.71.42

123.125.71.41

123.125.71.39

123.125.71.38

123.125.71.34

123.125.71.33

123.125.71.32

123.125.71.31

123.125.71.30

123.125.71.29

123.125.71.28

123.125.71.27

123.125.71.26

123.125.71.23

123.125.71.22

123.125.71.20

123.125.71.19

123.125.71.18

123.125.71.17

123.125.71.15

123.125.71.14

123.125.71.13

123.125.71.12

123.125.71.116

123.125.71.115

123.125.71.114

123.125.71.113

123.125.71.111

123.125.71.110

123.125.71.109

123.125.71.108

123.125.71.107

123.125.71.106

123.125.71.105

123.125.71.103

123.125.71.102

123.125.71.100

123.125.67.152

123.125.67.148

122.10.89.22

120.39.244.111

117.34.115.181

117.25.139.66

117.25.139.42

117.25.139.35

117.25.139.34

116.9.156.37

116.55.241.24

115.47.26.143

115.231.26.31

113.113.120.28

112.124.57.11

爬虫Agent:Baiduspider+(+http://www.baidu.com/search/spider.htm)

61.146.178.41

61.146.178.186

爬虫Agent:Baidu-YunGuance-VSBot(ce.baidu.com)

111.206.36.17

#网络搜集

180.149.130.*

180.76.5.66

百度云观测IP


安全检测服务User-Agent:

User-Agent:Baidu-YunGuanCe-ScanBot(ce.baidu.com) 可用性检测

其他服务User-Agent:

User-Agent:Baidu-YunGuanCe-SLABot(ce.baidu.com) 可用性检测

User-Agent:Baidu-YunGuance-PerfBot(ce.baidu.com) 速度检测

User-Agent:Baidu-YunGuance-VSBot(ce.baidu.com)  认证服务

User-Agent:Baidu-YunGuance-RSBot(ce.baidu.com)  即时速度检测

180.97.106.36

180.97.106.37

180.97.106.161

180.97.106.162

115.239.212.7

115.239.212.8

115.239.212.9

115.239.212.10

115.239.212.11

115.239.212.6

115.239.212.4

115.239.212.5

115.239.212.65

115.239.212.66

115.239.212.67

115.239.212.68

115.239.212.69

115.239.212.70

115.239.212.71

115.239.212.72

115.239.212.134

115.239.212.135

115.239.212.136

115.239.212.137

115.239.212.138

115.239.212.139

115.239.212.132

115.239.212.133

115.239.212.193

115.239.212.194

115.239.212.195

115.239.212.196

115.239.212.197

115.239.212.198

115.239.212.199

115.239.212.200

google爬虫列表


爬虫Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36

45.79.76.220

爬虫Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/27.0.1453 Safari/537.36

66.249.84.239

66.249.84.233

爬虫Agent:Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

66.249.79.26

66.249.79.142

66.249.79.138

66.249.75.7

66.249.75.23

66.249.75.15

66.249.69.48

66.249.69.15

66.249.65.55

66.249.65.162

203.208.60.158

203.208.60.152

爬虫Agent:Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

66.249.79.69

66.249.79.64

66.249.79.25

66.249.79.24

66.249.79.231

66.249.79.210

66.249.79.194

66.249.79.187

66.249.79.180

66.249.79.174

66.249.79.171

66.249.79.168

66.249.79.146

66.249.79.127

66.249.75.8

66.249.75.24

66.249.75.16

66.249.69.254

66.249.69.213

66.249.69.209

66.249.69.206

66.249.69.197

66.249.69.192

66.249.69.19

66.249.69.188

66.249.69.176

66.249.69.140

66.249.69.137

66.249.69.134

66.249.69.128

66.249.69.11

66.249.66.67

66.249.66.64

66.249.66.254

66.249.66.197

66.249.66.186

66.249.66.183

66.249.66.180

66.249.66.168

66.249.66.166

66.249.66.162

66.249.66.158

66.249.66.154

66.249.66.150

66.249.66.142

66.249.66.139

66.249.66.138

66.249.66.135

66.249.66.134

66.249.66.126

66.249.65.61

66.249.65.60

66.249.65.59

66.249.65.57

66.249.65.56

66.249.65.186

66.249.65.183

66.249.65.180

66.249.65.168

66.249.65.166

66.249.64.50

66.249.64.140

66.249.64.137

66.249.64.134

54.159.17.214

203.208.60.157

203.208.60.156

203.208.60.155

203.208.60.154

203.208.60.153

203.208.60.151

203.208.60.150

203.208.60.149

122.155.168.168

爬虫Agent:Mozilla/5.0 (compatible; Google-Site-Verification/1.0)

66.249.82.245

66.249.82.242

66.249.82.239

66.249.82.208

66.249.82.205

爬虫Agent:DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

66.249.66.192

#网上搜集

67.221.235.*

66.249.68.*

66.249.67.*

203.208.60.*

66.249.72.*

66.249.71.*

360爬虫列表


爬虫Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

42.236.49.171

42.236.10.*

爬虫Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

182.118.25.*

182.118.22.*

182.118.21.*

182.118.20.*

爬虫Agent:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider

180.153.187.*

180.153.186.*

180.153.185.*

爬虫Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

101.226.169.*

101.226.168.*

101.226.167.*

101.226.166.*

360爬虫这个倒是挺规矩的,基本上都是统一号段的IP都用一个UA。


必应爬虫列表


爬虫Agent:Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

40.77.167.95

40.77.167.39

40.77.167.19

207.46.13.183

207.46.13.151

207.46.13.64

207.46.13.13

157.55.39.212

157.55.39.203

157.55.39.188

157.55.39.169

157.55.39.131

#网上搜集

65.52.108.142

74.125.75.3

74.125.64.81

74.125.44.82

74.125.158.86

74.125.156.82

65.52.108.146

soso爬虫列表


#网上搜集

124.115.4.*

124.115.0.*

sogou爬虫列表


#网上搜集

220.181.94.231

220.181.94.229

220.181.94.223

220.181.125.71

220.181.125.69

220.181.125.45

123.126.50.76

123.126.50.70

220.181.94.237

220.181.94.235

220.181.94.233

220.181.94.225

220.181.94.224

220.181.94.213

220.181.125.43

220.181.125.162

220.181.125.108

202.85.214.159

123.126.50.81

123.126.50.78

123.126.50.77

123.126.50.75

123.126.50.74

123.126.50.72

123.126.50.71

123.126.50.69

123.126.50.68

123.126.50.66

yahoo爬虫列表


#网络搜集

72.30.142.223

67.195.37.168

67.195.37.154

202.160.180.198

202.160.180.176

202.160.180.118

202.160.179.16

202.160.179.127

202.160.178.70

110.75.176.30

110.75.176.29

110.75.176.28

110.75.176.27

110.75.176.26

110.75.176.25

110.75.173.176

110.75.173.175

110.75.173.174

110.75.173.173

110.75.173.172

110.75.173.171

74.6.18.249

203.209.252.21

202.160.189.241

202.160.189.234

202.160.188.215

202.160.184.15

202.160.182.11

202.160.181.190

202.160.180.8

202.160.180.73

202.160.180.66

202.160.180.59

202.160.180.53

202.160.180.45

202.160.180.39

202.160.180.32

202.160.180.27

202.160.180.191

202.160.180.187

202.160.180.17

202.160.180.165

202.160.180.163

202.160.180.16

202.160.180.158

202.160.180.155

202.160.180.154

202.160.180.148

202.160.180.138

202.160.180.136

有道爬虫列表


#网络搜集

61.135.217.27

61.135.249.9

61.135.249.89

61.135.249.78

61.135.249.75

61.135.249.62

61.135.249.6

61.135.249.59

61.135.249.37

61.135.249.32

61.135.249.249

61.135.249.239

61.135.249.236

61.135.249.232

61.135.249.231

61.135.249.229

61.135.249.220

61.135.249.216

61.135.249.213

61.135.249.210

61.135.249.191

61.135.249.187

61.135.249.175

61.135.249.171

61.135.249.146

61.135.249.144

61.135.249.143

61.135.249.137

61.135.249.122

61.135.249.112

61.135.248.228

61.135.248.226

61.135.248.219

61.135.248.210

61.135.248.203

61.135.220.9

61.135.220.3

msn爬虫列表


#网络搜集

207.46.204.38

207.46.204.37

207.46.204.35

207.46.204.128

207.46.199.244

207.46.199.242

207.46.199.213

207.46.194.95

207.46.194.91

207.46.194.88

207.46.194.85

207.46.194.78

207.46.194.67

207.46.194.55

207.46.194.140

207.46.194.130

207.46.194.129

207.46.204.44

207.46.204.43

207.46.204.42

207.46.204.40

207.46.204.39

207.46.204.34

207.46.204.31

207.46.204.30

207.46.204.138

207.46.204.137

207.46.204.133

207.46.204.129

207.46.199.249

207.46.199.246

207.46.199.240

207.46.199.238

207.46.199.229

207.46.199.218

207.46.199.216

207.46.199.215

207.46.195.167

207.46.194.96

207.46.194.94

207.46.194.90

207.46.194.89

207.46.194.87

207.46.194.83

207.46.194.82

207.46.194.81

207.46.194.77

207.46.194.75

207.46.194.74

207.46.194.73

针对爬虫IP,至于是喷杀虫药还是放进来自己喂,就看各位施主了。

此篇文章,可以理解为我闲的纯记录所用,不过目测会有人用到。

...


送给每个来过的人,不管你是刚开始学习的小白,还是独领风骚的老司机:不是每个人都能认同你,即使是你身边的人。人嘛,都这样,谁说不是呢?要一直相信在以后的日子里一定还会有来自各方诸多的刁难、抱怨,你不可能把所有人的嘴巴都堵上,也不可能让所有人的观点都和你一致。你要做的就是尽自己最大的努力,用实际行动去回敬那些曾反对、刁难过你的人。


我可以对一个人无限的好,前提是值得。



上一篇:为什么发帖没有权限 下一篇:ZXZx
您需要登录后才可以回帖 登录 | 立即注册

技术支持 JAVAEE V2.0 © 2016-2017