cm0002@lemmy.world to Technology@lemmy.worldEnglish · 2 个月前Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comexternal-linkmessage-square25fedilinkarrow-up1390arrow-down14cross-posted to: [email protected][email protected]
arrow-up1386arrow-down1external-linkOpen Source devs say AI crawlers dominate traffic, forcing blocks on entire countriesarstechnica.comcm0002@lemmy.world to Technology@lemmy.worldEnglish · 2 个月前message-square25fedilinkcross-posted to: [email protected][email protected]
minus-squaretal@lemmy.todaylinkfedilinkEnglisharrow-up11·2 个月前They can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.
They can just interleave requests to different hosts. Honestly, someone spidering the whole Web probably should be doing that regardless.