Scripts and data for analysing websites' robots.txt

DOI

Contains Jupyter notebook scripts, historical raw robots.txt data, and statistics on this data.

Identifier
DOI https://doi.org/10.17617/3.WW04D8
Metadata Access https://edmond.mpg.de/api/datasets/export?exporter=dataverse_json&persistentId=doi:10.17617/3.WW04D8
Provenance
Creator Nicolas Steinacker-Olsztyn
Publisher Edmond
Publication Year 2026
OpenAccess true
Contact nsteinac(at)mpi-inf.mpg.de
Representation
Language English
Resource Type Dataset
Version 1
Discipline Other