Automated Ruleset Generation for “HTTPS Everywhere”: Challenges, Implementation, and Insights

This paper details the implementation of a Web crawler aimed at automating ruleset construction for “HTTPS Everywhere,” with a goal to convert HTTP URLs to secure HTTPS equivalents for enhanced communication security. Developed within a seven-month timeframe, the crawler faced challenges in verifyin...

Full description

Saved in:
Bibliographic Details
Published inInternational journal of information security and privacy Vol. 18; no. 1; pp. 1 - 14
Main Authors Alharbi, Fares, Kashyap, Gautam Siddharth, Allehyani, Budoor Ahmad
Format Journal Article
LanguageEnglish
Published Hershey IGI Global 17.07.2024
Subjects
Automation
Classification schemes
Criminal investigations
Cyberterrorism
Digital currencies
International organizations
Internet
Law enforcement
Privacy
Security
Virtual private networks
Websites
Online AccessGet full text

Cover

More Information
Summary:This paper details the implementation of a Web crawler aimed at automating ruleset construction for “HTTPS Everywhere,” with a goal to convert HTTP URLs to secure HTTPS equivalents for enhanced communication security. Developed within a seven-month timeframe, the crawler faced challenges in verifying HTTPS support, varying based on SSL certificate existence and validity. Successful ruleset creation and testing in Firefox and Chrome, adhering to stylistic standards, demonstrated the potential for effective development. The paper explores improving productivity through alternative libraries like Scrapy and Scrapy Cloud. While certain goals, such as in-depth cryptocurrency analysis and web crawler background reading, were unmet due to time constraints, valuable insights were gained. The conclusion underscores the difficulties, successes, and promises of automating ruleset generation through web crawlers for “HTTPS Everywhere,” offering valuable recommendations for advancing web security.
ISSN:1930-1650
1930-1669
DOI:10.4018/IJISP.347330