Systems and methods for web scraping

Abstract

A method includes receiving a request from a client web browser, generating a first instruction responsive to the request, modifying header information in the first instruction to produce a first header-modified instruction, forwarding the first header-modified instruction to a first target website, receiving a first response from the first target website, modifying, using a man in the middle (MITM) proxy, the first response to produce a modified first response, and sending a result to the client web browser responsive to the modified first response.

CPC Classifications