Go2Proxy
Products
Pricing$1.11/GB
Cases
Locations
Resources

Scraping Proxies and Incremental Crawls with ETags

Go2proxy
2026-11-16

Scraping proxies can support an incremental collection workflow, but an ETag belongs to the representation supplied by a source, not to a universal business fact. A conditional response can help avoid transferring unchanged content when the source supports it. When using Go2Proxy data collection options, preserve the request context and the cached representation together.

Use conditional requests only within the destination's approved access rules. They do not replace a respectful schedule or authorize additional collection.

Go2Proxy data collection proxy page with a web scraping and code illustration

Keep validators with the accepted representation

Store the source URL, relevant request context and validator associated with the last successfully accepted body. A validator without its corresponding cached content is not enough to reconstruct a usable observation.

Keep market, language and other representation-changing inputs explicit. A page can vary across those conditions, so reusing one cached body for every regional view can create incorrect results.

Do not treat an ETag as a product identifier or a human-readable revision date. Preserve its role as a source-provided representation validator.

Handle conditional responses deliberately

Use the client's supported conditional-request mechanism and follow the source's response semantics. A not-modified response does not contain a fresh full body for the parser to consume.

Associate it with the appropriate previously accepted representation. If that body is missing or no longer valid for the task, recover through an approved full retrieval rather than pretending the empty response is the complete page.

Keep source observation time and body retrieval time separately where the report needs both. Reusing a validated representation is not the same event as downloading its full bytes again.

Validate changed bodies before replacing the cache

When the source returns a new representation, check its format and required content before promoting it. An error page should not replace a valid cached product document silently.

Preserve failed refresh attempts as unavailable or rejected updates. Do not label old content as newly collected merely because a refresh attempt occurred.

Version the parser separately from the source body. A new parser can change extracted records even when the underlying representation remains unchanged.

Measure the real benefit

Track full transfers, conditional results and accepted records separately. Reduced transferred bytes can be useful, but it should not hide stale or invalid data.

Compare equivalent source conditions. A change in market or response encoding can alter representation behavior and make a simple before-and-after bandwidth comparison misleading.

Do not assume every source supports validators consistently. Preserve a supported fallback and a bounded schedule for sources that do not provide the needed behavior.

A missing cache example

Imagine a worker retains an ETag in a small database but loses the corresponding body during storage cleanup. The next conditional request receives a not-modified response, and the parser reports an empty product record.

Detect the missing body before accepting the conditional result. Retrieve the approved full representation under a bounded recovery path and restore the validator-body relationship.

The proxy connection may be functioning correctly throughout. The defect is an incomplete cache design, which is why response status alone cannot define a successful incremental observation. The guide to website change-detection workflows explains how stable identity, preserved evidence and confirmation rules make a reported change reviewable.

ETag questions

Does an ETag identify the product itself?

No. It is a representation validator under the source's behavior. Keep product identity in the appropriate application fields.

Can a not modified response be parsed as an empty page?

No. Associate it with the correct accepted representation or recover the missing body through a supported request.

Does unchanged content mean the parser output cannot change?

No. A parser update can alter extraction. Preserve both source and processing versions.

Keep conditional retrieval tied to usable content

Manage validators, cached bodies and observation context as one relationship. That lets an incremental scraping proxy workflow save transfer work without silently weakening data quality.

Start a free proxy trial
Contact us