I’m looking for advice on building a collaborative caching system for APIs with strict rate limits that automatically commits updates to Git, allowing multiple users to share the scraping load and reduce server strain. The idea is to maintain a local dataset where each piece of data has a timestamp, and when anyone runs the script, it only fetches records older than a configurable threshold from the API, while serving everything else from the local cache. After fetching new data, the script would automatically commit changes to a shared Git repository, so subsequent users benefit from the updated cache without hitting the server. This way, the same task that would take days for one person could be completed in seconds by the next. Has anyone built something like this or know of existing tools/frameworks that support automated Git commits for collaborative data collection with timestamp-based incremental updates?
You must log in or # to comment.
Can’t speak for Git, but caching responses is a common enough problem that it’s built into the standard HTTP headers.
As for building a cache, you’d want to know a few things:
- What is a cache entry? In your case, seems to be an API response.
- How long do cache entries live? Do they live for a fixed time (TTL cache)? Do you have a max number of cached entries before you evict entries to make space? How do you determine which entries to evict if so?
- What will store the cache entries? It seems like you chose Git, but I don’t see any reason you couldn’t start simple just by using the filesystem (and depending on the complexity, optionally a SQL DB).
You seem locked into using Git, and if that’s the case, you still need to consider the second point there. Do you plan to evict cache entries? Git repos can grow unbounded in size, and it doesn’t give you many options for determining what entries to keep.
If I used sqlite or any other SQL database I don’t think users could collaborate on building the database, so I was thinking of json files committed to a git repository online.
What problem are you attempting to solve?
How to you imagine this solution solves that problem?
This could easily be implemented with an SQL-like database. Are you sure you want to use Git for this? The only advantage would be that you get historic data out of the box, but you’ll probably only fetch the latest data anyways
