How to Implement Caching Layers in Web3 Products
A web3 cache is a high-speed storage layer that stores a small subset of data instead of storing complete and permanent data in databases. This allows future requests for data to be processed faster than if they were kept in the primary storage location. The web3 products cache layer allows us to reuse previously retrieved and computed data.
This layer stores data on fast-access hardware like RAM or in-memory, and it can also be used for correlation with a software component. Caching web3 data improves data retrieval by reducing the need to access the slower storage layer underneath. So, this is a way to trade speed for capacity in a blockchain network.
Why is caching necessary for web3 products?
Blockchain does not allow for real-time data extraction. In the traditional process, users call the blockchain network using RPC (remote procedure call) via the web3 library, which is not faster. Let’s say a user wants to retrieve one-year transaction data for a web3 product from blockchain; then, he will not get an appropriate query to execute the same. This issue can be solved by adding a web3 caching layer to the architecture.
Why is caching necessary for web3 products?
Blockchain does not allow for real-time data extraction. In the traditional process, users call the blockchain network using RPC (remote procedure call) via the web3 library, which is not faster. Let’s say a user wants to retrieve one-year transaction data for a web3 product from blockchain; then, he will not get an appropriate query to execute the same. This issue can be solved by adding a web3 caching layer to the architecture.
How to implement web3 caching?
The web3 caching layer can be implemented in a docker-based container. The following components can be implemented as part of the caching layer:
- Blockchain follower
- Database
- In memory database
- API endpoints
- Socket connections
- B2B APIs
Blockchain follower
To sync blockchain data, we can create a blockchain follower service. It is a microservice that processes blockchain data. For every new transaction on the blockchain network, a listener method from web3 will be used to the blockchain network. We will receive the data from the nodes on the blockchain. The data comes in raw form but can be formatted according to the blockchain structure. This data can be processed and transformed into a structured format that can be saved to the database. For faster access to data, we can also index it.
To make this caching layer more reliable, efficient, and structured, we must consider the factors mentioned in the following section while syncing.
Database
We will need a database to save the incoming blockchain data. This database will contain the structured data that can be used to query the DB. It will be useful as:
- For complex queries, it is possible to obtain the desired result
- To display graphical information, aggregate data can be pulled
- Data can be indexed using one of the DB keys. This will improve data fetch timing.
We can use PostgreSQL, also known as Postgres, as a database. This is an open-source relational database management software that emphasizes extensibility as well as SQL compliance. With proper structuring, the data can be stored in a database, so it is easy to query according to the needs of web3 products, and indexing makes it easier to make queries.
In memory database
The web3 caching layer must be faster. So, things in the web3 system that don’t change often and are frequently needed on the client side can be stored in an in-memory database. Redis can be used as an in-memory structure store, a distributed key-value database, a cache and a message broker. It also has optional durability.
API endpoints
Once we create a caching layer, we must expose APIs on the client side for data retrieval from the DB. There are two ways to define APIs.
- One is the standard way to define APIs and validate all requests manually.
- GraphQL is another modern way to expose APIs in such a large system, and it is far more sophisticated and highly performant. GraphQL can query data and manipulate APIs, and it is a runtime that executes queries on existing data. Instead of exposing multiple APIs endpoints, GraphQL allows you to interact with the database caching layer as required.
Socket connections
Imagine a scenario where the user is viewing the web3 product screen, and a transaction has just occurred that increased their balance. It will be magical if the user can instantly see the updated balance without refreshing the webpage. This is how web3 will be able to display a live, synchronized version of the blockchain state. Socket.IO can achieve this. It can be used for real-time web apps. It allows bi-directional communication between web client and server on a real-time basis. This allows the system to deliver updates from the blockchain to clients instantly. Consequently, the user feels he can see the real-time data coming directly from the blockchain.
B2B APIs
Apart from GraphQL APIs, a few B2B APIs can be exposed by the web3 caching. B2B APIs can be defined and published for specific work purposes. These APIs won’t be able to query the database directly, but they will be pre-defined. Business logic will be provided in line with the API endpoint’s requirements, and these APIs will focus more on the business requirement. The caching layer described above will help it, as B2B APIs are much faster than web3 calls to the blockchain.
Security in the web3 products caching layer
Every connection and exposed APIs to web3 caching layer should be protected with some level of security. The following methods can secure the caching layer of web3 products.
API gateway — A secure access point for an organization is possible with an API gateway. These API gateways employ industry-standard encryption and access controls.
JWT authentication — Security is essential when it comes to protecting web resources. These resources can be protected by integrating JSON Web Tokens (JWT) into web applications. JWT allows secure information to be transmitted between two parties. Any number of JWTs can be activated on a blockchain.
Session validation — There are several ways to manage user sessions. You can either save them locally on the response node to your request or create a layer in the architecture that can store them in a secure and scalable way. You have many options for storing sessions in key/value storage. Many application frameworks provide libraries that abstract some integration plumbing and store these sessions in memory. We can also create our session handlers to keep the session alive.
Conclusion
A web3 cache is crucial for the business aspects of web3 products. By reducing the number of iterations required to request and deliver content, web3 caching can improve network utilization. This reduces the need for redundant infrastructure deployment. This results in significant cost savings and economic benefits for the entire internet ecosystem. It is also very energy efficient. Commercial caching providers can also operate on a large scale, using extensive infrastructure shared between multiple customers.
