In the world of modern applications, performance is important. Users expect fast responses and seamless interactions. One way to improve application performance is caching. Caching is a broad term and has its own meaning in different contexts. In this blog post, we focus on data caching which involves storing frequently accessed data in a faster and readily available location, reducing the need to repeatedly fetch the data from slower sources like databases.
There are various caching strategies, and each has its own advantages and drawbacks. Choosing the right caching strategies requires carefully consisderation of factors such as data access patterns, update frequency, application architecture, etc.
What is Cache?
Here is the definition of Cache according to Wikipedia:
In computing, a cache is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.
Another simple definition:
Caching this the process of storing frequently accessed data in a fast, in-memory, temporary storage to improve data access performance.
Querying data from databases can be slow in some situations. By using cache, when the data is accessed for the first time, it is stored in the cache. Next access requests to the same data will be served from the cache which reduces the load on the main databases.
General Flow
When data is requested, here is the the most basic flow:
- The system first checks the cache to see if requested data is available. If data is available (cache hit), it is returned directly from the cache.
- If the requested data is unavailable (cache miss), the system retrieves it from the database, and stores it in the cache for subsequent requests.
A high cache hit rate means the cache works efficiently. The cache hit rate is the number of cache hits divided by the number of cache requests.
Cache Storage
Cache typically stores frequently accessed data in memory. Database typically stores data on disks. Memory is usually more expensive than disk, so cache only stores a subset of application data. Cache is not designed to store long-term data. As a result, cache storage is not durable. Cache is optimized for heavy application traffic and high concurrency.
Cache Invalidation
If cached data is not accessed after a period of time, that data may become stale. Cache invalidation is an important process to maintain data consistency between storage and cache. When data is modified, updating cache and database simultaneously can be challenging. Stale cache data may lead to incorrect results and potential performance issues.
Below figure demonstrates common cache invalidation strategies:
Invalidation on Write: This strategy involves invalidating the data in cache when the application modifies the data in database.
Invalidation on Read: In this strategy, the application checks the correctness of the data when it is retrieved from the cache. If data is stale, it is read from the database and written back to the cache. This strategy is flexible, but it increases application complexity.
Time-to-live (TTL)
Each cache item is associated with a TTL (Time-To-Live) which specifies how long before the cache item is expired. When the cache item is expired, the application can fetch the latest data from database and update the expired cache value.
If TTL is too short, we will have a lot of cache misses and the application needs to fetch the data from database frequently. As a result, the application performance may degrade. If TTL is too long, cached data may become stale.
So, what value should we choose for TTL?
It depends!
We need to evaluate the change rate of the application data, and make sure the application can handle stale data. For example, we can have longer TTL for static assets.
We can add randomness to TTL (jitter) to avoid multiple cache items expire simultaneously after a period of time. This can introduce a peak in your application load and reduce database performance. Rather than having fixed TTL for all cache items, we add an offset to TTL.
TTL = Orginal TTL + Offset (Jitter)
Cache Replacement
There is a limit to cache storage, so it is impossible to store all application data in the cache. The best way is to store the most frequently accessed data in the cache and discard other data. There are a lot of cache replacement policies, below are common ones:
- LRU (Least Recently Used) Eviction: if the cache item has not been accessed after a period of time, it can be a candidate for least recently used items. Those items are removed when cache reaches its maximum capacity.
- LFU (Least Frequently Used) Eviction: if the cache item has not been accessed for a period of time, its access frequency is small. As such, it becomes a candidate for cache eviction.
- RR (Random Replacement): remove cache items from the cache randomly.
- MRU (Most Recently Used): remove the most recently used data.
- FIFO: remove cached data in the order that it was added to the cache.
Cache Aside
This is a commonly used caching strategy. As depicted in the figure below, the cache and the database operate independently. Our business service (application) manages read and write operations and ensures the consistency between the two.
- Read operation: the application checks the cache first. If we have a cache hit, the cached data is returned to the application. Otherwise, the application reads data from the database and updates the cache.
- Write operation: there are several methods that we can use to handle write operations
- Write to the database and invalidate the cache: this method evict the cache right after the write to the database is successful. The next read operation will retrieve the data from the database and update the cache. This works well for read-heavy application.
- Write to the database and update the cache: instead of evict the cache, we update the cache immediately after writing to the database. This method reduces the cache misses and maintains data consistency. However, the write operation latency increases as we need to write to both cach and database. This method also works well for read-heavy application.
General steps:
- The application requests data from cache using a key.
- If we get a cache hit, the cached data is returned to the application immediately.
- If we get a cache miss, the application requests data from the database.
- Database returns the data to the application.
- The application writes the data with the key to the cache for subsequent requests.
Pros
- If cache layer crashed, the application still works as they are independent. The application now requests data from the database.
- We can load data into the cache on demand.
- We can scale the caching servers independently of the database to distribute the load.
- The data format in the cache can be different from the database, providing flexibility when building application.
Cons
- Controlling cache in application code needs more effort.
- Can introduce frequent cache misses if ther are frequent updates in the database.
- Can increase latency for write operation, so this strategy is inefficient for write-heavy application.
Write-Around Cache
When using Write-Through or Write-Behind caching strategies, the cache may contain infrequently accessed data. Write-Around can be used to resolve that issue.
In a write operation, data is written directly to the database, bypassing the cache. The data is not stored in cache at this point. When a read operation is made to the cache, if there is a cache miss, the cache will fetches data from the database, updates the cache storage, and returns data to the application.
The application may reads stale date from the cache as we only update the database. We only update the cached data in next read request. How can we resolve this issue? One possible solution is to update the database and invalidate the associated cache entries. With this, we force a cache miss in the next read request.
In practice, we should use Write-Around together with other strategies like Read-Through or Cache-Aside.
General steps:
- The application writes data to the database directly.
- After data is written to the database, one of the following steps may be taken:
- Write to the cache using the key. This may lead to conflicts due to concurrent processes update the same key.
- Invalidate the cache key. Subsequent read requests result in cache miss, fresh data will be fetched from the database.
- Using TTL mechanism to invalidate stale cached data.
Pros
- Improve cache efficiency by preventing caching of infrequently accessed data.
- Work best for read-heavy applications.
- The data inside the database is always up-to-date even in the case of cache failure.
- Prevent redundant cache evictions and reduce the loads in cache.
- Decouple the cache and database systems.
Cons
- Recently written data may result in a cache miss in the next read operation.
- Cached data may become stale if the cache entries are not invalidated after write operation.
- Not a good option for write-heavy application as there're too many cache misses.
- Cannot reduce loads for the main database.
- Need to understand data access patterns in order to determine frequently/infrequently accessed data. Otherwise, Write-Around cache is not optimal if we store infrequently accessed data in the cache.
Write-Through Cache
Write-Through cache is similar to Read-Through cache; however, the cache is responsible for handling write operations.
The application first writes data directly to the cache, which then updates the underlying database synchronously. The write operation is considered complete when both cache write and database write are complete. With this strategy, we can ensure the data consistency between database and the cache.
For read operations, we can use a similar approach as Read-Through cache. Combining Write-Through and Read-Through can result in fast read operations and data consistency. However, the latency of the write operations increases which is a drawback in this cache strategy.
We still need to implement cache eviction policy even if the data in cache is the same as in the main database. The data access patterns may change and the cached data may become infrequently accessed entries, so we should not store data in the cache idenfinitely. The cache storage is also limited, and the data inside the database could be updated by external parties (the data in cache and in the main database are out of sync).
Pros
- Read operations always see the latest version of data, avoiding data stale.
- The application only needs to write data to the cache, reducing application complexity.
- Reduce data loss when application crashed or application server is down because the data inside the cache is in sync with data in the database.
Cons
- High latency for write operations, not suitable for write-heavy applications.
- Cache failure impacts application availability and performance as all read/write operations must go through the cache first.
- If data is written tot the database but not updated in the cache, the cached data becomes stale.
Write-Behind Cache
Write-Through cache introduces high latency issue for write operations. Write-Behind could be used to resolve that issue.
Write-Behind cache operates similarly to Write-Through cache, but with a key difference: data is written to the database asynchronously instead of synchronously. With this mechanism, we reduce the load for the main database, but still make sure read operations see the latest data. Write operations to the cache is fast, so this strategy also reduce the write latency.
Pros
- Improve write operations performance, suitable for write-heavy applications.
- Can be used in combination with Read-Through or Write-Behind strategy to handle mixed worloads.
- Reduce the load on the main database.
Cons
- Data in cache and the database can be out of sync.
- Cache failure results in data loss if not handled carefully.
- Write operations to the database may fail which could result in data inconsistency, we can implement retry mechanism to overcome this.
Read-Through Cache
In this caching strategy, application does not manage data in the cache. It is the responsibility of the cache to handle cache misses and communicating witht the database for data retrieval.
When your application requests the data, the cache is checked first. If the data is available (cache hit), it is returned to the application immediately. If not (cache miss), the cache retrieves data from database, stores the data in cache, and returns the data to your application. Subsequent requests to the same data will be served from cache.
Write operation in Read-Through cache is similar to other cache strategies including: Write-Around, Write-Through and Write-Behind. You need to consider your application requirements and choose an appropriate method.
General steps:
- The application requests data from cache using a key.
- If we get a cache hit, the cached data is returned to the application immediately.
- If we got a cache miss, the cache requests data from the database.
- The cache retrieves data from database, writes to cache using the key, and returns data to the application.
Pros
- Reduce read latency for frequently accessed data, suitable for read-heavy application.
- Can load data into the cache on demand to avoid redundant data in cache.
- Reduce the number of requests to database as data can be served from cache whenever possible.
- Simplify application logic as it does not need to manage in cache.
Cons
- Result in a cache miss if data is requested for the first time and cached data is expired (TTL exceeded).
- Infrequently accessed data may stay in the cache which consumes storage for frequently accessed data.
- Data may become stale in there is a change in database but the cached data is not expired yet. We can configure proper cache eviction strategy to overcome this.
- The system cannot tolerate cache failures because the cache is responsible for data retrieval process.
- The cache and database must share the same data format (data model), less flexible compared to Cache-Aside strategy.
Refresh-Ahead Cache
In Cache-Aside or Read-Through strategies, if cached data expires (due to TTL), it's reloaded into the cache only upon the next read request, resulting in a cache miss which increases read latency.
The goal of Refresh-Ahead cache is to configure the cache to reload latest data from the database asynchronously before the TTL. This can be done using a background process. With this strategy the data will be served from the cache with low latency.
Refresh-Ahead cache aims to preemptively reload the latest data from the database before it expires. This is typically achieved using a background process that asynchronously refreshes the cache. As a result, consistently available in the cache, minimizing read latency.
Pros
- Frequently read data is refreshed before expiration.
- Improve read latency as we don't need to wait the read operation from database.
- Reduce latency spikes if multiple cache entries expires at the same time.
Cons
- Adding complexity to application logic, make it harder to maintain.
- May add extra load on the cache and the database if all keys are refreshed at the same time.
- May not suitable for write-heavy applications.
