How does caching work under the hood for DXP sites? (on default setups)
Default DXP sites do not use a distributed cache (such as Azure Managed Cache Service or Redis - agreement must be made with episerver for this to be added). Each running application has its own instance of an in-process cache. When an item is evicted from cache in one running application instance (e.g. due to an content update) a message is sent to the other running instances (in DXP the message is sent through Azure Service Bus). The other running instances will then evict the cached item from their local caches when they receive the message.
The synchronization part of the ISynchronizedObjectInstanceCache only means that it has support for removing cached objects across multiple servers. It is not a distributed cache, so each server is responsible for populating the own (memory) cache.
When a cache item expires in one application it will not send any message to evict it from other application (e.g. items can be evicted from an application due to memory pressure).
For content there is a common key specified by IContentCacheKeyCreator.RootKeyName, every content item has a dependency to this key so by removing it all content items gets flushed from cache. Then there is also a key IContentCacheKeyCreator.VersionKey that is updated each time some content in e.g. deleted, published. It is used e.g. to clear output cache (where you do not really know exactlly which content items the generated html refers to).
When a content item is published/removed IContentCacheKeyCreator.VersionKey is updated.
Note: If custom items are cached with ISynchronizedObjectInstanceCache.Insert and removed with ISynchronizedObjectInstanceCache.RemoveLocal, the changes will not be synchronized across instances. .Remove must be used for this.
Content listings have same dependency to IContentCacheKeyCreator.RootKeyName and IContentCacheKeyCreator.VersionKey
Each application will clear items from cache when they receive a message, normally it is approximately the same time but can differ if e.g. one site is very busy.
Items that are removed due to time expirations or evicted due to memory pressure are not tracable (the removal is done from ASP.NET and not CMS). Removals done from within CMS can be traced by e.g. implementing an interceptor -see section "Intercepting existing services" in http://world.episerver.com/documentation/developer-guides/CMS/initialization/dependency-injection/ for ISynchronizedObjectInstanceCache and/or IObjectInstanceCache (the cause can probably be seen by looking at StackFrame). Note however that such a trace might impact upon performance so one recommendation is to only use it temporarily for fault tracing or such tasks.