Synchronous dispatch through a persistent broker
After receiving a message from a producer, the broker dispatches the messages to the consumers, as follows:
-
The broker pushes the message into the message store. Assuming that the
enableJournalDiskSyncsoption istrue, the message store also writes the message to disk, before the broker proceeds. -
The broker now sends the message to all of the interested consumers (but does not wait for consumer acknowledgments). For topics, the broker dispatches the message immediately, while for queues, the broker adds the message to a destination cursor.
-
The broker then sends a receipt back to the producer. The receipt can thus be sent back before the consumers have finished acknowledging messages (in the case of topic messages, consumer acknowledgments are usually not required anyway).
Concurrent store and dispatch
- Concurrent store and dispatch is enabled, by default, for queue.
- The bulk of the data is stored in rolling journal files (data logs).
- Where all broker events are continuously appended.
- In particular, pending messages are also stored in the data logs.
- BTree
- In order to facilitate rapid retrieval of messages from the data log.
- The complete B-tree index is stored on disk and part or all of the B-tree index is held in a cache in memory.
So, the secrete of the KahaDB is the rolling data storage and BTree indexing, which is good for fast appending and fast deleting, but not good for storing data, which is something the traditional RDBMs try to avoid.
I designed some similar storage solutions for some three system, based on different read/write requirements. I tried a flat indexing file with a BTree storage. I also tried an in-memory indexing. The similar part is: we need to separate the indexing from massive data storage.
