Debouncing Events

General
1

We’ve adopted to use an event bus in our modular monolith, and have seen a very positive return on that investment (in managing dependencies between packs, and the system capabilities enabled).

However, as we analyze our system behavior at runtime (under real traffic), we’ve observed a situation with which folks here might have experience.

For context, we’re building an Ads platform… to manage the sale of advertising space as well as to manage the delivery of those ads with other vendors.

We have fine grained things (like an Ad) and course grained things (like an Order, containing 1000s of Ads). When 1 Ad changes, an event is fired… then an event handler will execute to update an Order (eg, the total price of an Ad changes, triggering an update in a total price of a pending Order); that will in turn generate an event announcing the Order has updated.

So, Update Ad => Ad Updated Event => Update Order => Order Updated Event

However, when all 1000 ads in an Order change, you can guess what happens: 1000 updates of the order (999 are needless), and 1000 events announcing the Order was updated (999 are needless). Those 999 superfluous events are causing processing delays of other events.

Questions for the community:

  • do you have an event bus in your modular monolith?
  • did you find a need to debounce certain events?
  • does your event bus technology (eg, RabbitMQ) already support this requirement?
  • did you build your own? If so, describe your solution.
1 Like
2

Yes, yes, no-ish (Kafka no, sidekiq eh), yes

@technicalpickles knows a lot more!

3

Can I ask what it is about your ads changing that requires the order to change also?

I don’t want to know anything sensitive, but it sounds like there might be one or more data items shared between these two entities (I don’t mean that as a technical term) that might not need to be shared.

For example, I might save both a Customer name and a Customer id in an Order. But I might not change the name in the Order even if the name is later changed in the Customer record. The reason for this could be to preserve the Order history: this was the name of the person that placed the Order. But I can still look up the Customer’s current name using the Customer id in the Order. Of course, there could also be valid reasons to update Customer name in the Order. This is just an example.

It sounds like something common to all the ads is changing in exactly the same way. Is it possible that this piece of information could be kept only in the order (or somewhere else) and changed once there? For example, if the ads all have the same price, could the price live in just in the Order?

If the price must be recorded in both Ad and Order, then maybe the canonical price could be kept somewhere else, like a pricing table, and changes to this entity trigger change events to both Ads and Orders. Again, price is just an example.

I guess what I’m asking is if it would be possible to change the boundaries between these two concepts (and possibly add a third concept) to eliminate this issue.

I hope this is helpful!

2 Likes
4

I’d love to hear from @technicalpickles as well.
For others who might be interested, we did implement an event debouncing feature. We built it using NodeJS and UNIX sockets. Our RoR code indicates that publishing an event should be debounced with an extra argument:

line_item.publish_updated_event(:ad_metrics, debounce: true)

The event itself uses a concern to make it debounce-able:

class Orders::Events::CPMLineItemUpdatedEvent < CoreEvents::Event
  include CoreEvents::Debounced

  attribute :resource_id, Types::String
  attribute :changed_attributes, Types::Array.of(Types::String)

  def debounce_key
    "#{event_name}:#{resource_id}:#{changed_attributes.join(',')}"
  end

  def debounce_timeout
    3
  end
end

The event bus module sends the event to the NodeJS debounce service in such cases:

module CoreEvents
  class EventBus
    def publish(event, debounce: nil)
      if debounce && Rails.configuration.debounce_event_service_proxy.present?
        Rails.configuration.debounce_event_service_proxy.debounce_event(event.debounce_key, 
                                                                        event, 
                                                                        event.debounce_timeout)
      end
      ...
    end
    ...
  end
end

The Ruby proxy for this service isn’t too suprising:

require 'socket'

module CoreEvents
  class DebounceEventServiceProxy
    def debounce_event(descriptor, event, timeout)
      payload = {
        type: 'debounceEvent',
        data: {
          timeout:,
          descriptor:,
          klass: event.class.name,
          attributes: event.attributes
        }
      }
      socket.send ActiveSupport::JSON.encode(payload), 0
    end
    ...
  end
end

The NodeJS service does what you’d expect… setting (or resetting) a timer specific to this event (using descriptor as the unique identifier).

debounceEvent({ descriptor, attributes, klass, timeout }) {
    if (this._timers[descriptor]) {
      clearTimeout(this._timers[descriptor]);
    }

    this._timers[descriptor] = setTimeout(() => {
      delete this._timers[descriptor];
      this.publishEvent(descriptor, { attributes, klass })
    }, timeout * 1000);
  }

And then sends it back to Ruby when the timer finally fires.

We were able to setup automated (integration) tests to ensure this back-and-forth actually works.

HTH

gap

5

That work has been extracted to a separate gem: GitHub - Flytedesk/debounced: Scalable debounce service for Ruby

https://rubygems.org/gems/debounced

Let me know if you find it useful.