Implementing a Modular Monolith: Questions on Code Organization and Boundaries

This message was imported from the Ruby/Rails Modularity Slack server. Find more info in the import thread.

Message originally sent by slack user U70NCVWK9T0

Hey everyone. Thanks so much for the great talk @AlexEvanczuk. I’m an engineer on an infrastructure team at a company called Fullscript. This topic is near and dear to my heart. I spent roughly 6 months writing proposals and meeting with leadership to come up with a plan to de-risk and implement our move to a modular monolith. Luckily everyone is on board with the plan and we’re going to start it later this year. Rough plan tracks pretty consistently with your talk Alex.

• First step, move all the code into packages and get it to a place where everything is owned by one team. (I suspect the tricky part here will be splitting up our god objects that belong to multiple teams).
• Then start setting up and enforcing boundaries across the domains.
I have a few questions though! Was curious what folks are doing for interfaces… Do you have a common library that helps shape consistent apis across the app or do you let teams come up with their own solutions? Also what do you do about ActiveRecord (or is this a concern?). Mostly worried about chaining associations that would cross boundaries (store.owner.clerks.etc ) and folks calling active record methods (.where, .all, .update, etc.) instead of the public apis.

Also gotta say I really appreciate setting up this slack channel so we can share our challenges / solutions together. These are exciting times!

Message originally sent by slack user U783MOJYF8Z

At Shopify, we tried pretty specific conventions for interfaces but ran into lots of problems with them, as it’s hard to find a good “one size fits all” recommendation. We ended up with people using patterns they didn’t understand, and thus actually decreasing code quality and readability.

We now only have pretty general guidelines. Packwerk gives us the convention of app/public folders; active records shouldn’t be public, and we want everything public typed with sorbet (this also gives packwerk sharper teeth, as it understands sorbet’s signatures) and documented. An API site with documentation is then autogenerated. In addition, we’re using typed IDs (not rails’ standard integer IDs) for more type safety.

Packwerk also enforces acyclic dependencies which has a major influence on interface design, encouraging dependency inversion.

I’ve always thought that at some point we’d start preventing transactions across components, but the pain has not been strong enough to do that so far.

Sorbet signatures on package interfaces combined with the use of packwerk and the convention of having active records always be private is in our case enough to prevent accidental usage of Active Record Relations (your store.owner.clerks example).

Message originally sent by slack user U783MOJYF8Z

Note that Packwerk understands AR associations - we try to follow packwerk and not use associations between packages except in some special cases

Message originally sent by slack user U70NCVWK9T0

Thank you! This is super helpful and insightful.

Message originally sent by slack user U70NCVWK9T0

We don’t use sorbet (but anything’s possible!). I really think having some way to document these apis will be critical.

Message originally sent by slack user U783MOJYF8Z

We used a in-house library for dynamic type checks before, which was great, until we realized that it was too slow and not expressive enough. Sorbet is much better at this.

Message originally sent by slack user U783MOJYF8Z

Maybe another useful note…

At some point we built a generic query library to allow flexible queries across components (/packages) without leaking implementation details. It ended up being pretty similar to GraphQL.

It has seen limited adoption and caused a few performance regressions. At this point I think it’s theoretically great but in practice it hasn’t gained us much. And we’ve invested a lot of work in it.

Queries are now usually way more manual, with specific queries exposed on boundaries through dedicated classes. Callers need to assemble things like joins themselves. However, I think at some point in the future we’ll probably have some sort of cross-package batch loading mechanism. Maybe it’s an extension of the query library that we already have, or something more like graphql-batch.

Thanks for sharing these questions and your plans Andrew.

I suspect the tricky part here will be splitting up our god objects that belong to multiple teams

This is probably something for a separate conversation thread, but TLDR what we did was use packwerk to block new dependencies on our god models and ask folks to invert their dependencies on god models, as Philip suggested.

Similarly, we provide guidance to not use AR associations for models across packs (i.e. only use AR associations as an implementation detail within packs). This has been a hard convention to push because of what we’re so used to and comfortable doing.

We don’t use sorbet

Another separate conversation thread here but sorbet has been incredible for us at Gusto and would strongly recommend it :slightly_smiling_face: Happy to chat more about it

Lastly, regarding standard APIs and interfaces… we’re actually thinking about this right now. As Philip also noted, it’s been hard to suggest a well-liked and well-used technical/tactical approach to API design. Rather, we’ve been pushing API standards (i.e. use plain Ruby, but follow these standards). I’m actually thinking of “open sourcing” some of the internal guidance I’ve published around pack API/internal structure, if folks are interested.

Message originally sent by slack user U783MOJYF8Z

not use AR associations for models across packs (i.e. only use AR associations as an implementation detail within packs). This has been a hard convention to push because of what we’re so used to and comfortable doing.

This is pretty much what aggregates are in DDD, maybe that helps people understand the concept?

Message originally sent by slack user U783MOJYF8Z

I mean, you access an aggregate by its root, and only things that belong to the same aggregate are associated with the same root.

That means your associations form a forest of separate trees instead of a potentially fully connected graph.

Aggregates also serve to define transaction boundaries, which makes a lot of sense at least for the larger packages that we have

Message originally sent by slack user U70NCVWK9T0

I guess I’m confused by the use of AR objects themselves. Like what is preventing me from going through a public API that returns an object from the db and then calling .where("custom_sql_that_couples_everything_together") . In our app for example most things belong_to a store and have a store_id. In theory I can join to almost anything in the app through that store id.

Ah… there are a couple of mechanisms preventing this. The main thing we do is denote all AR records as “private” in the eyes of packwerk. Using the public folder from packwerk, everything is private by default.

At Gusto, we’re trying to take this a step further, and create a “package protection” that enforces that AR is never exposed at the boundary.

That is — the model itself is private, but with packwerk by default that doesn’t prevent you from returning it at the boundary. However, if you combine the two package protections: (1) all public API is strictly typed using sorbet (therefore all public API has method signatures) and (2) a package protection to ensure AR is not returned at the boundary, it gives some guard rails against this.

Message originally sent by slack user U70NCVWK9T0

That makes sense! I think the thing I was missing was the method signatures from sorbet. I’m very curious what you come up with!

Message originally sent by slack user U783MOJYF8Z

Exactly, that’s what I meant when I said sorbet gives packwerk sharper teeth. Any convention that states types explicitly in Ruby code would do that though, which is why our earlier approach of dynamic validation worked as well (declarative type statements triggering dynamic validation)