Feedback and Contributions Welcome for Rust Implementation of Packwerk: Packs

This message was imported from the Ruby/Rails Modularity Slack server. Find more info in the import thread.

Hi folks! I’ve been working on a rust implementation of packwerk I call packs: https://github.com/alexevanczuk/packs

It’s far from complete, but I wanted to share it in in the rough version it’s at now! All it can do right now is pre-generate a cache for the ruby implementation, but I’ll hopefully be able to improve the value as I have more time.

Feedback and contributions welcome!

Message originally sent by slack user U717FXJ8HWK

Nice! This just reinforces my opinion that writing tools for Ruby in more performant languages provides a much better developer experience. (e.g. Sorbet is written in C++ with performance in mind) :smile:

Message originally sent by slack user U717FXJ8HWK

https://github.com/charliermarsh/ruff also comes to mind as an example of a similar trend

Message originally sent by slack user U71VEMF3PJI

Hey Alex!

First of all, thank you for building so much amazing software for the Ruby community and for pushing the packwerk initiative forward. :pray:

A few questions for you (if you don’t mind):

  1. Is the goal to replace the Ruby implementation of packwerk with a Rust implementation?
  2. What is the motivation behind implementing packwerk in Rust? (Is it primarily performance?)
  3. Packwerk is a tool for enforcing boundaries in Ruby apps. Do you think that implementing packwerk in a language other than Ruby will result in less contributions from the community? (All Rubyists know Ruby, not all of them know Rust)
    Thanks! :slightly_smiling_face:

Thanks for the questions…

  1. Maybe! I think it could work too for packwerk to have a rust core (i.e. for parsing files and building the tree of references) and a ruby shell (for checkers, validators, etc.). I think there could be value in having the rust implementation do everything the ruby one does. Whether it’d be a replacement versus an alternative I’m not sure! Still exploring.
  2. Performance is a big one, since the ruby implementation can only get so fast, but the rust implementation can be so much faster. This makes it more useful as an editor integration (e.g. via an LSP) to give folks instant feedback. I also find Rust easier to work with than Ruby in a lot of ways. Besides that, rust is memory-safe (the current implementation takes up a lot of memory and could have memory leaks) and great for making multi-threaded applications (which is perfect for a CLI tool like packwerk that is bottlenecked by IO).
  3. It’s hard to say! I think there is a lot of value in learning a language like Rust, and there are great resources. Learning rust has already made me a better ruby developer. We can compare to sorbet as Oleg mentioned, which has had a ton of contributions from the Ruby community even though it was written in C++. In theory this implementation could also help modularize non-ruby apps (e.g. typescript apps), which would grow the community, but I’m not really focused on that now.

Message originally sent by slack user U71VEMF3PJI

Thank you for the thoughtful responses! Having a faster core so that it can be useful as an editor integration makes a lot of sense.

  1. It’s hard to say! I think there is a lot of value…
    I agree with your points here, and it’s cool to think about a future where packwerk could be applied to non-Ruby apps.

I guess the main feedback I’d like to give is that using a language other than Ruby means that there is more friction for the average Ruby developer to contribute to packwerk or to dive into it and explore how it works. But - that could be an acceptable trade-off. Or maybe a Ruby implementation will continue to exist like you mentioned, making this whole point moot. :smile:

Cheers!

Message originally sent by slack user U70NCVWK9T0

Oh wow. This is exciting. One of the main challenges we have with packwerk is the feedback loop is so slow. Developers only get feedback once they’ve written their code and pushed to CI. That works but it’s not a great experience and can slow people down. Ideally it should be more like Sorbet or Rubocop where you get feedback as you’re writing. :heart:

Message originally sent by slack user U7213XMGS3H

How fast could it get? Do you think it would be reasonable to run in an IDE and highlight violations as you type? The current Packwerk is prohibitively slow for something like that

Message originally sent by slack user U717FXJ8HWK

Also wondering: is this a side project or is it part of your day job? Or in other words, how much time do you intend to spend on this?

As far as speed, I think it could feel pretty instantaneous. Might be worth nothing I think it’s possible for the ruby implementation to feel almost instantaneous if it were implemented with an always booted up LSP (but a rust version wouldn’t need an LSP, or could do more demanding things with an LSP, I think).

Right now it’s mainly a side project, with a little bit of “experimental” work time spent on it. I’m hoping to find ways to pull it out of side project status, but for the time being my main goal is that anything that this can do should gracefully degrade into the ruby implementation if the rust version can’t be used.

Message originally sent by slack user U70TIGAX94P

Would certainly be interesting to have an alternative implementation to compare against.

Message originally sent by slack user U70TIGAX94P

I also think there is room for a more general tool that checks adherence of code to a predefined dependency graph and maybe more elaborate architecture specification, and has plugins for different languages.

The main ”magic” in packwerk is in finding the dependencies in the first place, and other languages with real per-file or per-module imports don’t’t need that part.

Message originally sent by slack user U70TIGAX94P

Let me retract that, imports in python are so messed up that python might need the same magic.

Hey folks I just finished my first big milestone for this, which is getting packs update and packs check working end to end. It still doesn’t support some things (e.g. custom inflections, custom public dirs, more listed in README) that may yield different results than bin/packwerk update .

Usage directions are listed in the README. I’d love it if anyone wants to give it a try! I’d really appreciate any feedback :slightly_smiling_face:

tried it out in my project (500k LOC) and works like a charm. Incredible speed. congratz :clap:

Thanks for trying it out @santib :slightly_smiling_face: Can you confirm that after you deleted the cache (rm -rf tmp/cache/packwerk), and maybe deleted-by-hand a violation from some package_todo.yml files, you get the same results with packs update?

ok I deleted the cache, and ran packs update and be packwerk update-deprecations and compared all the packages’ deprecations. just KIM that we are on packwerk 2.0.0

differences found:
• in the root package there is one with “ruby packwerk” missing with “rust packs”.

components/stripe_client_legacy:
  "::StripeClientLegacy::APIClientService":
    violations:
    - dependency
    files:
    - app/services/stripe/api_client_initializer_service.rb
    - spec/services/stripe/api_client_initializer_service_spec.rb

• in a package called “iam_admin” there is one missing for “rust packs” as well

components/iam:
  "::IAM::Group":
    violations:
    - privacy
    files:
    - components/iam_admin/spec/features/admin/iam/group_spec.rb
    - components/iam_admin/spec/forms/admin/iam/group_form_spec.rb
  "::IAM::Membership":
    violations:
    - privacy
    files:
    - components/iam_admin/spec/features/admin/iam/membership_spec.rb
    - components/iam_admin/spec/forms/admin/iam/group_form_spec.rb
    - components/iam_admin/spec/forms/admin/iam/membership_form_spec.rb

• in another package called shop, there are 4 missing dependency violations for the “rust packs” not sharing diff because it’s too long but missing classes are: "::B2b::FindLeadForUserService" , "::Canvas::APIClientService" living in the root package, "::E18n" living in the e18n package, and "::TaxClient::APIClientService" living in the tax_client package
• in another package called university_portal there is one missing

components/iam:
  "::IAM::Membership":
    violations:
    - privacy
    files:
    - components/university_portal/spec/requests/admin/university/school_admin_user_spec.rb

• in another package called users_management there are 2 violations missing:

  "::B2b::FindLeadForUserService":
    violations:
    - dependency
    files:
    - components/users_management/app/controllers/users/passwords_controller.rb

  "::Salesforce::EETransaction":
    violations:
    - dependency
    files:
    - components/users_management/app/forms/registration_form.rb
    - components/users_management/spec/forms/registration_form_spec.rb

I guess the privacy differences can be ignored since “rust packs” doesn’t check that, right? If that’s the case, then the differences get reduced to just 7, right?

Let me know how can I help you debug this better

rust packs does check for privacy actually, so those should show up too.

It looks like these are all due to missing support for inflections (API, IAM, EE). My guess is your inflections.rb file (or wherever you store them) list some of those inflections.

Can you confirm that? I wonder if I’m able to have rust parse the ruby inflections file to support custom inflections. The two other options are:
• require hard-coding relevant inflections in packwerk.yml (only to be read by the rust binary)
• Just removing inflections from your codebase. Most users wouldn’t want to do this though (although this is what I’m doing at Gusto for now, since we don’t like custom inflections anyways).