Collaboration on `query_packwerk` and `visualize_packwerk` for improved Packwerk data collection and visualization

Tools RubyAtScale
1

This message was imported from the Ruby/Rails Modularity Slack server. Find more info in the import thread.

use_packwerk (link) has some query mechanisms but its mostly meant to be human readable, not machine readable. I’ve been chatting with <@U78HE86J47X> about extracting some of that and other work he’s doing internally at Gusto into an open source gem called query_packwerk. It could be awesome for us as a community to collaborate on a common toolchain for collecting this data so we can build on each other’s work.

We have a tool internally called visualize_packwerk based on parse_packwerk that might be good to release as well (cc @stephan). It create a visualization using graphviz.

2

<@U70DMI6W1TM> <@U70DMI6W1TM> @iMacTia

We just released visualize_packwerk today: https://github.com/rubyatscale/visualize_packwerk

It doesn’t do a ton yet, but I hope it is a conversation starter! Even more, I hope we can improve it to make it more useful for you and others in the community.

3

<@U78HE86J47X> and I are going to look into releasing query_packwerk next week!

4

Message originally sent by slack user U70DMI6W1TM

Nice, congrats on the release!

5

Nice one @AlexEvanczuk! I’m adding it to my list to check it out!

6

Message originally sent by slack user U70TZKYOJ67

:clap: Sounds that it is exactly what I was looking for :grinning: I will check it out.

7

Awesome! Let me know if you all have any feedback. There’s a lot of room for improvement and configurability. For example, today when you give it some pack names, it renders those pack names on the graph as well as any of their dependencies. This isn’t necessarily desired behavior and maybe could be changed via a CLI. I just mostly wanted to publish what we had to get feedback and give ya something to work with :slightly_smiling_face:

8

Message originally sent by slack user U70TZKYOJ67

I had a quick look at https://github.com/rubyatscale/visualize_packwerk. Unfortunately we have some IT restrictions in using graphviz, so I won’t be able to use it. Are you aware of this one https://github.com/bellroy/graphwerk?

It seems that internally visualize_packwerk uses https://github.com/rubyatscale/parse_packwerk to read the YML files (packages, deprecated). To my understanding https://github.com/Shopify/packwerk/blob/main/lib/packwerk/package_set.rb would also allow to read the package.yml files, but there is no parser in packwerk to read the deprecated_references files.

For my needs, I might extend parse_packwerk to convert the YML data into a CSV format - although I am not sure whether the data as provided through the YML files is sufficient for my needs.

9

<@U70TZKYOJ67> That’s a bummer on graphviz… not sure why those restrictions are in place but perhaps they are amenable to removing them :thinking_face:

I think your understanding is correct. We built parse_packwerk because (A) packwerk did not yet have the available API to do what we wanted and (B) we needed to run this in production (where packwerk is currently in the development gemset and (C) we needed to run this in critical code paths, so we didn’t want the full, heavy, risky packwerk dependency in those code paths.

Long-term, ideally I’d like packwerk to use something like parse packwerk – an independent gem that represents the specification for packages that that the ruby modularization toolchain can depend on.

10

Converting the YML data to CSV could work!

ParsePackwerk.all.each { |p| ParsePackwerk::DeprecatedReferences.for(p).violations.each { |v|  puts [p.name, v.to_package_name, v.files.count].join(",") } }

That could be a good start maybe!