The way you organize source code in your projects has significant impact on the ergonomics of working with it. This is especially true as the project grows bigger in size and functionality. Properly structured project will make refactoring and changing functionality easier. A big ball of mud - will make your job more difficult.

In this subjective guide, I will walk you through what I found working well for me, when it comes to structuring Elixir and Phoenix projects.

A big ball of mud. One iteration at a time.

The worst thing you can do is to corner yourself into working with a big ball of mud: a system that lacks any perceivable architecture. I have done that several times, usually for the same reasons anyone else would: pressure from the business owners, lack of time, budget and resources. I also think certain tools I used in the past do make it really easy to structure projects this way.

The runtime of - say - Ruby is very much different from the runtime of Erlang. Things usually happen sequentially, and concurrency is rarely employed. As a result, there are no clear runtime boundaries, that would make it easier for us - developers - to draw the borders for our architectural components. Yes - we can force ourselves to make things better organized, create new directory structure, make sure our components interact only within designed borders - but there’s nothing in the language nor the runtime that would help us enforce it.

As a result, with increasing business pressure, we tend to bend the rules, put more and more of the code in the same place, slowly building unmanageable ball of mud - one step at a time.

Luckily for us, in the brave new world of Elixir we are given the tools, and it would be a shame not to use them.

Umbrella projects & OTP applications

Erlang comes with the concept of OTP applications. It is a simple - yet neat way to deploy, run, and manage multiple logical pieces of functionality within single BEAM instance, or a cluster of instances.

OTP applications can be either in a form of library applications - groups of modules exposing functionality to the rest of the cluster, or in a form of runtime entities - by starting own supervision trees.

While both are fine ways to group pieces of related functionality together, the second form is arguably more useful for structuring your application. It is future-proof, if you need to scale from single instance to a cluster of nodes, it also forces you to think in terms of business processes rather than entities defined by the data. This is something that struck me as odd as a newcomer from object-oriented programming, where we would - almost without thinking - structure our code in classes, wrapping some data around.

Elixir comes with a handy feature allowing you to easily use OTP applications in your project - an umbrella project.

To create an umbrella project, simply run:

mix new my_app --umbrella

And it will create an overall directory structure for you. Within apps/ folder, you can generate a number of OTP applications (with mix new appname or mix phx.new appname). You can also specify dependencies between those applications, to ensure they will be started in the proper order (with in_umbrella: true in respectful mix.exs files).

Whenever you start the project from the top-level directory of the umbrella project, Elixir would build the dependency tree of the OTP applications, and start them up in an order that ensures each app has its dependencies started before it is brought up.

OTP applications are my favorite way of breaking down Elixir projects into smaller chunks. I usually deal with web apps, and Phoenix is the tool I use a lot, but the process starts by creating an umbrella project. Then I would generate an ui app for the front-end, admin app for the back-end and core app for the business logic, breaking it down further as I go.

Having done that, you can still insist on building piles of mud within your individual OTP applications, or make one large pile of mud by tightly coupling them all together. The question remains if it is worth doing so in the long run.

OTP applications as microservices

Yay, a buzzword. Microservices. I could also squeeze in SOA I guess, but that might be the thing of the past already.

In previous section I mentioned that you can start the whole umbrella project from its root directory, and that’s very much useful. What you can also do, however, is to start the individual OTP applications by going to their respectful directories in apps/ and starting them individually. The same applies to running tests or mix tasks - issuing command to start those would run them within the scope of individual OTP applications - not recursively within umbrella project.

Let’s consider a custom e-commerce platform, consisting of three user-facing components: user interface for customers to purchase goods, admin interface for store owners to manage the inventory, and API for external services to read and query product database. One could structure such system using Elixir and OTP applications to consist of the following components:

• ui - our store front
• admin - administrative dashboard
• api - API layer for other machines to interact with
• db - bottom-level database access layer
• inventory - business logic related to querying and updating inventory
• uploads - handling file uploads, such as product images
• transactions - charging customers, interacting with payment gateway

Elixir comes with a handy mix task to print the dependencies between apps in umbrella project. Simply run mix app.tree --exclude logger --exclude elixir from top directory of the project and you’d see:

==> uploads
uploads
==> db
db
==> checkout
checkout
└── db
==> inventory
inventory
└── db
==> transactions
transactions
└── db
==> api
api
└── inventory
    └── db
==> ui
ui
├── transactions
│   └── db
├── inventory
│   └── db
└── checkout
    └── db
==> admin
admin
├── uploads
└── inventory
    └── db

The tree above, being read from top to bottom, stopping at lines marked with ==> also shows us in which order the individual OTP applications will be started. As we can see, first the applications having no dependencies - leafs in the tree - will be brought to life (uploads and db), then applications directly depending on those, after that another layer and so on.

Do not cross more than one layer boundary

The rule of thumb I use when structuring applications in this way, is to allow the top-level applications to directly communicate with layer below, but any communication crossing more than one line is forbidden. As an example, ui can retrieve products using inventory, that would query them directly in the db. Reaching out from ui to db is - however - strictly forbidden.

OTP applications can further help us with adding extra layers of abstraction. Our uploads application could be only a front-end to another aws_uploads application. If we keep the strict rule of not crossing more than one border when communicating between applications, we could then replace aws_uploads with ftp_uploads or google_cloud_uploads - without having to make a single change in layers above uploads.

The other interesting thing we can do in such set up, is to test the applications in isolation. If we want to test inventory without starting a SQL database and populating the data, we could do that. We can easily mock the interface of db, using the new mox for example. This gives us freedom to develop the individual applications separately, which comes in exceptionally handy during the times of refactoring. If you keep all your code in one Elixir app, big refactoring tasks are difficult because the application will not even compile until you make a significant amount of changes, and it is a big conceptual overhead to keep track of all the requirements at once. OTP applications make the code bases you work with smaller.

Directory structure and naming things

I like to keep the directory structure aligned with naming of things. For our e-commerce application from the previous section, I would go with a top-level module namespacing schema that respects names of individual applications. This maps nicely to Erlang programmers practice of prefixing modules with application name too.

In the ui application, all of the modules would start with Ui. prefix. For example Ui.RegistrationController.

Since I do not keep any complex business logic in the Phoenix applications, I would not create a separate web subdirectory and a UiWeb namespace. Instead, I would put all of my controllers in apps/ui/lib/controllers. Everything in Phoenix apps is concerned with “web”, there is no need to separate it any further from “non-web” stuff.

The rule of thumb that’s worth following is trying to keep the directory structure as flat as it is possible and manageable. You don’t really need to type more than necessary when opening/saving these files - as long as there is sane number of files in a directory. Ui.RegistrationController is in my humble opinion way better than Ui.WebUi.Accounts.RegistrationController. There is no need to multiply namespaces for the sake of multiplying namespaces.

If you keep the module names reasonably short, you can get away without aliases in your code. If you don’t use aliases much, and especially avoid alias XXX, as: YYY construct, your code is more understandable.

I learned that there is also no need to wrap all the modules in the whole project in a top-level namespace. There’s no need to have MyECommerce.Ui.RegistrationController and MyEcommerce.Db.Account. Ui.RegistrationController and Db.Account are good enough.

Server and client don’t belong together

One of the things I never liked in Elixir, and it seems I was in good company was the fact that in most of the tutorials and documentation, client and server implementation for processes were coupled in the same module. In classical example, GenServer would first define public API, then all of the callbacks.

It does not have to be like that, and I think it is hurtful in most cases if you write such code.

If you have layers of OTP applications in place, you may want to expose certain features in form of servers, usually using GenServer or similar OTP behaviour. Going back to our e-commerce example, one could expose file uploads functionality as such service. I mentioned before, that uploads can be further split into uploads and aws_uploads. The good practice in such case would be to put the client API in the uploads, and create processes for services in aws_uploads, implementing only life cycle functions and callbacks.

This breakdown will come in handy when you think of more complex deployment scenarios. For example, you could deploy your admin and aws_uploads applications to two different nodes in the cluster. The node running admin would also need to load the code from the uploads, but it may be completely unaware of the implementation details of concrete uploads mechanism defined in aws_uploads running on remote node.

Summary

Elixir comes with super cool concept of OTP applications, inherited from Erlang. Do use it. You can solve any problem with extra layers of abstractions (except the problem of too many layers of abstractions)!

Post by Hubert Łępicki

Hubert is partner at AmberBit. Rails, Elixir and functional programming are his areas of expertise.