What exactly is esy?


#1

Hey :wave:,

I am already working with esy on a backend GraphQL API.
But the more I use it, the more I run into problems which lets me believe, I am doing something wrong or am not really understanding all the different parts of native development involved with ocaml / reason.

I am also really new to native, so it is probably extra hard for me to understand everything.

My questions would be:

  1. What is esy exactly?
    I always thought it was just a package-manager (like npm or yarn) but it is probably more than that, right?

  2. How do the different parts of opam, dune and esy play into each other? What are each of these tools responsible for?

  3. Is esy a replacement for dune or opam or is it just an additional layer on top, providing some benefits?

  4. What are the benefits of using esy over just dune / opam? Other than being able to consume packages published to npm?

  5. What are esy-ocaml and esy-packages and why do I need them?
    As I understand they package some often used libraries (like libffi) for esy but I don’t understand why I would need them.
    Does that mean, when I require them in my esy.json / package.json I don’t have to install them globally on my system / in my docker container?

  6. What are some good resources to learn esy / native reason?

Just to clarify, I don’t want to sound like “nobody needs this, why does this all exist”.
I am just trying to understand how this all works and plays into each other.

Thank you all for your work and this awesome community! :heart:


#2

Hi,
Glad you asked! I’d like to take some time and individually answer each of these questions. Fact is, I had the same ones when I started out and prepare some notes.

Undertstanding the Reason toolchain - compilation of notes I prepared for myself and planned to use it to explain what I planned to do to improve native development experience.
Undertstanding native toolchain in general (any language) - my notes summarising tools as I understood them


#3
  1. It’s a package manager, similar to npm, yarn, opam.
  2. Opam is the traditional package manager. Esy is a new package manager, it has support for opam packages. Dune is a build system for OCaml. You use it to define build rules with dependancies… think of it kind of like a specialized Makefile, except it makes compiling ocaml code very very convenient. It’s also very fast.
  3. I think esy is an alternate package manager and sandboxing environment, and it can be seen used as a replacement for opam. If I understand correctly, esy has support for opam packages, but opam doesn’t have support for esy packages. Also, it’s possible to use esy to develop opam packages directly, as described here https://esy.sh/docs/en/opam-workflow.html
  4. Not sure what the official reasons are, but esy provides a user experience more similar to yarn/npm.
  5. You can use esy to package non-ocaml dependencies. As you mentioned, this makes it possible to include them in the sandbox, and allows you to avoid depend on the packages that are installed globally.
  6. The official docs at esy.sh, esy bugtracker https://github.com/esy/esy/issues, both the ocaml and reason discord servers, and package.json et al. for Onivim2 (for inspiration)

#4

Indeed.

While esy does package management, the ideal native development experience needs a tool that does the following things too, along with fetching sources and making them available for use.

  1. Make sure all the transitive packages are completely accounted for, given a package.json. Should one of the dependencies be removed, no matter how deep in the dependency graph, it should be recorded so that someone else pulling in the latest changes also has the dependency removed for their project.
  2. Isolated environment: A different project installing the same dependency, but different version, must not affect your project. This might seem trivial coming from npm/yarn world, but languages compiling to machine code have to take a few precautionary steps.
  3. Bundled environment, so that the tool being built has all the dependencies (libraries and or other binaries) bundled together so that they can be easily moved around (same machine, or another).

If you use esy, you wont need opam to manage dependencies - esy does it for you. Dune is a build system - think webpack or Gnu Make if you wish to compare.

As I was trying to say, reproducibility is important for a pleasant developer experience. Opam does a great job fetching and install the dependencies, but it isn’t focussed on making sure a given project installs the exact same version of dependencies on two different machines locally to a project. Opam + Docker or nix2opam is the popular workflow to ensure reproducibility among opam users. Esy handles reproducibility too.

Esy doesn’t replace Dune. Esy can replace opam for a given project.
If you feel esy, opam, dune are too many tools bringing too much complexity - we hear you! There’s a WIP project - pesy - that tries to unify the tools. Same package.json to drive both package management and build system!

Reason projects can leverage three other languages (other than Reason itself) - JS, C and OCaml. It’s not surprise, but what isn’t obvious is that each of these languages have their own tools incompatible with each other. Esy wants to unify the experience across all 4 languages with the NPM workflow most are already familiar with. With esy, you’ll hopefully have a single great tool to work with all of them.

These are packages using the C toolchain packaged for esy, so that

  1. They are installed in the esy sandbox
  2. You dont have to install them separately in Docker or the machine. Aside, from manually making sure these packages are present, having esy packages make sure two users on different machines have the same copy of the packages

PS: in a hurry. The post hasn’t been proof read