What exactly is esy?

#1

Hey :wave:,

I am already working with esy on a backend GraphQL API.
But the more I use it, the more I run into problems which lets me believe, I am doing something wrong or am not really understanding all the different parts of native development involved with ocaml / reason.

I am also really new to native, so it is probably extra hard for me to understand everything.

My questions would be:

  1. What is esy exactly?
    I always thought it was just a package-manager (like npm or yarn) but it is probably more than that, right?

  2. How do the different parts of opam, dune and esy play into each other? What are each of these tools responsible for?

  3. Is esy a replacement for dune or opam or is it just an additional layer on top, providing some benefits?

  4. What are the benefits of using esy over just dune / opam? Other than being able to consume packages published to npm?

  5. What are esy-ocaml and esy-packages and why do I need them?
    As I understand they package some often used libraries (like libffi) for esy but I don’t understand why I would need them.
    Does that mean, when I require them in my esy.json / package.json I don’t have to install them globally on my system / in my docker container?

  6. What are some good resources to learn esy / native reason?

Just to clarify, I don’t want to sound like “nobody needs this, why does this all exist”.
I am just trying to understand how this all works and plays into each other.

Thank you all for your work and this awesome community! :heart:

#2

Hi,
Glad you asked! I’d like to take some time and individually answer each of these questions. Fact is, I had the same ones when I started out and prepare some notes.

Undertstanding the Reason toolchain - compilation of notes I prepared for myself and planned to use it to explain what I planned to do to improve native development experience.
Undertstanding native toolchain in general (any language) - my notes summarising tools as I understood them

#3
  1. It’s a package manager, similar to npm, yarn, opam.
  2. Opam is the traditional package manager. Esy is a new package manager, it has support for opam packages. Dune is a build system for OCaml. You use it to define build rules with dependancies… think of it kind of like a specialized Makefile, except it makes compiling ocaml code very very convenient. It’s also very fast.
  3. I think esy is an alternate package manager and sandboxing environment, and it can be seen used as a replacement for opam. If I understand correctly, esy has support for opam packages, but opam doesn’t have support for esy packages. Also, it’s possible to use esy to develop opam packages directly, as described here https://esy.sh/docs/en/opam-workflow.html
  4. Not sure what the official reasons are, but esy provides a user experience more similar to yarn/npm.
  5. You can use esy to package non-ocaml dependencies. As you mentioned, this makes it possible to include them in the sandbox, and allows you to avoid depend on the packages that are installed globally.
  6. The official docs at esy.sh, esy bugtracker https://github.com/esy/esy/issues, both the ocaml and reason discord servers, and package.json et al. for Onivim2 (for inspiration)
#4

Indeed.

While esy does package management, the ideal native development experience needs a tool that does the following things too, along with fetching sources and making them available for use.

  1. Make sure all the transitive packages are completely accounted for, given a package.json. Should one of the dependencies be removed, no matter how deep in the dependency graph, it should be recorded so that someone else pulling in the latest changes also has the dependency removed for their project.
  2. Isolated environment: A different project installing the same dependency, but different version, must not affect your project. This might seem trivial coming from npm/yarn world, but languages compiling to machine code have to take a few precautionary steps.
  3. Bundled environment, so that the tool being built has all the dependencies (libraries and or other binaries) bundled together so that they can be easily moved around (same machine, or another).

If you use esy, you wont need opam to manage dependencies - esy does it for you. Dune is a build system - think webpack or Gnu Make if you wish to compare.

As I was trying to say, reproducibility is important for a pleasant developer experience. Opam does a great job fetching and install the dependencies, but it isn’t focussed on making sure a given project installs the exact same version of dependencies on two different machines locally to a project. Opam + Docker or nix2opam is the popular workflow to ensure reproducibility among opam users. Esy handles reproducibility too.

Esy doesn’t replace Dune. Esy can replace opam for a given project.
If you feel esy, opam, dune are too many tools bringing too much complexity - we hear you! There’s a WIP project - pesy - that tries to unify the tools. Same package.json to drive both package management and build system!

Reason projects can leverage three other languages (other than Reason itself) - JS, C and OCaml. It’s not surprise, but what isn’t obvious is that each of these languages have their own tools incompatible with each other. Esy wants to unify the experience across all 4 languages with the NPM workflow most are already familiar with. With esy, you’ll hopefully have a single great tool to work with all of them.

These are packages using the C toolchain packaged for esy, so that

  1. They are installed in the esy sandbox
  2. You dont have to install them separately in Docker or the machine. Aside, from manually making sure these packages are present, having esy packages make sure two users on different machines have the same copy of the packages

PS: in a hurry. The post hasn’t been proof read

#5

AFAIK this is not accurate. There are at least two ways to get reproducible builds with opam:

  • opam lock with opam install --locked
  • opam switch export

And also AFAIK, the first one is the popular workflow for getting reproducible builds with opam across users. Here is one public “real” but small example from one of my projects.

Here is an example with opam switch export.

AFAIK Nix is not “popular” with opam users. I do personally use Nix from time to time, but once I have opam in Nix, I still install opam packages using opam lock files.

#6

I’d like the clarify that the subtext wasn’t that opam doesn’t offer ways to lock dependencies - just that the workflow to lock down dependencies was retrofitted. The workflow always seemed like an afterthought - back when I started and even now!

This is changing in the upcoming release of opam though and I might change my mind.

#7

Agreed. It is definitely bolted on, and it takes some deliberate effort to use it.

#8

However: Is Esy useful for the ReasonML ecosystem, after Rescript broke off and ReasonML took a native development direction (so similar to Ocaml)?

(This is a bit of a rant: I tried to build Onivim under FreeBSD and Esy is a blocker. After few hours of wrestling with it, I started questioning the whole point of creating and maintaining such a tool. Please reply if I’m making a point to you, please ignore if you find this post annoying).

When ReasonML was supposed to produce JS output, it made perfect sense to have a project management tool which understands NPM. However, now it seems this is not the case, and ReasonML projects will mostly use Ocaml and ReasonML dependencies. So: why not use opam for this? It’s stable and well supported tool.

On othe other hand Esy, feels hacky and unmature.

  • Is the npm ecosystem and package.json really something to take inspiration from? The npm ecosystem is a mess, the npm lifecycle methods are confusing and different in npm and yarn, the JSON format can’t contain comments, and these are just some annoyances off the top of my head - what’s the sense of replicating it onto opam ecosystem?
  • Esy is hacky, for example the esy-ocaml project - it’s necessary to maintain a fork of ocaml repository, just to add poorly portable esy-configure and esy-build scripts to it, and package.json.
  • There are over 250 open issues on esy/esy @ Github, with pretty unimpressive test suite. It is surely hard to maintain such a project and it’s obviously understaffed like many FLOSS projects - wouldn’t it make more sense to put energy into improving opam?

Why not ditch Esy and build ReasonML apps with opam? From discussion above I understand that opam can also build reproducible builds and have isolated environments, but some people just find the DX bad?

#9

Have you tried opam’s reproducible build DX? What do you think of it? Could it work for a project like Onivim?

#10

Tbh, opam always worked very robustly for me, and I understand that opam link/ switch mechanics has parallel semantics to node_modules/lock files in NPM ecosystem.

Probably? Would be some effort to check - most of Onivim deps are @opam/* anyway. Others are native React libs that do not link to JS libraries. It would require to menially create an opam file for each Reason project that uses esy, but then I don’t see why this wouldn’t be built by opam.

#11

If you check an opam lock file, the dependencies look like this:

depends: [
  "angstrom" {= "0.15.0"}
...

The problem with this is that it’s relying solely on version numbers, with no check on whether the number refers to the same actual version that it did at the time of creation. I think this defeats the purpose of a lock file. If you look at every other lock file out there (except I guess pip freeze), they all include package version check checksums to ensure build reproducibility.