PPX: no need for separate ppx and ppx6 with ocaml-migrate-parsetree 1.6.0

#1

This post is mainly for PPX authors. However, if you are a PPX user, you may want to ping your favorite PPXs about it :slight_smile:

Until now, each PPX binary has been tied to one compiler version. So, we have had to have:

  • foo/ppx for BuckleScript 5.x, which is based on OCaml 4.02, and
  • foo/ppx6 for BuckleScript 6.x+, which is based on OCaml 4.06.

This is an annoyance: users have to choose to preprocess with foo/ppx vs. foo/ppx6, there is pain when upgrading, PPX releases are twice as large as they need to be, binary builds in CI are doubled, more work and stress for maintainers, a looming need for foo/ppx8, etc.

ocaml-migrate-parsetree 1.6.0 removes the dependency of PPX on the compiler version, allowing a single foo/ppx binary to be used with all versions of BuckleScript (including future ones).


Explanation

A PPX based on ocaml-migrate-parsetree already could read any version of AST. However, until now, ocaml-migrate-parsetree would always output an AST of the same version that it was compiled on. So, if a PPX binary was compiled for BuckleScript 4.06, it could still read a 4.02 AST. It just wouldn’t write a 4.02 AST back out — it was locked to writing a 4.06 AST. This is why we needed to build separate binaries.

ocaml-migrate-parsetree 1.6.0 writes out an AST of the same version that was read, so it no longer matters which version of the compiler the PPX has been compiled with. You can read and write back out 4.02 ASTs with a PPX that was compiled for 4.06. The only thing that matters is that ocaml-migrate-parsetree itself supports ASTs of the highest compiler version you want to support.


Usage

  1. Write your transformation always against the latest AST version supported by ocaml-migrate-parsetree (currently 4.10; check the ocaml-migrate-parsetree changelogs). This maximizes the forward-compatibility of each release of your PPX. For example, if you write against 4.10 now, the next release of your PPX will remain fully compatible even with a BuckleScript that upgrades to, say, 4.08.

  2. Compile your PPX with any version of OCaml. There is no longer any connection between (1) the version of AST BuckleScript is sending to your PPX, (2) the version of AST your transformation uses internally and is written against, and (3) the version of compiler your PPX is built with. So, you can have just one build of your PPX on the latest compiler version (currently 4.09), and use the latest OCaml and/or Reason features to develop.

  3. Calling into ocaml-migrate-parsetree to trigger this new feature is done like this:

    let () = {
  Migrate_parsetree.Driver.register(
    ~name="my_ppx",
    ~args=[],
    Migrate_parsetree.Versions.ocaml_410,
    My_ppx_mapper
  );

  let argv =
    switch (Sys.argv) {
    | [|program, input_file, output_file|] =>
      [|program, input_file, "-o", output_file, "--dump-ast" |]
    | _ =>
      Sys.argv
      /* Or print some error message, because BuckleScript should
         never pass any other pattern of arguments. */
    };

  Migrate_parsetree.Driver.run_main(~argv, ());
};

    Adjust this according to whether your PPX takes arguments. The important thing is to prefix the output file with -o, add --dump-ast, and pass the modified arguments to run_main rather than run_as_ppx_rewriter.

    The reason we are modifying arguments is because BuckleScript passes them without -o and --dump-ast, which is fine — we just need to rewrite them for a different mode of ocaml-migrate-parsetree and its expectations of what’s in argv.

  4. Require ocaml-migrate-parsetree >= 1.6.0 in the esy.json or opam file used to build your PPX.

  5. You may still want to install a ppx6 binary for compatibility with existing usage, but there is no need to build or distribute a separate one — it can just be a copy of ppx made during installation.


Real example

I’ve been using this in Bisect_ppx since summer 2019 to save a headache. See:


Happy preprocessing!