ANN: bs-platform 5.2.1 and 6.2.1 released


#21

Hi! As I posted elsewhere, this also bit me really hard. Let me see if I can offer my viewpoint on this issue — and OP is definitely not alone in his pain, here; in fact, I’m pretty sure everybody using BuckleScript to write libraries is probably feeling some of this pain (and I’d guess that the only reason it’s only showing up in these discussions now, is that Reason is really disproportionately being used for “apps” and other end-user products at the moment.)

So, since y’all askd for input on Discord regarding this and use-cases you don’t understand; in no particular order, here are my thoughts:

  1. I, personally, absolutely did not assume BuckleScript would maintain ABI-compatibility between major versions, or even a modicum of that. The team is way too small for that! That’s not the point or argument really worth discussing in this thread, at least in-my-humble-opinion. Totally happy with some of the ABI-breaking changes, like the 5.2 modules-as-objects, and the 7.0 records-as-objects. Awesome work!

  2. However, I would expect that breaking ABI-changes would be … well … breaking changes. This change in particular being shipped in 5.2 is … really, really bad, again IMHO. This is a compiler. Compiled output is an implicit contract. Totally, go ahead and break the ABI, but please communicated that in SemVer, as we usually do in the JavaScript community? (That said, I think that is a discussion for another thread, and has more to do with the weird OCaml-linked-ish version-naming scheme. More importantly, 7.0 seems to have been handled correctly, so I’m really not concerned about this particular mistake right now. ¯\_(ツ)_/¯)

  3. As with @ryb73, I found it very surprising behaviour that the compiler sometimes takes into account compilation-artifacts shipped with npm packages: I completely assumed that …

    • either consuming-clients’ compilers would fail at runtime if artifacts were missing from the node_modules tree (i.e. the fact that require("bs-thingie/thingie.bs.js") shows up in the output, implied to me that BuckleScript treated dependencies as black-boxes)

    • or the consumer’s compiler would completely ignore .bs.js files shipped in npm modules, and build all intermediate bs-dependencies’ ML source-code into lib/bs/ or similar.

    Personally, despite now understanding better how this works, I still think this behaviour is surprising. It may be too late to change this, but I believe one of the two above behaviours is far more intuitive: either always expect ML-build-products to be pre-compiled, or completely ignore them and build the entire tree.

  4. Similarly, speaking of surprising behaviour, I am shocked to discover that bsb will write into node_modules. Wow. Please don’t do that! If building is necessary, copy tertiary files into lib/bs or similar, and produce build-products there. :anguished:

  5. (Hey, this part is actually something I can help with!) Again, same as @ryb73, a big reason that all of this tripped me up, is the lack of documentation catering to library/module-authors, and more details about build-system-npm-interop. Once I understand best-practices for this, I think it’ll be a very good usage of my time to go write that page, heh, and submit a pull-request to the documentation-site. That said, some of the above needs to be addressed before any of this really makes sense to document, you know? It’s pretty broken, at the moment.


Now. All of the above notwithstanding, I still think y’all are missing @ryb73’s main point. Yes, some use-cases Just Work if you ship npm packages with no JavaScript in them (even if I, personally, think that’s a horrible solution; I can see how some of you might disagree, or not care, or think that’s normal!); but there’s extremely-common use-cases that are — as far as I can tell — actually not possible with the way things currently work. Most simply put; a dependency-tree cannot currently contain mixed JavaScript and OCaml dependencies, without either some amount of dependency-vendoring, or every single ancestor-package integrating BuckleScript into its build-system.

Let me explain, first in the abstract, and then by presenting one of the package-trees that the v5.2 release broke for me, in the real world, in an unfixable way.

Let’s say you and I are writing JavaScript libraries in OCaml. One of the reasons our team chose BuckleScript, is that it is advertised as being ‘invisible’ to code-consumers (see the quotes @ryb73 pulled out, above): we’re expecting to write BuckleScript, and our users will just ‘see’ standard JavaScript, and not need to know BuckleScript exists at all. (You know, the same way Babel, or TypeScript, or CoffeeScript, or any other JavaScript-ecosystem compiler can be expected to function!)

Our primary product is the task-doer library. This provides a published JavaScript API, is written in ML, and compiled with BuckleScript. As a part of its functionality, it depends on the ML library bs-thingie: bs-thingie shows up in the npm "dependencies" at a particular version, has standard SemVer versioning, is used in other projects unrelated to ours, the whole shebang (you know … a normal … library!). Similarly, downstream from us, lies a JavaScript client named cool-app — they aren’t involved with our team, have no idea BuckleScript exists, and expect task-doer to Just Work.

As things currently stand, task-doer on npm contains JavaScript source-code (in .bs.js files, but the downstream consumers don’t care about that); that source-code includes lines like:

var Thingie = require("bs-thingie/thingie.bs.js");

Of course, this is all fine and dandy, because task-doer has correctly declared its’ dependency upon bs-thingiebs-thingie/ was downloaded and unpacked by npm, and contains thingie.bs.js, itself compiled by the author thereof, archived, and published to npm. Our client, cool-app, can just require("task-doer") like any other npm library, and away we go, everything is fine … riiiiiiiight up until v5.2 or v7.0 of BuckleScript lands, and either the we (the authors of task-doer) or the authors of bs-thingie upgrade our BuckleScript versions. (Obviously! Yes, that’s how ABIs work, I know.)

The above suggestions boil down to ‘solving’ this situation in, basically, one of two ways:

  1. Don’t publish JavaScript, to the JavaScript package-manager. In the above example, this would involve, say, adding .bs.js to .npmignore or similar. However, doing so means every consumer, anywhere in the npm dependency-tree, needs to install a version of bs-platform, and somehow work it into their build-system: in our above example, this would involve informing cool-app “hey, we don’t provide a JavaScript API anymore; please install BuckleScript to keep using this library.” (And if cool-app is actually cool-js-library? They have to do the same thing to their consumers!)

  2. At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. In the JS world, this usually looks like Babel or Rollup or similar squashing the dependency-tree into a single JavaScript file. Our hypothetical task-doer authors would, in this case, basically need to remove bs-thingie and all of their other dependencies from the actual semantic dependency-tree, and add build-tooling outside of bsb, like Rollup or Webpack, to produce the new “JavaScript interface” for their JS consumers.

Both of these have huge downsides, obviously, and ones I won’t get into here; the downsides of vendoring are well-studied elsewhere, as is the breaking of dependency encapsulation. In particular both are absolutely untenable for my own use-cases — let’s study one in particular.

I publish a library, bs-uchar (details unimportant; but it shims the Uchar.t type in a version-independent manner, to help library-authors publish code that’s usable on both BuckleScript v4/5, and v6/7.) This is intended for wider public consumption, as is all of my work; it must stay a stable product that isn’t tightly coupled to my other work.

Next, I’ve a much more involved effort, bs-sedlex (a JavaScript port of the Sedlex lexer-generator ppx), that depends on bs-uchar, for obvious reasons; again, this is intended for general public consumption, and must not be tightly coupled to the above or following.

The last of my own involved projects, excmd.js (a parser library) depends both on bs-uchar directly, as well as on bs-sedlex. Notably, this library involves both ML-APIs for ML consumers and stable TypeScript/JavaScript interfaces for JS consumers.

Finally, the downstream project to which I am contributing here, Tridactyl (a Vi-mode total-interface-overhaul for FireFox) depends on excmd.js, and has no idea that BuckleScript exists.

Let’s look at what the two existing solutions would involve, to implement in my own projects:

  1. Don’t publish JavaScript, to the JavaScript package-manager. I’d have to remove all the .bs.js files from any involved library that I control; and then convince the Tridactyl developers — who don’t care, and shouldn’t! — to add BuckleScript to their (already mind-bendingly complicated, as it turns out) build-system. (I’m not even sure how you’d invoke bsb to compile dependencies in node_modules, if you don’t have any ML-sources in the actual project?). All just for me and my work. Further, because I’m partial to dualistic JavaScript/ML interfaces, I would have to somehow split the “top” library out, extracting any ML sources from excmd.js into another library … and publish an adjacent ‘JavaScript-interface library’ that, (again, somehow?) abstracted over non-existent .bs.js files that any additional downstream consumers of excmd.js would be taking on the responsibility to build.

  2. At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. Okay, in this dependency-chain, I guess I’m lucky — there’s no interleaved JavaScript-ML-JavaScript-ML series; there’s no ML above excmd.js. So, to implement this solution, I’d have to move all of the above-mentioned dependencies into devDependencies or similar. Then, add a Rollup stage to my build-system, and bake a single lib.js file or similar, post-BuckleScript bs -make-world compilation, with every dependency vendored into it, into my npm archive. (I am, of course, now throwing away all of the advantages of npm and SemVer, of a dynamic dependency tree, of semantic security-analysis, of Yarn resolutions, and making debugging a nightmare for my consumers. Woo! Vendoring deps!)

Neither of these are great options, in ways that I hope are, by this point, obvious!

This got really long-winded; so I’m gonna cut it off here — but, well, I clearly have quite a lot to say on the topic. My biggest concern is that there may not be much we can do about this, even if y’all accept my point that it’s a big problem; it may simply be too late.


Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
#22

However, I would expect that breaking ABI-changes would be … well … breaking changes. This change in particular being shipped in 5.2 is … really, really bad, again IMHO. This is a compiler. Compiled output is an implicit contract. Totally, go ahead and break the ABI, but please communicated that in SemVer, as we usually do in the JavaScript community?

I think there’s been a communication gap. The ‘ABI’ concept here is really more of an analogy than an exact description of BuckleScript’s compiled output. The way the BuckleScript documentation describes it is, non-shared i.e. internal data types:

Variants (including option and list), BuckleScript objects and others can be exported as well, but you should not rely on their internal representation on the JS side. Aka, don’t grab a BS list and start manipulating its structure on the JS side.

‘ABI’ would imply ‘interface’ i.e. outside-facing. But BuckleScript compiled output of the non-shared types are not outside-facing i.e. not guaranteed to be stable.

I believe one of the two above behaviours is far more intuitive: either always expect ML-build-products to be pre-compiled, or completely ignore them and build the entire tree.

Yeah, the latter is what BuckleScript does. It does build the entire tree. You have to use bsb -clean-world and bsb -make-world to guarantee it’s clean.

The above suggestions boil down to ‘solving’ this situation in, basically, one of two ways:

There’s a third way: write BuckleScript libraries specifically designed to be used from JavaScript. I.e.:

  • Ensure you don’t use the BuckleScript runtime in the output .bs.js
  • Ensure you don’t expose any internal data types e.g. lists, variants, nested modules, etc.
  • And so on.

Concretely for your library–can you point to the breakage? Maybe we can brainstorm some way to fix it. No guarantees but maybe it’s not as bad as we’re thinking right now.


Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
#23

There’s a third way: write BuckleScript libraries specifically designed to be used from JavaScript. I.e.:

I feel like an exhaustive list of these with practical examples would be a hugely valuable piece of documentation or a blog post. I’m also wondering if the limitation list is practically same when one wants to migrate from JS codebase to ReasonML and run both side by side for a while. Maybe not exactly the same but at least partially?


#24

Let’s talk about that in a new thread, this one is getting quite long in the tooth :slight_smile: