Hi! As I posted elsewhere, this also bit me really hard. Let me see if I can offer my viewpoint on this issue — and OP is definitely not alone in his pain, here; in fact, I’m pretty sure everybody using BuckleScript to write libraries is probably feeling some of this pain (and I’d guess that the only reason it’s only showing up in these discussions now, is that Reason is really disproportionately being used for “apps” and other end-user products at the moment.)
So, since y’all askd for input on Discord regarding this and use-cases you don’t understand; in no particular order, here are my thoughts:
-
I, personally, absolutely did not assume BuckleScript would maintain ABI-compatibility between major versions, or even a modicum of that. The team is way too small for that! That’s not the point or argument really worth discussing in this thread, at least in-my-humble-opinion. Totally happy with some of the ABI-breaking changes, like the 5.2 modules-as-objects, and the 7.0 records-as-objects. Awesome work!
-
However, I would expect that breaking ABI-changes would be … well … breaking changes. This change in particular being shipped in 5.2 is … really, really bad, again IMHO. This is a compiler. Compiled output is an implicit contract. Totally, go ahead and break the ABI, but please communicated that in SemVer, as we usually do in the JavaScript community? (That said, I think that is a discussion for another thread, and has more to do with the weird OCaml-linked-ish version-naming scheme. More importantly, 7.0 seems to have been handled correctly, so I’m really not concerned about this particular mistake right now. ¯\_(ツ)_/¯)
-
As with @ryb73, I found it very surprising behaviour that the compiler sometimes takes into account compilation-artifacts shipped with npm packages: I completely assumed that …
-
either consuming-clients’ compilers would fail at runtime if artifacts were missing from the
node_modules
tree (i.e. the fact thatrequire("bs-thingie/thingie.bs.js")
shows up in the output, implied to me that BuckleScript treated dependencies as black-boxes) -
or the consumer’s compiler would completely ignore
.bs.js
files shipped in npm modules, and build all intermediatebs-dependencies
’ ML source-code intolib/bs/
or similar.
Personally, despite now understanding better how this works, I still think this behaviour is surprising. It may be too late to change this, but I believe one of the two above behaviours is far more intuitive: either always expect ML-build-products to be pre-compiled, or completely ignore them and build the entire tree.
-
-
Similarly, speaking of surprising behaviour, I am shocked to discover that
bsb
will write into node_modules. Wow. Please don’t do that! If building is necessary, copy tertiary files intolib/bs
or similar, and produce build-products there. -
(Hey, this part is actually something I can help with!) Again, same as @ryb73, a big reason that all of this tripped me up, is the lack of documentation catering to library/module-authors, and more details about build-system-npm-interop. Once I understand best-practices for this, I think it’ll be a very good usage of my time to go write that page, heh, and submit a pull-request to the documentation-site. That said, some of the above needs to be addressed before any of this really makes sense to document, you know? It’s pretty broken, at the moment.
Now. All of the above notwithstanding, I still think y’all are missing @ryb73’s main point. Yes, some use-cases Just Work if you ship npm packages with no JavaScript in them (even if I, personally, think that’s a horrible solution; I can see how some of you might disagree, or not care, or think that’s normal!); but there’s extremely-common use-cases that are — as far as I can tell — actually not possible with the way things currently work. Most simply put; a dependency-tree cannot currently contain mixed JavaScript and OCaml dependencies, without either some amount of dependency-vendoring, or every single ancestor-package integrating BuckleScript into its build-system.
Let me explain, first in the abstract, and then by presenting one of the package-trees that the v5.2 release broke for me, in the real world, in an unfixable way.
Let’s say you and I are writing JavaScript libraries in OCaml. One of the reasons our team chose BuckleScript, is that it is advertised as being ‘invisible’ to code-consumers (see the quotes @ryb73 pulled out, above): we’re expecting to write BuckleScript, and our users will just ‘see’ standard JavaScript, and not need to know BuckleScript exists at all. (You know, the same way Babel, or TypeScript, or CoffeeScript, or any other JavaScript-ecosystem compiler can be expected to function!)
Our primary product is the
task-doer
library. This provides a published JavaScript API, is written in ML, and compiled with BuckleScript. As a part of its functionality, it depends on the ML librarybs-thingie
:bs-thingie
shows up in the npm"dependencies"
at a particular version, has standard SemVer versioning, is used in other projects unrelated to ours, the whole shebang (you know … a normal … library!). Similarly, downstream from us, lies a JavaScript client namedcool-app
— they aren’t involved with our team, have no idea BuckleScript exists, and expecttask-doer
to Just Work.As things currently stand,
task-doer
on npm contains JavaScript source-code (in.bs.js
files, but the downstream consumers don’t care about that); that source-code includes lines like:var Thingie = require("bs-thingie/thingie.bs.js");
Of course, this is all fine and dandy, because
task-doer
has correctly declared its’ dependency uponbs-thingie
—bs-thingie/
was downloaded and unpacked by npm, and containsthingie.bs.js
, itself compiled by the author thereof, archived, and published to npm. Our client,cool-app
, can justrequire("task-doer")
like any other npm library, and away we go, everything is fine … riiiiiiiight up until v5.2 or v7.0 of BuckleScript lands, and either the we (the authors oftask-doer
) or the authors ofbs-thingie
upgrade our BuckleScript versions. (Obviously! Yes, that’s how ABIs work, I know.)
The above suggestions boil down to ‘solving’ this situation in, basically, one of two ways:
-
Don’t publish JavaScript, to the JavaScript package-manager. In the above example, this would involve, say, adding
.bs.js
to.npmignore
or similar. However, doing so means every consumer, anywhere in the npm dependency-tree, needs to install a version ofbs-platform
, and somehow work it into their build-system: in our above example, this would involve informingcool-app
“hey, we don’t provide a JavaScript API anymore; please install BuckleScript to keep using this library.” (And ifcool-app
is actuallycool-js-library
? They have to do the same thing to their consumers!) -
At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. In the JS world, this usually looks like Babel or Rollup or similar squashing the dependency-tree into a single JavaScript file. Our hypothetical
task-doer
authors would, in this case, basically need to removebs-thingie
and all of their other dependencies from the actual semantic dependency-tree, and add build-tooling outside ofbsb
, like Rollup or Webpack, to produce the new “JavaScript interface” for their JS consumers.
Both of these have huge downsides, obviously, and ones I won’t get into here; the downsides of vendoring are well-studied elsewhere, as is the breaking of dependency encapsulation. In particular both are absolutely untenable for my own use-cases — let’s study one in particular.
I publish a library,
bs-uchar
(details unimportant; but it shims theUchar.t
type in a version-independent manner, to help library-authors publish code that’s usable on both BuckleScript v4/5, and v6/7.) This is intended for wider public consumption, as is all of my work; it must stay a stable product that isn’t tightly coupled to my other work.Next, I’ve a much more involved effort,
bs-sedlex
(a JavaScript port of the Sedlex lexer-generator ppx), that depends onbs-uchar
, for obvious reasons; again, this is intended for general public consumption, and must not be tightly coupled to the above or following.The last of my own involved projects,
excmd.js
(a parser library) depends both onbs-uchar
directly, as well as onbs-sedlex
. Notably, this library involves both ML-APIs for ML consumers and stable TypeScript/JavaScript interfaces for JS consumers.Finally, the downstream project to which I am contributing here, Tridactyl (a Vi-mode total-interface-overhaul for FireFox) depends on
excmd.js
, and has no idea that BuckleScript exists.
Let’s look at what the two existing solutions would involve, to implement in my own projects:
-
Don’t publish JavaScript, to the JavaScript package-manager. I’d have to remove all the
.bs.js
files from any involved library that I control; and then convince the Tridactyl developers — who don’t care, and shouldn’t! — to add BuckleScript to their (already mind-bendingly complicated, as it turns out) build-system. (I’m not even sure how you’d invokebsb
to compile dependencies innode_modules
, if you don’t have any ML-sources in the actual project?). All just for me and my work. Further, because I’m partial to dualistic JavaScript/ML interfaces, I would have to somehow split the “top” library out, extracting any ML sources fromexcmd.js
into another library … and publish an adjacent ‘JavaScript-interface library’ that, (again, somehow?) abstracted over non-existent.bs.js
files that any additional downstream consumers ofexcmd.js
would be taking on the responsibility to build. -
At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. Okay, in this dependency-chain, I guess I’m lucky — there’s no interleaved JavaScript-ML-JavaScript-ML series; there’s no ML above
excmd.js
. So, to implement this solution, I’d have to move all of the above-mentioned dependencies intodevDependencies
or similar. Then, add a Rollup stage to my build-system, and bake a singlelib.js
file or similar, post-BuckleScriptbs -make-world
compilation, with every dependency vendored into it, into my npm archive. (I am, of course, now throwing away all of the advantages of npm and SemVer, of a dynamic dependency tree, of semantic security-analysis, of Yarn resolutions, and making debugging a nightmare for my consumers. Woo! Vendoring deps!)
Neither of these are great options, in ways that I hope are, by this point, obvious!
This got really long-winded; so I’m gonna cut it off here — but, well, I clearly have quite a lot to say on the topic. My biggest concern is that there may not be much we can do about this, even if y’all accept my point that it’s a big problem; it may simply be too late.