ANN: bs-platform 5.2.1 and 6.2.1 released


#5

Hi Ryan, I can’t reproduce this issue. Here is what I did:

git clone https://github.com/ryb73/bs-modules-example.git
cd bs-modules-example
yarn
cd b
npx bsb -make-world
cd ..
node b

Output:

-- 100 --

#6

That’s the thing – if you check out the code and compile package B yourself, it’ll work fine. That’s because running bsb -make-world recompiles the entire project, dependencies and all, using whichever version of bs-platform is installed. Doing so overwrites the JS files that were published by package A.

The problem doesn’t manifest itself until the point of distribution. If you follow the directions in the README by npm installing package B and running the binary, you should see undefined in place of 100.

Take a look at the package.json for each project:

{
  "name": "@ryb73/bs-51-52-interop-a",
  ...
  "files": [
    "/bsconfig.json",
    "/src",
    "/lib/js/src"
  ],
  ...
  "devDependencies": {
    "bs-platform": "^5.1.0"
  }
}
{
  "name": "@ryb73/bs-51-52-interop-b",
  ...
  "files": [
    "/bsconfig.json",
    "/src",
    "/lib/js/src"
  ],
  ...
  "devDependencies": {
    "bs-platform": "^5.2.0"
  },
  "dependencies": {
    "@ryb73/bs-51-52-interop-a": "0.1.0"
  }
 ...
}

Notice that @ryb73/bs-51-52-interop-a is a dependency, but bs-platform is a devDependency.

Notice also that I’m distributing the compiled JS files but not node_modules. This matches node/npm best practices.

And here’s the problem: when you run npm i @ryb73/bs-51-52-interop-b, npm downloads the published B source files which were compiled using 5.2.0. Because @ryb73/bs-51-52-interop-a is a dependency, npm will also download the published A source files which were compiled using 5.1.0. The published source files for A and the published source files for B are incompatible.

This is why (assuming my understanding of how Bucklescript packages should be published is correct) it’s important that new non-major versions of bs-platform maintain not only backwards compatibility of OCaml/Reason interfaces, but also backwards compatibility of the compiled JS interfaces.

It’s also possible that I’m doing something wrong with my distribution process. If that’s the case, it’d be great if the Reason team would communicate clearly

  • Which files I should publish as a Buscklescript library/app author
  • Into which dependency bucket (dep/devDeps/peerDeps/optionalDeps) I should put bs-platform and bs-dependencies
  • What constitutes a major vs minor vs patch update

#7

For BuckleScript packages, the source of truth is the OCaml/Reason source files, not the JavaScript output. The JS output is more of a ‘snapshot’ of what it looked like the last time it was checked in. You can check in JS files or not–it’s optional–but you should definitely not consider them the source of truth for consuming BuckleScript packages.

The correct practice here is to run bsb -make-world in the consuming project so that BuckleScript can recompile the entire dependency tree. Otherwise it can’t guarantee consistency of modules across different versions.


#8

I feel like I’m missing something. Essentially what you’re saying is that distributing BuckleScript binaries is not supported? Projects like yarn, lerna, babel-cli, typescript, webpack… these types of projects cannot/should not be implemented in BuckleScript?

Or if I’m at an organization that has a large number of modules written in JS, I can’t dip my toe into BuckleScript by rewriting one module without introducing BuckleScript compilation into the integration pipeline for all modules that depend on it?


#9

I don’t see any problems with distributing BuckleScript ‘binaries’ i.e. scripts to run with NodeJS, as long as they were compiled with bsb -make-world.

If you have a large number of JS modules, and want to migrate a few of them to BuckleScript, so that other JS modules can call them, that is also well-supported provided you keep in mind a couple of interop issues. In the context of nested modules, in the past you had to ensure that you exposed them in a way that made sense to JavaScript consumers, i.e. you wouldn’t want to expose BuckleScript’s previous array representation of nested modules. This was a known issue and (I assume) a long-term goal to fix to make interop easier. Now that modules are encoded as JS objects, this has been achieved.

What is not supported, is compiling BuckleScript modules to JS, then targeting those output JS files from the output JS of other BuckleScript modules. BuckleScript needs to control the compilation of the entire module graph, for it to guarantee consistency.

I hope that clears things up.


#10

Unfortunately it raises more questions than it answers :smile:

Let’s focus on just the example project for now. What changes would you make to project B so that users can just do npm i @ryb73/bs-51-52-interop-b and then run bs-51-52-interop-b? I want the end user to not have to run bsb.

Thanks for bearing with me, btw. This has been a point of confusion for me for a long time now.


#11

No worries. Regarding your question, as far as I can tell it should be enough to run npx bsb -make-world in bs-51-52-interop-b, check in that output, and publish. The user should now be able to run it directly.


#12

That’s essentially what I did to publish the existing version and it doesn’t work (I used yarn build instead of npx bsb -make-world but should be the same thing; I can confirm later if not)

I’m starting to wonder if has something to do with the fact that I’m not using in-source?


#13

npx bsb -make-world and yarn build in directory b should be the same thing, by definition. I ran the former. After that here is what I have:

node_modules/@ryb73/bs-51-52-interop-a/lib/js/src/A.js

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';


var Inner = {
  value: 100
};

exports.Inner = Inner;
/* No side effect */

b/lib/js/src/index.js

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';

var A = require("@ryb73/bs-51-52-interop-a/lib/js/src/A.js");

console.log("--", A.Inner.value, "--");

/*  Not a pure module */

#14

Yep, that looks right. Now publish it.

It doesn’t work.

The reason is that the node_modules/@ryb73/bs-51-52-interop-a/lib/js/src/A.js doesn’t get published. When a user runs npm i @ryb73/bs-51-52-interop-b, they’re not getting the A.js that you posted. They’re getting the A.js that’s published to @ryb73/bs-51-52-interop-a, which is:

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';


var Inner = /* module */[/* value */100];

exports.Inner = Inner;
/* No side effect */

#15

True. For a script that needs to be set up with npm install or similar, it will download dependency packages and try to use them, and it won’t work. However, keep in mind that:

  • Scripts like webpack etc., are generally not distributed in this form, they’re distributed as JS bundles with some kind of wrapper. So they shouldn’t face this issue.
  • Modules which import other JavaScript modules from npm, have no guarantee of what kind of JavaScript they’re getting. It may be ES4, 5, 6, Babel with non-standard extensions … anything really. It’s almost always the case that these modules need to run some kind of compilation pipeline before they’re usable. BuckleScript is not an exception to this.
  • BuckleScript has broken JavaScript output compatibility in the past, significantly with the change which mapped option types to JavaScript undefined. It typically does this when it finds a really good, idiomatic mapping that can have a lot of benefits like code size reduction or usability.

Care needs to be taken but in general the nested module output is not a breaking change at the BuckleScript level.


#16

Ok, this argument makes sense. Thanks for clearing it up.

I think part of my confusion came from the BuckleScript docs. For example:

Package Management

We use NPM and Yarn. Since the generated output is clean enough, you can publish them at NPM prepublishOnly time and remove all trace of BuckleScript beforehand. The consumer wouldn’t even know you wrote in BS, not JS! Check your node_modules right now; you might have been using some transitive BS code without knowing!

This makes it sound like the recommendation is to publish the compiled JS, but from talking to you it sounds like that’s not recommended.


#17

That depends. It’s totally possible to publish JavaScript output of BuckleScript compilation as a JavaScript library. I’ve done it myself: https://www.npmjs.com/package/@yawaramin/dbc

But it does require you to understand the BuckleScript -> JavaScript mapping mechanisms, and use only the explicitly guaranteed mappings for JavaScript interop, e.g. OCaml array -> JavaScript array, OCaml tuple -> JavaScript array, etc. We probably need to update the mappings a bit: https://bucklescript.github.io/docs/en/common-data-types#cheat-sheet


#18

I mean yeah, I’ve been doing it too, for years. What I’m just learning is that it should be the exception rather than the rule. IMO the docs muddy that distinction.


#19

Agreed, they could use some fine print to explain this.


#20

Hi, are you aware of tools like https://github.com/zeit/ncc
There are two kinds of backwards compatibility, source level backwards compatibility and ABI backwards compatibility (generated JS output).

The latter is very hard to maintain, it is not relevant under some cases (mono-repo style). If we have to bump major version for breaking ABI compatibility, essentially we have to bump major version for each release (Note ocaml native compiler does not provide ABI backwards compatibility either), maybe we should revisit it when reach a high level of stability.

We would like to learn more about your use case so that we can better server your use case.


#21

Hi! As I posted elsewhere, this also bit me really hard. Let me see if I can offer my viewpoint on this issue — and OP is definitely not alone in his pain, here; in fact, I’m pretty sure everybody using BuckleScript to write libraries is probably feeling some of this pain (and I’d guess that the only reason it’s only showing up in these discussions now, is that Reason is really disproportionately being used for “apps” and other end-user products at the moment.)

So, since y’all askd for input on Discord regarding this and use-cases you don’t understand; in no particular order, here are my thoughts:

  1. I, personally, absolutely did not assume BuckleScript would maintain ABI-compatibility between major versions, or even a modicum of that. The team is way too small for that! That’s not the point or argument really worth discussing in this thread, at least in-my-humble-opinion. Totally happy with some of the ABI-breaking changes, like the 5.2 modules-as-objects, and the 7.0 records-as-objects. Awesome work!

  2. However, I would expect that breaking ABI-changes would be … well … breaking changes. This change in particular being shipped in 5.2 is … really, really bad, again IMHO. This is a compiler. Compiled output is an implicit contract. Totally, go ahead and break the ABI, but please communicated that in SemVer, as we usually do in the JavaScript community? (That said, I think that is a discussion for another thread, and has more to do with the weird OCaml-linked-ish version-naming scheme. More importantly, 7.0 seems to have been handled correctly, so I’m really not concerned about this particular mistake right now. ¯\_(ツ)_/¯)

  3. As with @ryb73, I found it very surprising behaviour that the compiler sometimes takes into account compilation-artifacts shipped with npm packages: I completely assumed that …

    • either consuming-clients’ compilers would fail at runtime if artifacts were missing from the node_modules tree (i.e. the fact that require("bs-thingie/thingie.bs.js") shows up in the output, implied to me that BuckleScript treated dependencies as black-boxes)

    • or the consumer’s compiler would completely ignore .bs.js files shipped in npm modules, and build all intermediate bs-dependencies’ ML source-code into lib/bs/ or similar.

    Personally, despite now understanding better how this works, I still think this behaviour is surprising. It may be too late to change this, but I believe one of the two above behaviours is far more intuitive: either always expect ML-build-products to be pre-compiled, or completely ignore them and build the entire tree.

  4. Similarly, speaking of surprising behaviour, I am shocked to discover that bsb will write into node_modules. Wow. Please don’t do that! If building is necessary, copy tertiary files into lib/bs or similar, and produce build-products there. :anguished:

  5. (Hey, this part is actually something I can help with!) Again, same as @ryb73, a big reason that all of this tripped me up, is the lack of documentation catering to library/module-authors, and more details about build-system-npm-interop. Once I understand best-practices for this, I think it’ll be a very good usage of my time to go write that page, heh, and submit a pull-request to the documentation-site. That said, some of the above needs to be addressed before any of this really makes sense to document, you know? It’s pretty broken, at the moment.


Now. All of the above notwithstanding, I still think y’all are missing @ryb73’s main point. Yes, some use-cases Just Work if you ship npm packages with no JavaScript in them (even if I, personally, think that’s a horrible solution; I can see how some of you might disagree, or not care, or think that’s normal!); but there’s extremely-common use-cases that are — as far as I can tell — actually not possible with the way things currently work. Most simply put; a dependency-tree cannot currently contain mixed JavaScript and OCaml dependencies, without either some amount of dependency-vendoring, or every single ancestor-package integrating BuckleScript into its build-system.

Let me explain, first in the abstract, and then by presenting one of the package-trees that the v5.2 release broke for me, in the real world, in an unfixable way.

Let’s say you and I are writing JavaScript libraries in OCaml. One of the reasons our team chose BuckleScript, is that it is advertised as being ‘invisible’ to code-consumers (see the quotes @ryb73 pulled out, above): we’re expecting to write BuckleScript, and our users will just ‘see’ standard JavaScript, and not need to know BuckleScript exists at all. (You know, the same way Babel, or TypeScript, or CoffeeScript, or any other JavaScript-ecosystem compiler can be expected to function!)

Our primary product is the task-doer library. This provides a published JavaScript API, is written in ML, and compiled with BuckleScript. As a part of its functionality, it depends on the ML library bs-thingie: bs-thingie shows up in the npm "dependencies" at a particular version, has standard SemVer versioning, is used in other projects unrelated to ours, the whole shebang (you know … a normal … library!). Similarly, downstream from us, lies a JavaScript client named cool-app — they aren’t involved with our team, have no idea BuckleScript exists, and expect task-doer to Just Work.

As things currently stand, task-doer on npm contains JavaScript source-code (in .bs.js files, but the downstream consumers don’t care about that); that source-code includes lines like:

var Thingie = require("bs-thingie/thingie.bs.js");

Of course, this is all fine and dandy, because task-doer has correctly declared its’ dependency upon bs-thingiebs-thingie/ was downloaded and unpacked by npm, and contains thingie.bs.js, itself compiled by the author thereof, archived, and published to npm. Our client, cool-app, can just require("task-doer") like any other npm library, and away we go, everything is fine … riiiiiiiight up until v5.2 or v7.0 of BuckleScript lands, and either the we (the authors of task-doer) or the authors of bs-thingie upgrade our BuckleScript versions. (Obviously! Yes, that’s how ABIs work, I know.)

The above suggestions boil down to ‘solving’ this situation in, basically, one of two ways:

  1. Don’t publish JavaScript, to the JavaScript package-manager. In the above example, this would involve, say, adding .bs.js to .npmignore or similar. However, doing so means every consumer, anywhere in the npm dependency-tree, needs to install a version of bs-platform, and somehow work it into their build-system: in our above example, this would involve informing cool-app “hey, we don’t provide a JavaScript API anymore; please install BuckleScript to keep using this library.” (And if cool-app is actually cool-js-library? They have to do the same thing to their consumers!)

  2. At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. In the JS world, this usually looks like Babel or Rollup or similar squashing the dependency-tree into a single JavaScript file. Our hypothetical task-doer authors would, in this case, basically need to remove bs-thingie and all of their other dependencies from the actual semantic dependency-tree, and add build-tooling outside of bsb, like Rollup or Webpack, to produce the new “JavaScript interface” for their JS consumers.

Both of these have huge downsides, obviously, and ones I won’t get into here; the downsides of vendoring are well-studied elsewhere, as is the breaking of dependency encapsulation. In particular both are absolutely untenable for my own use-cases — let’s study one in particular.

I publish a library, bs-uchar (details unimportant; but it shims the Uchar.t type in a version-independent manner, to help library-authors publish code that’s usable on both BuckleScript v4/5, and v6/7.) This is intended for wider public consumption, as is all of my work; it must stay a stable product that isn’t tightly coupled to my other work.

Next, I’ve a much more involved effort, bs-sedlex (a JavaScript port of the Sedlex lexer-generator ppx), that depends on bs-uchar, for obvious reasons; again, this is intended for general public consumption, and must not be tightly coupled to the above or following.

The last of my own involved projects, excmd.js (a parser library) depends both on bs-uchar directly, as well as on bs-sedlex. Notably, this library involves both ML-APIs for ML consumers and stable TypeScript/JavaScript interfaces for JS consumers.

Finally, the downstream project to which I am contributing here, Tridactyl (a Vi-mode total-interface-overhaul for FireFox) depends on excmd.js, and has no idea that BuckleScript exists.

Let’s look at what the two existing solutions would involve, to implement in my own projects:

  1. Don’t publish JavaScript, to the JavaScript package-manager. I’d have to remove all the .bs.js files from any involved library that I control; and then convince the Tridactyl developers — who don’t care, and shouldn’t! — to add BuckleScript to their (already mind-bendingly complicated, as it turns out) build-system. (I’m not even sure how you’d invoke bsb to compile dependencies in node_modules, if you don’t have any ML-sources in the actual project?). All just for me and my work. Further, because I’m partial to dualistic JavaScript/ML interfaces, I would have to somehow split the “top” library out, extracting any ML sources from excmd.js into another library … and publish an adjacent ‘JavaScript-interface library’ that, (again, somehow?) abstracted over non-existent .bs.js files that any additional downstream consumers of excmd.js would be taking on the responsibility to build.

  2. At some single “boundary” between ML and JavaScript sources, vendor all of the BuckleScript output. Okay, in this dependency-chain, I guess I’m lucky — there’s no interleaved JavaScript-ML-JavaScript-ML series; there’s no ML above excmd.js. So, to implement this solution, I’d have to move all of the above-mentioned dependencies into devDependencies or similar. Then, add a Rollup stage to my build-system, and bake a single lib.js file or similar, post-BuckleScript bs -make-world compilation, with every dependency vendored into it, into my npm archive. (I am, of course, now throwing away all of the advantages of npm and SemVer, of a dynamic dependency tree, of semantic security-analysis, of Yarn resolutions, and making debugging a nightmare for my consumers. Woo! Vendoring deps!)

Neither of these are great options, in ways that I hope are, by this point, obvious!

This got really long-winded; so I’m gonna cut it off here — but, well, I clearly have quite a lot to say on the topic. My biggest concern is that there may not be much we can do about this, even if y’all accept my point that it’s a big problem; it may simply be too late.


Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
#22

However, I would expect that breaking ABI-changes would be … well … breaking changes. This change in particular being shipped in 5.2 is … really, really bad, again IMHO. This is a compiler. Compiled output is an implicit contract. Totally, go ahead and break the ABI, but please communicated that in SemVer, as we usually do in the JavaScript community?

I think there’s been a communication gap. The ‘ABI’ concept here is really more of an analogy than an exact description of BuckleScript’s compiled output. The way the BuckleScript documentation describes it is, non-shared i.e. internal data types:

Variants (including option and list), BuckleScript objects and others can be exported as well, but you should not rely on their internal representation on the JS side. Aka, don’t grab a BS list and start manipulating its structure on the JS side.

‘ABI’ would imply ‘interface’ i.e. outside-facing. But BuckleScript compiled output of the non-shared types are not outside-facing i.e. not guaranteed to be stable.

I believe one of the two above behaviours is far more intuitive: either always expect ML-build-products to be pre-compiled, or completely ignore them and build the entire tree.

Yeah, the latter is what BuckleScript does. It does build the entire tree. You have to use bsb -clean-world and bsb -make-world to guarantee it’s clean.

The above suggestions boil down to ‘solving’ this situation in, basically, one of two ways:

There’s a third way: write BuckleScript libraries specifically designed to be used from JavaScript. I.e.:

  • Ensure you don’t use the BuckleScript runtime in the output .bs.js
  • Ensure you don’t expose any internal data types e.g. lists, variants, nested modules, etc.
  • And so on.

Concretely for your library–can you point to the breakage? Maybe we can brainstorm some way to fix it. No guarantees but maybe it’s not as bad as we’re thinking right now.


Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
Breaking BuckleScript changes, library maintainership, and versioning compiled-output: How are y'all handling this?
#23

There’s a third way: write BuckleScript libraries specifically designed to be used from JavaScript. I.e.:

I feel like an exhaustive list of these with practical examples would be a hugely valuable piece of documentation or a blog post. I’m also wondering if the limitation list is practically same when one wants to migrate from JS codebase to ReasonML and run both side by side for a while. Maybe not exactly the same but at least partially?


#24

Let’s talk about that in a new thread, this one is getting quite long in the tooth :slight_smile: