ANN: bs-platform 5.2.1 and 6.2.1 released


#1

Dear users,
We made a release for bs-platform 5.2.1 last Friday, bs-platform 5.2.1 is a bug fix release for 5.2.0,
changes for 5.2.1
changes for 5.2.0
This release is supposed to be a stable release, it comes with some exiting features in 5.2.0 and we encourage you to upgrade it.
Happy hacking!


#2

Thanks for the update!

I just noticed that 5.2.0 appears to be a breaking change due to the changes to the way local modules are compiled, i.e. projects compiled with <5.2.0 will be incompatible with projects compiled with >= 5.2.0. Am I understanding this correctly?


#3

It is not a breaking change, local module representation is internal, unless people do crazy things over the boundary between ocaml and JS, otherwise such change is transparent


#4

It’s not just the boundary between ocaml and JS, but the boundary between packages. I’ve set up an example here: https://github.com/ryb73/bs-modules-example

The problem that arises in the example is that package a was compiled and published using bs-platform@5.1.0, while package b was compiled and published using bs-platform@5.2.0.


#5

Hi Ryan, I can’t reproduce this issue. Here is what I did:

git clone https://github.com/ryb73/bs-modules-example.git
cd bs-modules-example
yarn
cd b
npx bsb -make-world
cd ..
node b

Output:

-- 100 --

#6

That’s the thing – if you check out the code and compile package B yourself, it’ll work fine. That’s because running bsb -make-world recompiles the entire project, dependencies and all, using whichever version of bs-platform is installed. Doing so overwrites the JS files that were published by package A.

The problem doesn’t manifest itself until the point of distribution. If you follow the directions in the README by npm installing package B and running the binary, you should see undefined in place of 100.

Take a look at the package.json for each project:

{
  "name": "@ryb73/bs-51-52-interop-a",
  ...
  "files": [
    "/bsconfig.json",
    "/src",
    "/lib/js/src"
  ],
  ...
  "devDependencies": {
    "bs-platform": "^5.1.0"
  }
}
{
  "name": "@ryb73/bs-51-52-interop-b",
  ...
  "files": [
    "/bsconfig.json",
    "/src",
    "/lib/js/src"
  ],
  ...
  "devDependencies": {
    "bs-platform": "^5.2.0"
  },
  "dependencies": {
    "@ryb73/bs-51-52-interop-a": "0.1.0"
  }
 ...
}

Notice that @ryb73/bs-51-52-interop-a is a dependency, but bs-platform is a devDependency.

Notice also that I’m distributing the compiled JS files but not node_modules. This matches node/npm best practices.

And here’s the problem: when you run npm i @ryb73/bs-51-52-interop-b, npm downloads the published B source files which were compiled using 5.2.0. Because @ryb73/bs-51-52-interop-a is a dependency, npm will also download the published A source files which were compiled using 5.1.0. The published source files for A and the published source files for B are incompatible.

This is why (assuming my understanding of how Bucklescript packages should be published is correct) it’s important that new non-major versions of bs-platform maintain not only backwards compatibility of OCaml/Reason interfaces, but also backwards compatibility of the compiled JS interfaces.

It’s also possible that I’m doing something wrong with my distribution process. If that’s the case, it’d be great if the Reason team would communicate clearly

  • Which files I should publish as a Buscklescript library/app author
  • Into which dependency bucket (dep/devDeps/peerDeps/optionalDeps) I should put bs-platform and bs-dependencies
  • What constitutes a major vs minor vs patch update

#7

For BuckleScript packages, the source of truth is the OCaml/Reason source files, not the JavaScript output. The JS output is more of a ‘snapshot’ of what it looked like the last time it was checked in. You can check in JS files or not–it’s optional–but you should definitely not consider them the source of truth for consuming BuckleScript packages.

The correct practice here is to run bsb -make-world in the consuming project so that BuckleScript can recompile the entire dependency tree. Otherwise it can’t guarantee consistency of modules across different versions.


#8

I feel like I’m missing something. Essentially what you’re saying is that distributing BuckleScript binaries is not supported? Projects like yarn, lerna, babel-cli, typescript, webpack… these types of projects cannot/should not be implemented in BuckleScript?

Or if I’m at an organization that has a large number of modules written in JS, I can’t dip my toe into BuckleScript by rewriting one module without introducing BuckleScript compilation into the integration pipeline for all modules that depend on it?


#9

I don’t see any problems with distributing BuckleScript ‘binaries’ i.e. scripts to run with NodeJS, as long as they were compiled with bsb -make-world.

If you have a large number of JS modules, and want to migrate a few of them to BuckleScript, so that other JS modules can call them, that is also well-supported provided you keep in mind a couple of interop issues. In the context of nested modules, in the past you had to ensure that you exposed them in a way that made sense to JavaScript consumers, i.e. you wouldn’t want to expose BuckleScript’s previous array representation of nested modules. This was a known issue and (I assume) a long-term goal to fix to make interop easier. Now that modules are encoded as JS objects, this has been achieved.

What is not supported, is compiling BuckleScript modules to JS, then targeting those output JS files from the output JS of other BuckleScript modules. BuckleScript needs to control the compilation of the entire module graph, for it to guarantee consistency.

I hope that clears things up.


#10

Unfortunately it raises more questions than it answers :smile:

Let’s focus on just the example project for now. What changes would you make to project B so that users can just do npm i @ryb73/bs-51-52-interop-b and then run bs-51-52-interop-b? I want the end user to not have to run bsb.

Thanks for bearing with me, btw. This has been a point of confusion for me for a long time now.


#11

No worries. Regarding your question, as far as I can tell it should be enough to run npx bsb -make-world in bs-51-52-interop-b, check in that output, and publish. The user should now be able to run it directly.


#12

That’s essentially what I did to publish the existing version and it doesn’t work (I used yarn build instead of npx bsb -make-world but should be the same thing; I can confirm later if not)

I’m starting to wonder if has something to do with the fact that I’m not using in-source?


#13

npx bsb -make-world and yarn build in directory b should be the same thing, by definition. I ran the former. After that here is what I have:

node_modules/@ryb73/bs-51-52-interop-a/lib/js/src/A.js

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';


var Inner = {
  value: 100
};

exports.Inner = Inner;
/* No side effect */

b/lib/js/src/index.js

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';

var A = require("@ryb73/bs-51-52-interop-a/lib/js/src/A.js");

console.log("--", A.Inner.value, "--");

/*  Not a pure module */

#14

Yep, that looks right. Now publish it.

It doesn’t work.

The reason is that the node_modules/@ryb73/bs-51-52-interop-a/lib/js/src/A.js doesn’t get published. When a user runs npm i @ryb73/bs-51-52-interop-b, they’re not getting the A.js that you posted. They’re getting the A.js that’s published to @ryb73/bs-51-52-interop-a, which is:

// Generated by BUCKLESCRIPT, PLEASE EDIT WITH CARE
'use strict';


var Inner = /* module */[/* value */100];

exports.Inner = Inner;
/* No side effect */

#15

True. For a script that needs to be set up with npm install or similar, it will download dependency packages and try to use them, and it won’t work. However, keep in mind that:

  • Scripts like webpack etc., are generally not distributed in this form, they’re distributed as JS bundles with some kind of wrapper. So they shouldn’t face this issue.
  • Modules which import other JavaScript modules from npm, have no guarantee of what kind of JavaScript they’re getting. It may be ES4, 5, 6, Babel with non-standard extensions … anything really. It’s almost always the case that these modules need to run some kind of compilation pipeline before they’re usable. BuckleScript is not an exception to this.
  • BuckleScript has broken JavaScript output compatibility in the past, significantly with the change which mapped option types to JavaScript undefined. It typically does this when it finds a really good, idiomatic mapping that can have a lot of benefits like code size reduction or usability.

Care needs to be taken but in general the nested module output is not a breaking change at the BuckleScript level.


#16

Ok, this argument makes sense. Thanks for clearing it up.

I think part of my confusion came from the BuckleScript docs. For example:

Package Management

We use NPM and Yarn. Since the generated output is clean enough, you can publish them at NPM prepublishOnly time and remove all trace of BuckleScript beforehand. The consumer wouldn’t even know you wrote in BS, not JS! Check your node_modules right now; you might have been using some transitive BS code without knowing!

This makes it sound like the recommendation is to publish the compiled JS, but from talking to you it sounds like that’s not recommended.


#17

That depends. It’s totally possible to publish JavaScript output of BuckleScript compilation as a JavaScript library. I’ve done it myself: https://www.npmjs.com/package/@yawaramin/dbc

But it does require you to understand the BuckleScript -> JavaScript mapping mechanisms, and use only the explicitly guaranteed mappings for JavaScript interop, e.g. OCaml array -> JavaScript array, OCaml tuple -> JavaScript array, etc. We probably need to update the mappings a bit: https://bucklescript.github.io/docs/en/common-data-types#cheat-sheet


#18

I mean yeah, I’ve been doing it too, for years. What I’m just learning is that it should be the exception rather than the rule. IMO the docs muddy that distinction.


#19

Agreed, they could use some fine print to explain this.


#20

Hi, are you aware of tools like https://github.com/zeit/ncc
There are two kinds of backwards compatibility, source level backwards compatibility and ABI backwards compatibility (generated JS output).

The latter is very hard to maintain, it is not relevant under some cases (mono-repo style). If we have to bump major version for breaking ABI compatibility, essentially we have to bump major version for each release (Note ocaml native compiler does not provide ABI backwards compatibility either), maybe we should revisit it when reach a high level of stability.

We would like to learn more about your use case so that we can better server your use case.