Binary distributions

These notes are for those wishing to compile a binary distribution of Julia for distribution on various platforms. We love users spreading Julia as far and wide as they can, trying it out on as wide an array of operating systems and hardware configurations as possible. As each platform has specific gotchas and processes that must be followed in order to create a portable, working Julia distribution, we have separated most of the notes by OS.

Note that while the code for Julia is MIT-licensed, with a few exceptions, the distribution created by the techniques described herein will be GPL licensed, as various dependent libraries such as SuiteSparse are GPL licensed. We do hope to have a non-GPL distribution of Julia in the future.

Versioning and Git

The Makefile uses both the VERSION file and commit hashes and tags from the git repository to generate the base/version_git.jl with information we use to fill the splash screen and the versioninfo() output. If you for some reason don't want to have the git repository available when building you should pregenerate the base/version_git.jl file with:

make -C base version_git.jl.phony

Julia has lots of build dependencies where we use patched versions that has not yet been included by the popular package managers. These dependencies will usually be automatically downloaded when you build, but if you want to be able to build Julia on a computer without internet access you should create a full-source-dist archive with the special make target

make full-source-dist

that creates a julia-version-commit.tar.gz archive with all required dependencies.

When compiling a tagged release in the git repository, we don't display the branch/commit hash info in the splash screen. You can use this line to show a release description of up to 45 characters. To set this line you have to create a Make.user file containing:

override TAGGED_RELEASE_BANNER = "my-package-repository build"

Target Architectures

By default, Julia optimizes its system image to the native architecture of the build machine. This is usually not what you want when building packages, as it will make Julia fail at startup on any machine with incompatible CPUs (in particular older ones with more restricted instruction sets).

We therefore recommend that you pass the MARCH variable when calling make, setting it to the baseline target you intend to support. This will determine the target CPU for both the Julia executable and libraries, and the system image (the latter can also be set using JULIA_CPU_TARGET). Typically useful values for x86 CPUs are x86-64 and core2 (for 64-bit builds) and pentium4 (for 32-bit builds). Unfortunately, CPUs older than Pentium 4 are currently not supported (see this issue).

The full list of CPU targets supported by LLVM can be obtained by running llc -mattr=help.

Linux

On Linux, make binary-dist creates a tarball that contains a fully functional Julia installation. If you wish to create a distribution package such as a .deb, or .rpm, some extra effort is needed. See the julia-debian repository for an example of what metadata is needed for creating .deb packages for Debian and Ubuntu-based systems. See the Fedora package for RPM-based distributions. Although we have not yet experimented with it, Alien could be used to generate Julia packages for various Linux distributions.

Julia supports overriding standard installation directories via prefix and other environment variables you can pass when calling make and make install. See Make.inc for their list. DESTDIR can also be used to force the installation into a temporary directory.

By default, Julia loads $prefix/etc/julia/startup.jl as an installation-wide initialization file. This file can be used by distribution managers to set up custom paths or initialization code. For Linux distribution packages, if $prefix is set to /usr, there is no /usr/etc to look into. This requires the path to Julia's private etc directory to be changed. This can be done via the sysconfdir make variable when building. Simply pass sysconfdir=/etc to make when building and Julia will first check /etc/julia/startup.jl before trying $prefix/etc/julia/startup.jl.

OS X

To create a binary distribution on OSX, build Julia first, then cd to contrib/mac/app, and run make with the same makevars that were used with make when building Julia proper. This will then create a .dmg file in the contrib/mac/app directory holding a completely self-contained Julia.app.

Alternatively, Julia may be built as a framework by invoking make with the darwinframework target and DARWIN_FRAMEWORK=1 set. For example, make DARWIN_FRAMEWORK=1 darwinframework.

Windows

Instructions for creating a Julia distribution on Windows are described in the build devdocs for Windows.

Notes on BLAS and LAPACK

Julia builds OpenBLAS by default, which includes the BLAS and LAPACK libraries. On 32-bit architectures, Julia builds OpenBLAS to use 32-bit integers, while on 64-bit architectures, Julia builds OpenBLAS to use 64-bit integers (ILP64). It is essential that all Julia functions that call BLAS and LAPACK API routines use integers of the correct width.

Most BLAS and LAPACK distributions provided on linux distributions, and even commercial implementations ship libraries that use 32-bit APIs. In many cases, a 64-bit API is provided as a separate library.

When using vendor provided or OS provided libraries, a make option called USE_BLAS64 is available as part of the Julia build. When doing make USE_BLAS64=0, Julia will call BLAS and LAPACK assuming a 32-bit API, where all integers are 32-bit wide, even on a 64-bit architecture.

Other libraries that Julia uses, such as SuiteSparse also use BLAS and LAPACK internally. The APIs need to be consistent across all libraries that depend on BLAS and LAPACK. The Julia build process will build all these libraries correctly, but when overriding defaults and using system provided libraries, this consistency must be ensured.

Also note that Linux distributions sometimes ship several versions of OpenBLAS, some of which enable multithreading, and others only working in a serial fashion. For example, in Fedora, libopenblasp.so is threaded, but libopenblas.so is not. We recommend using the former for optimal performance. To choose an OpenBLAS library whose name is different from the default libopenblas.so, pass LIBBLAS=-l$(YOURBLAS) and LIBBLASNAME=lib$(YOURBLAS) to make, replacing $(YOURBLAS) with the name of your library. You can also add .so.0 to the name of the library if you want your package to work without requiring the unversioned .so symlink.

Finally, OpenBLAS includes its own optimized version of LAPACK. If you set USE_SYSTEM_BLAS=1 and USE_SYSTEM_LAPACK=1, you should also set LIBLAPACK=-l$(YOURBLAS) and LIBLAPACKNAME=lib$(YOURBLAS). Else, the reference LAPACK will be used and performance will typically be much lower.

Starting with Julia 1.7, Julia uses libblastrampoline to pick a different BLAS at runtime.

Point releasing 101

Creating a point/patch release consists of several distinct steps.

Backporting commits

Some pull requests are labeled "backport pending x.y", e.g. "backport pending 0.6". This designates that the next subsequent release tagged from the release-x.y branch should include the commit(s) in that pull request. Once the pull request is merged into master, each of the commits should be cherry picked to a dedicated branch that will ultimately be merged into release-x.y.

Creating a backports branch

First, create a new branch based on release-x.y. The typical convention for Julia branches is to prefix the branch name with your initials if it's intended to be a personal branch. For the sake of example, we'll say that the author of the branch is Jane Smith.

git fetch origin
git checkout release-x.y
git rebase origin/release-x.y
git checkout -b js/backport-x.y

This ensures that your local copy of release-x.y is up to date with origin before you create a new branch from it.

Cherry picking commits

Now we do the actual backporting. Find all merged pull requests labeled "backport pending x.y" in the GitHub web UI. For each of these, scroll to the bottom where it says "someperson merged commit 123abc into master XX minutes ago". Note that the commit name is a link; if you click it, you'll be shown the contents of the commit. If this page shows that 123abc is a merge commit, go back to the PR page–-we don't want merge commits, we want the actual commits. However, if this does not show a merge commit, it means that the PR was squash-merged. In that case, use the git SHA of the commit, listed next to commit on this page.

Once you have the SHA of the commit, cherry-pick it onto the backporting branch:

git cherry-pick -x -e <sha>

There may be conflicts which need to be resolved manually. Once conflicts are resolved (if applicable), add a reference to the GitHub pull request that introduced the commit in the body of the commit message.

After all of the relevant commits are on the backports branch, push the branch to GitHub.

Checking for performance regressions

Point releases should never introduce performance regressions. Luckily the Julia benchmarking bot, Nanosoldier, can run benchmarks against any branch, not just master. In this case we want to check the benchmark results of js/backport-x.y against release-x.y. To do this, awaken the Nanosoldier from his robotic slumber using a comment on your backporting pull request:

@nanosoldier `runbenchmarks(ALL, vs=":release-x.y")`

This will run all registered benchmarks on release-x.y and js/backport-x.y and produce a summary of results, marking all improvements and regressions.

If Nanosoldier finds any regressions, try verifying locally and rerun Nanosoldier if necessary. If the regressions are deemed to be real rather than just noise, you'll have to find a commit on master to backport that fixes it if one exists, otherwise you should determine what caused the regression and submit a patch (or get someone who knows the code to submit a patch) to master, then backport the commit once that's merged. (Or submit a patch directly to the backport branch if appropriate.)

Building test binaries

After the backport PR has been merged into the release-x.y branch, update your local clone of Julia, then get the SHA of the branch using

git rev-parse origin/release-x.y

Keep that handy, as it's what you'll enter in the "Revision" field in the buildbot UI.

For now, all you need are binaries for Linux x86-64, since this is what's used for running PackageEvaluator. Go to https://buildog.julialang.org, submit a job for nuke_linux64, then queue up a job for package_linux64, providing the SHA as the revision. When the packaging job completes, it will upload the binary to the julialang2 bucket on AWS. Retrieve the URL, as it will be used for PackageEvaluator.

Checking for package breakages

Point releases should never break packages, with the possible exception of packages that are doing some seriously questionable hacks using Base internals that are not intended to be user-facing. (In those cases, maybe have a word with the package author.)

Checking whether changes made in the forthcoming new version will break packages can be accomplished using PackageEvaluator, often called "PkgEval" for short. PkgEval is what populates the status badges on GitHub repos and on pkg.julialang.org. It typically runs on one of the non-benchmarking nodes of Nanosoldier and uses Vagrant to perform its duties in separate, parallel VirtualBox virtual machines.

Setting up PackageEvaluator

Clone PackageEvaluator and create a branch called backport-x.y.z, and check it out. Note that the required changes are a little hacky and confusing, and hopefully that will be addressed in a future version of PackageEvaluator. The changes to make will be modeled off of this commit.

The setup script takes its first argument as the version of Julia to run and the second as the range of package names (AK for packages named A-K, LZ for L-Z). The basic idea is that we're going to tweak that a bit to run only two versions of Julia, the current x.y release and our backport version, each with three ranges of packages.

In the linked diff, we're saying that if the second argument is LZ, use the binaries built from our backport branch, otherwise (AK) use the release binaries. Then we're using the first argument to run a section of the package list: A-F for input 0.4, G-N for 0.5, and O-Z for 0.6.

Running PackageEvaluator

To run PkgEval, find a hefty enough machine (such as Nanosoldier node 1), then run

git clone https://github.com/JuliaCI/PackageEvaluator.jl.git
cd PackageEvaluator.jl/scripts
git checkout backport-x.y.z
./runvagrant.sh

This produces some folders in the scripts/ directory. The folder names and their contents are decoded below:

Investigating results

Once that's done, you can use ./summary.sh from that same directory to produce a summary report of the findings. We'll do so for each of the folders to aggregate overall results by version.

./summary.sh 0.4AK/*.json > summary_release.txt
./summary.sh 0.5AK/*.json >> summary_release.txt
./summary.sh 0.6AK/*.json >> summary_release.txt
./summary.sh 0.4LZ/*.json > summary_backport.txt
./summary.sh 0.5LZ/*.json >> summary_backport.txt
./summary.sh 0.6LZ/*.json >> summary_backport.txt

Now we have two files, summary_release.txt and summary_backport.txt, containing the PackageEvaluator test results (pass/fail) for each package for the two versions.

To make these easier to ingest into a Julia, we'll convert them into CSV files then use the DataFrames package to process the results. To convert to CSV, copy each .txt file to a corresponding .csv file, then enter Vim and execute ggVGI"<esc> then :%s/\.json /",/g. (You don't have to use Vim; this just is one way to do it.) Now process the results with Julia code similar to the following.

using DataFrames

release = readtable("summary_release.csv", header=false, names=[:package, :release])
backport = readtable("summary_backport.csv", header=false, names=[:package, :backport])

results = join(release, backport, on=:package, kind=:outer)

for result in eachrow(results)
    a = result[:release]
    b = result[:backport]
    if (isna(a) && !isna(b)) || (isna(b) && !isna(a))
        color = :yellow
    elseif a != b && occursin("pass", b)
        color = :green
    elseif a != b
        color = :red
    else
        continue
    end
    printstyled(result[:package], ": Release ", a, " -> Backport ", b, "\n", color=color)
end

This will write color-coded lines to stdout. All lines in red must be investigated as they signify potential breakages caused by the backport version. Lines in yellow should be looked into since it means a package ran on one version but not on the other for some reason. If you find that your backported branch is causing breakages, use git bisect to identify the problematic commits, git revert those commits, and repeat the process.

Merging backports into the release branch

After you have ensured that

• the backported commits pass all of Julia's unit tests,
• there are no performance regressions introduced by the backported commits as compared to the release branch, and
• the backported commits do not break any registered packages,

then the backport branch is ready to be merged into release-x.y. Once it's merged, go through and remove the "backport pending x.y" label from all pull requests containing the commits that have been backported. Do not remove the label from PRs that have not been backported.

The release-x.y branch should now contain all of the new commits. The last thing we want to do to the branch is to adjust the version number. To do this, submit a PR against release-x.y that edits the VERSION file to remove -pre from the version number. Once that's merged, we're ready to tag.

Tagging the release

It's time! Check out the release-x.y branch and make sure that your local copy of the branch is up to date with the remote branch. At the command line, run

git tag v$(cat VERSION)
git push --tags

This creates the tag locally and pushes it to GitHub.

After tagging the release, submit another PR to release-x.y to bump the patch number and add -pre back to the end. This denotes that the branch state reflects a prerelease version of the next point release in the x.y series.

Follow the remaining directions in the Makefile.

Signing binaries

Some of these steps will require secure passwords. To obtain the appropriate passwords, contact Elliot Saba (staticfloat) or Alex Arslan (ararslan). Note that code signing for each platform must be performed on that platform (e.g. Windows signing must be done on Windows, etc.).

Linux

Code signing must be done manually on Linux, but it's quite simple. First obtain the file julia.key from the CodeSigning folder in the juliasecure AWS bucket. Add this to your GnuPG keyring using

gpg --import julia.key

This will require entering a password that you must obtain from Elliot or Alex. Next, set the trust level for the key to maximum. Start by entering a gpg session:

gpg --edit-key julia

At the prompt, type trust, then when asked for a trust level, provide the maximum available (likely 5). Exit GnuPG.

Now, for each of the Linux tarballs that were built on the buildbots, enter

gpg -u julia --armor --detach-sig julia-x.y.z-linux-<arch>.tar.gz

This will produce a corresponding .asc file for each tarball. And that's it!

macOS

Code signing should happen automatically on the macOS buildbots. However, it's important to verify that it was successful. On a system or virtual machine running macOS, download the .dmg file that was built on the buildbots. For the sake of example, say that the .dmg file is called julia-x.y.z-osx.dmg. Run

mkdir ./jlmnt
hdiutil mount -readonly -mountpoint ./jlmnt julia-x.y.z-osx.dmg
codesign -v jlmnt/Julia-x.y.app

Be sure to note the name of the mounted disk listed when mounting! For the sake of example, we'll assume this is disk3. If the code signing verification exited successfully, there will be no output from the codesign step. If it was indeed successful, you can detach the .dmg now:

hdiutil eject /dev/disk3
rm -rf ./jlmnt

If you get a message like

Julia-x.y.app: code object is not signed at all

then you'll need to sign manually.

To sign manually, first retrieve the OS X certificates from the CodeSigning folder in the juliasecure bucket on AWS. Add the .p12 file to your keychain using Keychain.app. Ask Elliot Saba (staticfloat) or Alex Arslan (ararslan) for the password for the key. Now run

hdiutil convert julia-x.y.z-osx.dmg -format UDRW -o julia-x.y.z-osx_writable.dmg
mkdir ./jlmnt
hdiutil mount -mountpoint julia-x.y.z-osx_writable.dmg
codesign -s "AFB379C0B4CBD9DB9A762797FC2AB5460A2B0DBE" --deep jlmnt/Julia-x.y.app

This may fail with a message like

Julia-x.y.app: resource fork, Finder information, or similar detritus not allowed

If that's the case, you'll need to remove extraneous attributes:

xattr -cr jlmnt/Julia-x.y.app

Then retry code signing. If that produces no errors, retry verification. If all is now well, unmount the writable .dmg and convert it back to read-only:

hdiutil eject /dev/disk3
rm -rf ./jlmnt
hdiutil convert julia-x.y.z-osx_writable.dmg -format UDZO -o julia-x.y.z-osx_fixed.dmg

Verify that the resulting .dmg is in fact fixed by double clicking it. If everything looks good, eject it then drop the _fixed suffix from the name. And that's it!

Windows

Signing must be performed manually on Windows. First obtain the Windows 10 SDK, which contains the necessary signing utilities, from the Microsoft website. We need the SignTool utility which should have been installed somewhere like C:\Program Files (x86)\Windows Kits\10\App Certification Kit. Grab the Windows certificate files from CodeSigning on juliasecure and put them in the same directory as the executables. Open a Windows CMD window, cd to where all the files are, and run

set PATH=%PATH%;C:\Program Files (x86)\Windows Kits\10\App Certification Kit;
signtool sign /f julia-windows-code-sign_2017.p12 /p "PASSWORD" ^
   /t http://timestamp.verisign.com/scripts/timstamp.dll ^
   /v julia-x.y.z-win32.exe

Note that ^ is a line continuation character in Windows CMD and PASSWORD is a placeholder for the password for this certificate. As usual, contact Elliot or Alex for passwords. If there are no errors, we're all good!

Uploading binaries

Now that everything is signed, we need to upload the binaries to AWS. You can use a program like Cyberduck or the aws command line utility. The binaries should go in the julialang2 bucket in the appropriate folders. For example, Linux x86-64 goes in julialang2/bin/linux/x.y. Be sure to delete the current julia-x.y-latest-linux-<arch>.tar.gz file and replace it with a duplicate of julia-x.y.z-linux-<arch>.tar.gz.

We also need to upload the checksums for everything we've built, including the source tarballs and all release binaries. This is simple:

shasum -a 256 julia-x.y.z* | grep -v -e sha256 -e md5 -e asc > julia-x.y.z.sha256
md5sum julia-x.y.z* | grep -v -e sha256 -e md5 -e asc > julia-x.y.z.md5

Note that if you're running those commands on macOS, you'll get very slightly different output, which can be reformatted by looking at an existing file. Mac users will also need to use md5 -r instead of md5sum. Upload the .md5 and .sha256 files to julialang2/bin/checksums on AWS.

Ensure that the permissions on AWS for all uploaded files are set to "Everyone: READ."

For each file we've uploaded, we need to purge the Fastly cache so that the links on the website point to the updated files. As an example:

curl -X PURGE https://julialang-s3.julialang.org/bin/checksums/julia-x.y.z.sha256

Sometimes this isn't necessary but it's good to do anyway.