Package Status¶

The Pkg.status() function prints out a summary of the state of packages you have installed. Initially, you’ll have no packages installed:

julia>Pkg.status()INFO:Initializingpackagerepository/Users/stefan/.julia/v0.5INFO:CloningMETADATAfromgit://github.com/JuliaLang/METADATA.jlNopackagesinstalled.

Your package directory is automatically initialized the first time you run a Pkg command that expects it to exist – which includes Pkg.status(). Here’s an example non-trivial set of required and additional packages:

julia>Pkg.status()Requiredpackages:-Distributions0.2.8-UTF160.2.0Additionalpackages:-NumericExtensions0.2.17-Stats0.2.6

These packages are all on registered versions, managed by Pkg. Packages can be in more complicated states, indicated by annotations to the right of the installed package version; we will explain these states and annotations as we encounter them. For programmatic usage, Pkg.installed() returns a dictionary, mapping installed package names to the version of that package which is installed:

julia>Pkg.installed()Dict{String,VersionNumber}with4entries:"Distributions"=>v"0.2.8""Stats"=>v"0.2.6""UTF16"=>v"0.2.0""NumericExtensions"=>v"0.2.17"

Adding and Removing Packages¶

Julia’s package manager is a little unusual in that it is declarative rather than imperative. This means that you tell it what you want and it figures out what versions to install (or remove) to satisfy those requirements optimally – and minimally. So rather than installing a package, you just add it to the list of requirements and then “resolve” what needs to be installed. In particular, this means that if some package had been installed because it was needed by a previous version of something you wanted, and a newer version doesn’t have that requirement anymore, updating will actually remove that package.

Your package requirements are in the file ~/.julia/v0.5/REQUIRE. You can edit this file by hand and then call Pkg.resolve() to install, upgrade or remove packages to optimally satisfy the requirements, or you can do Pkg.edit(), which will open REQUIRE in your editor (configured via the EDITOR or VISUAL environment variables), and then automatically call Pkg.resolve() afterwards if necessary. If you only want to add or remove the requirement for a single package, you can also use the non-interactive Pkg.add() and Pkg.rm() commands, which add or remove a single requirement to REQUIRE and then call Pkg.resolve().

You can add a package to the list of requirements with the Pkg.add() function, and the package and all the packages that it depends on will be installed:

julia>Pkg.status()Nopackagesinstalled.julia>Pkg.add("Distributions")INFO:CloningcacheofDistributionsfromgit://github.com/JuliaStats/Distributions.jl.gitINFO:CloningcacheofNumericExtensionsfromgit://github.com/lindahua/NumericExtensions.jl.gitINFO:CloningcacheofStatsfromgit://github.com/JuliaStats/Stats.jl.gitINFO:InstallingDistributionsv0.2.7INFO:InstallingNumericExtensionsv0.2.17INFO:InstallingStatsv0.2.6INFO:REQUIREupdated.julia>Pkg.status()Requiredpackages:-Distributions0.2.7Additionalpackages:-NumericExtensions0.2.17-Stats0.2.6

What this is doing is first adding Distributions to your ~/.julia/v0.5/REQUIRE file:

$cat~/.julia/v0.5/REQUIREDistributions

It then runs Pkg.resolve() using these new requirements, which leads to the conclusion that the Distributions package should be installed since it is required but not installed. As stated before, you can accomplish the same thing by editing your ~/.julia/v0.5/REQUIRE file by hand and then running Pkg.resolve() yourself:

$echoUTF16>>~/.julia/v0.5/REQUIREjulia>Pkg.resolve()INFO:CloningcacheofUTF16fromgit://github.com/nolta/UTF16.jl.gitINFO:InstallingUTF16v0.2.0julia>Pkg.status()Requiredpackages:-Distributions0.2.7-UTF160.2.0Additionalpackages:-NumericExtensions0.2.17-Stats0.2.6

This is functionally equivalent to calling Pkg.add("UTF16"), except that Pkg.add() doesn’t change REQUIRE until after installation has completed, so if there are problems, REQUIRE will be left as it was before calling Pkg.add(). The format of the REQUIRE file is described in Requirements Specification; it allows, among other things, requiring specific ranges of versions of packages.

When you decide that you don’t want to have a package around any more, you can use Pkg.rm() to remove the requirement for it from the REQUIRE file:

julia>Pkg.rm("Distributions")INFO:RemovingDistributionsv0.2.7INFO:RemovingStatsv0.2.6INFO:RemovingNumericExtensionsv0.2.17INFO:REQUIREupdated.julia>Pkg.status()Requiredpackages:-UTF160.2.0julia>Pkg.rm("UTF16")INFO:RemovingUTF16v0.2.0INFO:REQUIREupdated.julia>Pkg.status()Nopackagesinstalled.

Once again, this is equivalent to editing the REQUIRE file to remove the line with each package name on it then running Pkg.resolve() to update the set of installed packages to match. While Pkg.add() and Pkg.rm() are convenient for adding and removing requirements for a single package, when you want to add or remove multiple packages, you can call Pkg.edit() to manually change the contents of REQUIRE and then update your packages accordingly. Pkg.edit() does not roll back the contents of REQUIRE if Pkg.resolve() fails – rather, you have to run Pkg.edit() again to fix the files contents yourself.

Because the package manager uses libgit2 internally to manage the package git repositories, users may run into protocol issues (if behind a firewall, for example), when running Pkg.add(). By default, all GitHub-hosted packages wil be accessed via ‘https’; this default can be modified by calling Pkg.setprotocol!(). The following command can be run from the command line in order to tell git to use ‘https’ instead of the ‘git’ protocol when cloning all repositories, wherever they are hosted:

gitconfig--globalurl."https://".insteadOfgit://

However, this change will be system-wide and thus the use of Pkg.setprotocol!() is preferable.

Offline Installation of Packages¶

For machines with no Internet connection, packages may be installed by copying the package root directory (given by Pkg.dir()) from a machine with the same operating system and environment.

Pkg.add() does the following within the package root directory:

1. Adds the name of the package to REQUIRE.
2. Downloads the package to .cache, then copies the package to the package root directory.
3. Recursively performs step 2 against all the packages listed in the package’s REQUIRE file.
4. Runs Pkg.build()

Installing Unregistered Packages¶

Julia packages are simply git repositories, clonable via any of the protocols that git supports, and containing Julia code that follows certain layout conventions. Official Julia packages are registered in the METADATA.jl repository, available at a well-known location [1]. The Pkg.add() and Pkg.rm() commands in the previous section interact with registered packages, but the package manager can install and work with unregistered packages too. To install an unregistered package, use Pkg.clone(url), where url is a git URL from which the package can be cloned:

julia> Pkg.clone("git://example.com/path/to/Package.jl.git")
INFO: Cloning Package from git://example.com/path/to/Package.jl.git
Cloning into 'Package'...
remote: Counting objects: 22, done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 22 (delta 8), reused 22 (delta 8)
Receiving objects: 100% (22/22), 2.64 KiB, done.
Resolving deltas: 100% (8/8), done.

By convention, Julia repository names end with .jl (the additional .git indicates a “bare” git repository), which keeps them from colliding with repositories for other languages, and also makes Julia packages easy to find in search engines. When packages are installed in your .julia/v0.5 directory, however, the extension is redundant so we leave it off.

If unregistered packages contain a REQUIRE file at the top of their source tree, that file will be used to determine which registered packages the unregistered package depends on, and they will automatically be installed. Unregistered packages participate in the same version resolution logic as registered packages, so installed package versions will be adjusted as necessary to satisfy the requirements of both registered and unregistered packages.

Updating Packages¶

When package developers publish new registered versions of packages that you’re using, you will, of course, want the new shiny versions. To get the latest and greatest versions of all your packages, just do Pkg.update():

julia>Pkg.update()INFO:UpdatingMETADATA...INFO:Computingchanges...INFO:UpgradingDistributions:v0.2.8=>v0.2.10INFO:UpgradingStats:v0.2.7=>v0.2.8

The first step of updating packages is to pull new changes to ~/.julia/v0.5/METADATA and see if any new registered package versions have been published. After this, Pkg.update() attempts to update packages that are checked out on a branch and not dirty (i.e. no changes have been made to files tracked by git) by pulling changes from the package’s upstream repository. Upstream changes will only be applied if no merging or rebasing is necessary – i.e. if the branch can be “fast-forwarded”. If the branch cannot be fast-forwarded, it is assumed that you’re working on it and will update the repository yourself.

Finally, the update process recomputes an optimal set of package versions to have installed to satisfy your top-level requirements and the requirements of “fixed” packages. A package is considered fixed if it is one of the following:

1. Unregistered: the package is not in METADATA – you installed it with Pkg.clone().
2. Checked out: the package repo is on a development branch.
3. Dirty: changes have been made to files in the repo.

If any of these are the case, the package manager cannot freely change the installed version of the package, so its requirements must be satisfied by whatever other package versions it picks. The combination of top-level requirements in ~/.julia/v0.5/REQUIRE and the requirement of fixed packages are used to determine what should be installed.

You can also update only a subset of the installed packages, by providing arguments to the Pkg.update function. In that case, only the packages provided as arguments and their dependencies will be updated:

julia>Pkg.update("Example")INFO:UpdatingMETADATA...INFO:Computingchanges...INFO:UpgradingExample:v0.4.0=>0.4.1

This partial update process still computes the new set of package versions according to top-level requirements and “fixed” packages, but it additionally considers all other packages except those explicitly provided, and their dependencies, as fixed.

Checkout, Pin and Free¶

You may want to use the master version of a package rather than one of its registered versions. There might be fixes or functionality on master that you need that aren’t yet published in any registered versions, or you may be a developer of the package and need to make changes on master or some other development branch. In such cases, you can do Pkg.checkout(pkg) to checkout the master branch of pkg or Pkg.checkout(pkg,branch) to checkout some other branch:

julia>Pkg.add("Distributions")INFO:InstallingDistributionsv0.2.9INFO:InstallingNumericExtensionsv0.2.17INFO:InstallingStatsv0.2.7INFO:REQUIREupdated.julia>Pkg.status()Requiredpackages:-Distributions0.2.9Additionalpackages:-NumericExtensions0.2.17-Stats0.2.7julia>Pkg.checkout("Distributions")INFO:CheckingoutDistributionsmaster...INFO:Nopackagestoinstall,updateorremove.julia>Pkg.status()Requiredpackages:-Distributions0.2.9+masterAdditionalpackages:-NumericExtensions0.2.17-Stats0.2.7

Immediately after installing Distributions with Pkg.add() it is on the current most recent registered version – 0.2.9 at the time of writing this. Then after running Pkg.checkout("Distributions"), you can see from the output of Pkg.status() that Distributions is on an unregistered version greater than 0.2.9, indicated by the “pseudo-version” number 0.2.9+.

When you checkout an unregistered version of a package, the copy of the REQUIRE file in the package repo takes precedence over any requirements registered in METADATA, so it is important that developers keep this file accurate and up-to-date, reflecting the actual requirements of the current version of the package. If the REQUIRE file in the package repo is incorrect or missing, dependencies may be removed when the package is checked out. This file is also used to populate newly published versions of the package if you use the API that Pkg provides for this (described below).

When you decide that you no longer want to have a package checked out on a branch, you can “free” it back to the control of the package manager with Pkg.free(pkg):

julia>Pkg.free("Distributions")INFO:FreeingDistributions...INFO:Nopackagestoinstall,updateorremove.julia>Pkg.status()Requiredpackages:-Distributions0.2.9Additionalpackages:-NumericExtensions0.2.17-Stats0.2.7

After this, since the package is on a registered version and not on a branch, its version will be updated as new registered versions of the package are published.

If you want to pin a package at a specific version so that calling Pkg.update() won’t change the version the package is on, you can use the Pkg.pin() function:

julia>Pkg.pin("Stats")INFO:CreatingStatsbranchpinned.47c198b1.tmpjulia>Pkg.status()Requiredpackages:-Distributions0.2.9Additionalpackages:-NumericExtensions0.2.17-Stats0.2.7pinned.47c198b1.tmp

After this, the Stats package will remain pinned at version 0.2.7 – or more specifically, at commit 47c198b1, but since versions are permanently associated a given git hash, this is the same thing. Pkg.pin() works by creating a throw-away branch for the commit you want to pin the package at and then checking that branch out. By default, it pins a package at the current commit, but you can choose a different version by passing a second argument:

julia>Pkg.pin("Stats",v"0.2.5")INFO:CreatingStatsbranchpinned.1fd0983b.tmpINFO:Nopackagestoinstall,updateorremove.julia>Pkg.status()Requiredpackages:-Distributions0.2.9Additionalpackages:-NumericExtensions0.2.17-Stats0.2.5pinned.1fd0983b.tmp

Now the Stats package is pinned at commit 1fd0983b, which corresponds to version 0.2.5. When you decide to “unpin” a package and let the package manager update it again, you can use Pkg.free() like you would to move off of any branch:

julia>Pkg.free("Stats")INFO:FreeingStats...INFO:Nopackagestoinstall,updateorremove.julia>Pkg.status()Requiredpackages:-Distributions0.2.9Additionalpackages:-NumericExtensions0.2.17-Stats0.2.7

After this, the Stats package is managed by the package manager again, and future calls to Pkg.update() will upgrade it to newer versions when they are published. The throw-away pinned.1fd0983b.tmp branch remains in your local Stats repo, but since git branches are extremely lightweight, this doesn’t really matter; if you feel like cleaning them up, you can go into the repo and delete those branches [2].

Custom METADATA Repository¶

By default, Julia assumes you will be using the official METADATA.jl repository for downloading and installing packages. You can also provide a different metadata repository location. A common approach is to keep your metadata-v2 branch up to date with the Julia official branch and add another branch with your custom packages. You can initialize your local metadata repository using that custom location and branch and then periodically rebase your custom branch with the official metadata-v2 branch. In order to use a custom repository and branch, issue the following command:

julia>Pkg.init("https://me.example.com/METADATA.jl.git","branch")

The branch argument is optional and defaults to metadata-v2. Once initialized, a file named META_BRANCH in your ~/.julia/vX.Y/ path will track the branch that your METADATA repository was initialized with. If you want to change branches, you will need to either modify the META_BRANCH file directly (be careful!) or remove the vX.Y directory and re-initialize your METADATA repository using the Pkg.init command.

Initial Setup¶

Since packages are git repositories, before doing any package development you should setup the following standard global git configuration settings:

$gitconfig--globaluser.name"FULL NAME"$gitconfig--globaluser.email"EMAIL"

where FULLNAME is your actual full name (spaces are allowed between the double quotes) and EMAIL is your actual email address. Although it isn’t necessary to use GitHub to create or publish Julia packages, most Julia packages as of writing this are hosted on GitHub and the package manager knows how to format origin URLs correctly and otherwise work with the service smoothly. We recommend that you create a free account on GitHub and then do:

$gitconfig--globalgithub.user"USERNAME"

where USERNAME is your actual GitHub user name. Once you do this, the package manager knows your GitHub user name and can configure things accordingly. You should also upload your public SSH key to GitHub and set up an SSH agent on your development machine so that you can push changes with minimal hassle. In the future, we will make this system extensible and support other common git hosting options like BitBucket and allow developers to choose their favorite. Since the package development functions has been moved to the PkgDev package, you need to run Pkg.add("PkgDev");importPkgDev to access the functions starting with PkgDev. in the document below.

Making changes to an existing package¶

Pkg.checkout("Images")# check out the master branch<here,makesureyourbugisstillabugandhasn'tbeenfixedalready>cd(Pkg.dir("Images"));gitcheckout-bmyfixes# create a branch for your changes<editcode># be sure to add a test for your bugPkg.test("Images")# make sure everything works now;gitcommit-a-m"Fix foo by calling bar"# write a descriptive messageusingPkgDevPkgDev.submit("Images")

julia>cd(Pkg.dir("Foo"))# go to Foo's foldershell>gitcommandarguments...# command will apply to Foo

WIP: add new 1-line whizbang function (currently breaks package)
Finish whizbang function
Fix typo in variable name
Oops, don't forget to supply default argument
Split into two 1-line functions
Rats, forgot to export the second function
...

gitcheckoutfixbargitreset--hardfixbar_backup

Creating a new Package¶

julia>PkgDev.generate("FooBar","MIT")INFO:InitializingFooBarrepo:/Users/stefan/.julia/v0.5/FooBarINFO:Origin:git://github.com/StefanKarpinski/FooBar.jl.gitINFO:GeneratingLICENSE.mdINFO:GeneratingREADME.mdINFO:Generatingsrc/FooBar.jlINFO:Generatingtest/runtests.jlINFO:GeneratingREQUIREINFO:Generating.travis.ymlINFO:Generatingappveyor.ymlINFO:Generating.gitignoreINFO:CommittingFooBargeneratedfiles

$cd~/.julia/v0.5/FooBar&&gitshow--statcommit84b8e266dae6de30ab9703150b3bf771ec7b6285Author:StefanKarpinski<stefan@karpinski.org>Date:WedOct1617:57:582013-0400FooBar.jlgeneratedfiles.license:MITauthors:StefanKarpinskiyears:2013user:StefanKarpinskiJuliaVersion0.3.0-prerelease+3217[5fcfb13*].gitignore|2++.travis.yml|13+++++++++++++LICENSE.md|22+++++++++++++++++++++++README.md|3+++REQUIRE|1+appveyor.yml|34++++++++++++++++++++++++++++++++++src/FooBar.jl|5+++++test/runtests.jl|5+++++8fileschanged,85insertions(+)

git://github.com/StefanKarpinski/FooBar.jl.git

julia>Pkg.clone("git://github.com/StefanKarpinski/FooBar.jl.git")INFO:CloningFooBarfromgit@github.com:StefanKarpinski/FooBar.jl.git

julia>PkgDev.register("FooBar")INFO:RegisteringFooBaratgit://github.com/StefanKarpinski/FooBar.jl.gitINFO:CommittingMETADATAforFooBar

$cd~/.julia/v0.5/METADATA&&gitshowcommit9f71f4becb05cadacb983c54a72eed744e5c019dAuthor:StefanKarpinski<stefan@karpinski.org>Date:WedOct1618:46:022013-0400RegisterFooBardiff--gita/FooBar/urlb/FooBar/urlnewfilemode100644index0000000..30e525e---/dev/null+++b/FooBar/url@@-0,0+1@@+git://github.com/StefanKarpinski/FooBar.jl.git

julia>PkgDev.publish()INFO:ValidatingMETADATAINFO:NonewpackageversionstopublishINFO:SubmittingMETADATAchangesINFO:ForkingJuliaLang/METADATA.jltoStefanKarpinskiINFO:Pushingchangesasbranchpull-request/ef45f54bINFO:Tocreateapull-requestopen:https://github.com/StefanKarpinski/METADATA.jl/compare/pull-request/ef45f54b

ERROR:keynotfound:"token"

julia>PkgDev.tag("FooBar")INFO:TaggingFooBarv0.0.1INFO:CommittingMETADATAforFooBar

$cd~/.julia/v0.5/FooBar&&gittagv0.0.1

$cd~/.julia/v0.5/FooBar&&gitshowcommitde77ee4dc0689b12c5e8b574aef7f70e8b311b0eAuthor:StefanKarpinski<stefan@karpinski.org>Date:WedOct1623:06:182013-0400TagFooBarv0.0.1diff--gita/FooBar/versions/0.0.1/sha1b/FooBar/versions/0.0.1/sha1newfilemode100644index0000000..c1cb1c1---/dev/null+++b/FooBar/versions/0.0.1/sha1@@-0,0+1@@+84b8e266dae6de30ab9703150b3bf771ec7b6285

julia>PkgDev.publish()INFO:ValidatingMETADATAINFO:PushingFooBarpermanenttags:v0.0.1INFO:SubmittingMETADATAchangesINFO:ForkingJuliaLang/METADATA.jltoStefanKarpinskiINFO:Pushingchangesasbranchpull-request/3ef4f5c4INFO:Tocreateapull-requestopen:https://github.com/StefanKarpinski/METADATA.jl/compare/pull-request/3ef4f5c4

cd~/.julia/v0.5/METADATAgitremoteaddUSERNAMEhttps://github.com/USERNAME/METADATA.jl.git

Fixing Package Requirements¶

If you need to fix the registered requirements of an already-published package version, you can do so just by editing the metadata for that version, which will still have the same commit hash – the hash associated with a version is permanent:

$cd~/.julia/v0.5/METADATA/FooBar/versions/0.0.1&&catrequiresjulia0.3-$virequires

Since the commit hash stays the same, the contents of the REQUIRE file that will be checked out in the repo will not match the requirements in METADATA after such a change; this is unavoidable. When you fix the requirements in METADATA for a previous version of a package, however, you should also fix the REQUIRE file in the current version of the package.

Requirements Specification¶

The ~/.julia/v0.5/REQUIRE file, the REQUIRE file inside packages, and the METADATA package requires files use a simple line-based format to express the ranges of package versions which need to be installed. Package REQUIRE and METADATArequires files should also include the range of versions of julia the package is expected to work with.

Here’s how these files are parsed and interpreted.

• Everything after a # mark is stripped from each line as a comment.
• If nothing but whitespace is left, the line is ignored.
• If there are non-whitespace characters remaining, the line is a requirement and the is split on whitespace into words.

The simplest possible requirement is just the name of a package name on a line by itself:

Distributions

This requirement is satisfied by any version of the Distributions package. The package name can be followed by zero or more version numbers in ascending order, indicating acceptable intervals of versions of that package. One version opens an interval, while the next closes it, and the next opens a new interval, and so on; if an odd number of version numbers are given, then arbitrarily large versions will satisfy; if an even number of version numbers are given, the last one is an upper limit on acceptable version numbers. For example, the line:

Distributions0.1

is satisfied by any version of Distributions greater than or equal to 0.1.0. Suffixing a version with - allows any pre-release versions as well. For example:

Distributions0.1-

is satisfied by pre-release versions such as 0.1-dev or 0.1-rc1, or by any version greater than or equal to 0.1.0.

This requirement entry:

Distributions0.10.2.5

is satisfied by versions from 0.1.0 up to, but not including 0.2.5. If you want to indicate that any 0.1.x version will do, you will want to write:

Distributions0.10.2-

If you want to start accepting versions after 0.2.7, you can write:

Distributions0.10.2-0.2.7

If a requirement line has leading words that begin with @, it is a system-dependent requirement. If your system matches these system conditionals, the requirement is included, if not, the requirement is ignored. For example:

@osxHomebrew

will require the Homebrew package only on systems where the operating system is OS X. The system conditions that are currently supported are (hierarchically):

• @unix
  • @linux
  • @bsd
    • @osx
• @windows

The @unix condition is satisfied on all UNIX systems, including Linux and BSD. Negated system conditionals are also supported by adding a ! after the leading @. Examples:

@!windows@unix@!osx

The first condition applies to any system but Windows and the second condition applies to any UNIX system besides OS X.

Runtime checks for the current version of Julia can be made using the built-in VERSION variable, which is of type VersionNumber. Such code is occasionally necessary to keep track of new or deprecated functionality between various releases of Julia. Examples of runtime checks:

VERSION<v"0.3-"#exclude all pre-release versions of 0.3v"0.2-"<=VERSION<v"0.3-"#get all 0.2 versions, including pre-releases, up to the abovev"0.2"<=VERSION<v"0.3-"#To get only stable 0.2 versions (Note v"0.2" == v"0.2.0")VERSION>=v"0.2.1"#get at least version 0.2.1

See the section on version number literals for a more complete description.