Mirroring Hackage

Posted in category haskell on 2014-09-23
Tags: haskell

Table of Contents

Hackage is the Haskell community’s central package archive. Package authors use it to publish their libraries and programs while other Haskell programmers use tools like cabal-install to download and install packages.

All good and well, but when you want to cabal update from a Windows station behind a corporate proxy, it might get very tricky and far unreliable. In order to avoid these types of issues it might be possible to mirror whole Hackage archive and set it up as local repo.

There are plenty of other reasons to have Hackage archive locally, though:

There are several solutions available at the moment that could help achieving save:

I personally tried making both of those solution to work, but I failed making it aware of default credentials when making HTTP requests to download package index from Hackage. And thus, I decided to write a .Net based solution in F# that would be aware of current default credentials when running it from within a corporate network. It’s called hackage-mirror (no obvious fancy naming talents here).

HackageMirror is a command line utility that requires one single argument - path to target directory which will be used to build local repo - Hackage archive mirror. The program is fault-tolerant in such a way that it will skip packages that are not available any longer (with proper logging), but will stop in case of all other types of failures - primary non-successful HTTP return codes (other than 404 - not found). The most frequent one that I personally experiences was 407 - Proxy authentication required. In case when program stops, it’s possible to restart the program with same arguments and it will catch up where it stopped last time.

The structure of the Local Repo

In order to enable local folder with bunch of packages to be enabled as Local Repo certain requirements must be fulfilled:

  1. index.tar.gz should be downloaded to /00-index.tar.gz
  2. /00-index.tar.gz should be decompressed into /00-index.tar and /preferred-versions
  3. /00-index.tar should be decompressed into the list of folders for each package of the following pattern: /package-name/[version]/package-name.cabal with potentially multiple versions
  4. for each /package-name/[version]/ there should be package downloaded from the official Hackage archive into /package-name/[version]/package-name.tar.gz

So, the target folder structure will look like following:

# /Target Path/
#   /00-index.tar.gz
#   /00-index.tar
#   /preferred-versions
#   /package_1/
#     /0.1/
#       /package_1.cabal
#       /package_1.tar.gz
#     /0.2/
#       /package_1.cabal
#       /package_1.tar.gz
#     /0.3/
#       /package_1.cabal
#       /package_1.tar.gz
#   /package_N/
#     /0.0.1/
#       /package_N.cabal
#       /package_N.tar.gz
#     /0.0.2/
#       /package_N.cabal
#       /package_N.tar.gz

Setting up Cabal to use Local Repo

Once HackageMirror is done its job, cabal should be configured to use /Target Path/ as local repo. This is done through cabal configuration file located under:

Locate following line in you cabal configuration file:

remote-repo: hackage.haskell.org:http://hackage.haskell.org/packages/archive
remote-repo-cache: C:\Users\UserName\AppData\Roaming\cabal\packages
-- local-repo:

And change it like following:

-- remote-repo: hackage.haskell.org:http://hackage.haskell.org/packages/archive
-- remote-repo-cache: C:\Users\UserName\AppData\Roaming\cabal\packages
local-repo: C:\Hackage

After this re-configuration is done, it’s safe to update your cabal:

$ cabal update
$ cabal install cabal-install

Project is hosted publicly on GitHub. It is implemented in two versions: Haskell and F#, so pick whatever suits you best.