10x Genomics
Chromium Single Cell Gene Expression

Cell Ranger2.1, printed on 11/22/2024

HDF5 Gene-Barcode Matrix Format

In addition to MEX format, we also provide matrices in the Hierarchical Data Format (abbreviated HDF5 or H5). H5 is a binary format that can compress and access data much more efficiently than text formats such as MEX - which is especially useful when dealing with large datasets.

For more information on the format, see the Introduction to HDF5.

H5 files are supported in both Python and R.

File Format

The top-level of the file contains a list of HDF5 groups, one per genome. So for example if your data contains both hg19 and mm10, there will be two top-level groups. The file hierarchy would look something like this:

root
└── hg19
    ├── barcodes
    ├── data
    ...
    └── shape
└── mm10
    ├── barcodes
    ├── data
    ...
    └── shape

Most of the time, your analysis will only contain a single genome.

Within each genome group, the matrix is stored in Compressed Sparse Column (CSC) format. For more details on the format, see this SciPy introduction. CSC represents the matrix in column-major order, such that each barcode is represented by a contiguous chunk of data values.

Loading matrices into R

The latest version of cellrangerRkit is required to load a gene-barcode matrix from H5. For example, running the following code loads a filtered human (hg19) gene-barcode matrix into R:

library(cellrangerRkit)
genome <- "hg19"
pipestance_path <- "/opt/sample345"
gene_bc_matrix <- load_cellranger_matrix_h5(pipestance_path, genome=genome)

Loading matrices into Python

There are two ways to load the H5 matrix into Python:

1. Using cellranger

This method requires that you add cellranger/lib/python to your $PYTHONPATH.

E.g. if you installed Cell Ranger into /opt/cellranger-2.1.1, then you would call:

$ export PYTHONPATH=/opt/cellranger-2.1.1/lib/python:$PYTHONPATH

Then in Python, call:

import cellranger.matrix as cr_matrix
filtered_matrix_h5 = "/opt/sample345/outs/filtered_gene_bc_matrices_h5.h5"
genome = "hg19"
gene_bc_matrices = cr_matrix.GeneBCMatrices.load_h5(filtered_matrix_h5)
gene_bc_matrix = gene_bc_matrices.get_matrix(genome)

2. Using PyTables

This method is a bit more involved, and requires the SciPy and PyTables libraries.

import collections
import scipy.sparse as sp_sparse
import tables
 
GeneBCMatrix = collections.namedtuple('GeneBCMatrix', ['gene_ids', 'gene_names', 'barcodes', 'matrix'])
 
def get_matrix_from_h5(filename, genome):
    with tables.open_file(filename, 'r') as f:
        try:
            group = f.get_node(f.root, genome)
        except tables.NoSuchNodeError:
            print "That genome does not exist in this file."
            return None
        gene_ids = getattr(group, 'genes').read()
        gene_names = getattr(group, 'gene_names').read()
        barcodes = getattr(group, 'barcodes').read()
        data = getattr(group, 'data').read()
        indices = getattr(group, 'indices').read()
        indptr = getattr(group, 'indptr').read()
        shape = getattr(group, 'shape').read()
        matrix = sp_sparse.csc_matrix((data, indices, indptr), shape=shape)
        return GeneBCMatrix(gene_ids, gene_names, barcodes, matrix)
 
filtered_matrix_h5 = "/opt/sample345/outs/filtered_gene_bc_matrices_h5.h5"
genome = "hg19"
gene_bc_matrix = get_matrix_from_h5(filtered_matrix_h5, genome)