Python API reference

See Python usage for an introduction with examples.

DataLad commands

The commands added to DataLad by datalad-fuse, available from datalad.api and as methods of datalad.api.Dataset.

fusefs(mount_path[, dataset, foreground, ...])	FUSE File system providing transparent access to files under DataLad control
fsspec_head(path[, dataset, lines, bytes, ...])	Show leading lines/bytes of an annexed file by fetching its data from a remote URL
fsspec_cache_clear([dataset, recursive])	Clear fsspec cache

Opening files: datalad_fuse.fsspec

class datalad_fuse.fsspec.DatasetAdapter(path, caching, mode_transparent=False)

Read access to the files of a single dataset.

Files that are not annexed, and annexed files whose content is present locally, are opened from disk. Annexed files without local content are opened from one of the http(s) URLs found for their git-annex key (see get_urls()), reading only the needed parts of the file.

Parameters:

• path (str or Path) -- Top directory of the dataset (any git or git-annex repository).

• caching (bool) -- If true, keep the data fetched from remote URLs in a sparse on-disk cache under <path>/.git/datalad/cache/fsspec/, to be reused by subsequent reads (for a week after they were first cached). If false, data are only buffered in memory while a file is open.

• mode_transparent (bool) -- If true, paths of key files under .git/annex/objects/ (the targets of annexed symlinks) are opened as annexed content, fetched from a remote URL if not present locally.

Notes

Call close() (or use contextlib.closing()) when done, to stop the git annex processes started for the dataset.

close()

Stop the batched git annex processes started for the dataset

Return type:

None

get_file_state(relpath)

Determine whether a file is annexed and has its content present

Results are cached (for the most recently queried files).

Parameters:

relpath (str) -- Path of the file relative to the top directory of the dataset.

Returns:

The state of the file, and its git-annex key if it is annexed.

Return type:

tuple of (FileState, AnnexKey or None)

get_urls(key)

Yield candidate http(s) URLs for the content of an annex key

URLs are yielded in the order in which they are tried by open():

1. http(s) URLs recorded in git-annex for the key, as reported by git annex whereis (e.g. those of the web special remote);

2. annex/objects/... locations on the http(s) git remotes that git annex whereis lists as having the key, including the annex/objects endpoint of Forgejo-aneksajo instances.

URLs on S3 special remotes with exporttree=yes are not included; open() falls back to them via get_exporttree_urls().

Parameters:

key (str) -- A git-annex key, e.g. str(AnnexKey).

Return type:

Iterator[str]

get_exporttree_urls(relpath, key)

Yield versioned URLs for file on S3 exporttree remotes.

Workaround for datasets lacking proper versioned URLs in git-annex metadata. Constructs URLs from the remote’s publicurl + fileprefix and resolves the correct S3 object version by matching key.size.

Parameters:

• relpath (str) -- File path relative to dataset root (tree path).

• key (AnnexKey) -- Annex key with expected file size for version matching.

Yields:

str -- Versioned HTTP URLs (...?versionId=...) or unversioned URLs as fallback.

Return type:

Iterator[str]

open(relpath, mode='rb', encoding='utf-8', errors=None)

Open a file of the dataset for reading

Parameters:

• relpath (str) -- Path of the file relative to the top directory of the dataset.

• mode (str) -- "rb" (default) to get a binary file object, "r" or "rt" to get a text file object.

• encoding (str) -- Encoding to use in text mode.

• errors (str, optional) -- How to handle encoding errors in text mode, as for open().

Returns:

A seekable, read-only file object. Files read from disk are regular Python file objects; files read from a URL are fsspec file objects.

Return type:

file object

Raises:

• NotImplementedError -- If mode is not one of the supported read modes.

• IOError -- If the content of an annexed file is not present locally and none of its candidate URLs could be opened.

clear()

Remove the on-disk cache of the dataset (only if caching)

Return type:

None

class datalad_fuse.fsspec.FsspecAdapter(root, caching, mode_transparent=False)

Read access to the files of a dataset and its installed subdatasets.

Each path is mapped to the (sub)dataset containing it, and a DatasetAdapter is created for that dataset on first use. Use it as a context manager, so that the git annex processes started for the datasets are stopped on exit.

Parameters:

• root (str or Path) -- Top directory of the (super)dataset.

• caching (bool) -- Passed to each DatasetAdapter.

• mode_transparent (bool) -- Passed to each DatasetAdapter.

Notes

Use an absolute root, and absolute paths under it for the methods. Paths relative to root or to the current directory are not supported.

get_dataset_path(path)

Return the top directory of the (sub)dataset containing path

Parameters:

path (str | Path)

Return type:

Path

resolve_dataset(filepath)

Return the adapter for the dataset containing filepath

Returns:

The adapter, and the path of filepath relative to the dataset’s top directory.

Return type:

tuple of (DatasetAdapter, str)

Parameters:

filepath (str | Path)

open(filepath, mode='rb', encoding='utf-8', errors=None)

Open a file for reading; see DatasetAdapter.open()

Parameters:

• filepath (str | Path)

• mode (str)

• encoding (str)

• errors (str | None)

Return type:

IO

get_file_state(filepath)

Return state and key of a file; see DatasetAdapter.get_file_state

Parameters:

filepath (str | Path)

Return type:

tuple[FileState, AnnexKey | None]

is_under_annex(filepath)

Tell whether a file is annexed

Parameters:

filepath (str | Path)

Return type:

bool

get_commit_datetime(filepath)

Return the date of HEAD in the dataset containing filepath

Parameters:

filepath (str | Path)

Return type:

datetime

class datalad_fuse.fsspec.FileState

Bases: Enum

State of a file in a dataset, as returned by get_file_state()

NOT_ANNEXED = 1

The file is not annexed (e.g. committed to git directly); it is read from disk.

NO_CONTENT = 2

The file is annexed but its content is not present locally; it is read from a remote URL.

HAS_CONTENT = 3

The file is annexed and its content is present locally; it is read from disk.

git-annex helpers: datalad_fuse.utils

class datalad_fuse.utils.AnnexKey(backend, name, size=None, mtime=None, chunk_size=None, chunk_number=None, suffix=None)

A git-annex key, parsed into its fields

See <https://git-annex.branchable.com/internals/key_format/>. str() of an instance gives back the key.

Examples

>>> k = AnnexKey.parse("SHA256E-s1024--0123abcd.nwb")
>>> k.backend, k.size, k.name, k.suffix
('SHA256E', 1024, '0123abcd', '.nwb')
>>> str(k)
'SHA256E-s1024--0123abcd.nwb'

Parameters:

• backend (str)

• name (str)

• size (int | None)

• mtime (int | None)

• chunk_size (int | None)

• chunk_number (int | None)

• suffix (str | None)

classmethod parse(s)

Parse a key; raises ValueError if s is not a valid key

Parameters:

s (str)

Return type:

AnnexKey

classmethod parse_filename(s)

Parse a key from the name of a file under .git/annex/objects/

Such names escape some characters of the key (e.g. / as %).

Parameters:

s (str)

Return type:

AnnexKey

class datalad_fuse.utils.AnnexDir(topdir)

A (hashing) directory under .git/annex/objects/ of a repository

Parameters:

topdir (str)

topdir: str

Top directory of the repository

datalad_fuse.utils.is_annex_dir_or_key(path)

Tell whether path points into .git/annex/objects/

Returns an AnnexKey for a key file (.git/annex/objects/Xx/Yy/KEY/KEY), an AnnexDir for a directory leading to one, and None otherwise.

Parameters:

path (str | Path)

Return type:

AnnexDir | AnnexKey | None

FUSE file system: datalad_fuse.fuse_

class datalad_fuse.fuse_.DataLadFUSE(root, caching, mode_transparent=False)

Bases: Operations

mfusepy file system exposing a dataset, as used by datalad fusefs

Files are read via an FsspecAdapter, so annexed files without local content are read from their remote URLs. Unless mode_transparent is set, annexed files appear as regular files. For files without local content, the size is taken from their annex key and the modification time is the date of the HEAD commit. Files cannot be written to.

Parameters:

• root (str) -- Absolute path, without symbolic links (see os.path.realpath), to the top directory of the dataset to expose.

• caching (bool) -- Whether to cache remote data on disk; see DatasetAdapter.

• mode_transparent (bool) -- Whether to expose the .git directories of the datasets (hidden by default). Annexed files without local content then appear as symlinks into .git/annex/objects/.

Examples

Mount a dataset with extra FUSE options:

from mfusepy import FUSE
FUSE(DataLadFUSE("/abs/path/to/ds", caching=False), "/mnt/point",
     foreground=True, ro=True)