{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Accessing Neuropixels Visual Coding Data\n", "\n", "## Tutorial overview\n", "\n", "This Jupyter notebook covers the various methods for accessing the Allen Institute Neuropixels Visual Coding dataset. We will go over how to request data, where it's stored, and what the various files contain. If you're having trouble downloading the data, or you just want to know more about what's going on under the hood, this is a good place to start.\n", "\n", "Currently, we do not have a web interface for browsing through the available cells and experiments, as with the [two-photon imaging Visual Coding dataset](http://observatory.brain-map.org/visualcoding). Instead, the data must be retrieved through the AllenSDK (Python 3.6+), or via requests sent to [api.brain-map.org](http://mouse.brain-map.org/static/api).\n", "\n", "Functions related to data analysis will be covered in other tutorials. For a full list of available tutorials, see the [SDK documentation](https://allensdk.readthedocs.io/en/latest/visual_coding_neuropixels.html).\n", "\n", "## Options for data access\n", "\n", "The **`EcephysProjectCache` object of the AllenSDK** is the easiest way to interact with the data. This object abstracts away the details of on-disk file storage, and delivers the data to you as ready-to-analyze Python objects. The cache will automatically keep track of which files are stored locally, and will download additional files on an as-needed basis. Usually you won't need to worry about how these files are structured, but this tutorial will cover those details in case you want to analyze them without using the AllenSDK (e.g., in Matlab). This tutorial begins with an introduction to this approach.\n", "\n", "If you have an **Amazon Web Services (AWS)** account, you can use an `EcephysProjectCache` object to access the data via the Allen Brain Observatory Simple Storage Service (S3) bucket. This is an AWS Public Dataset located at `arn:aws:s3:::allen-brain-observatory` in region `us-west-2`. Launching a Jupyter notebook instance on AWS will allow you to access the complete dataset without having to download anything locally. This includes around 80 TB of raw data files, which are not accessible via the AllenSDK. The only drawback is that you'll need to pay for the time that your instance is running—but this can still be economical in many cases. A brief overview of this approach can be found below.\n", "\n", "A third option is to directly download the data via **api.brain-map.org**. This should be used only as a last resort if the other options are broken or are not available to you. Instructions for this can be found at the end of this tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using the AllenSDK to retrieve data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Most users will want to access data via the AllenSDK. This requires nothing more than a Python interpreter and some free disk space to store the data locally.\n", "\n", "How much data is there? If you want to download the complete dataset (58 experiments), you'll need 855 GB of space, split across the following files:\n", "\n", "1. CSV files containing information about sessions, probes, channels and units (58.1 MB)\n", "2. NWB files containing spike times, behavior data, and stimulus information for each session (146.5 GB total, min file size = 1.7 GB, max file size = 3.3 GB)\n", "3. NWB files containing LFP data for each probe (707 GB total, min file size = 0.9 GB, max file size = 2.7 GB)\n", "\n", "Before downloading the data, you must decide where the `manifest.json` file lives. This file serves as the map that guides the `EcephysProjectCache` object to the file locations.\n", "\n", "When you initialize a local cache for the first time, it will create the manifest file at the path that you specify. This file lives in the same directory as the rest of the data, so make sure you put it somewhere that has enough space available. \n", "\n", "When you need to access the data in subsequent analysis sessions, you should point the `EcephysProjectCache` object to an _existing_ `manifest.json` file; otherwise, it will try to re-download the data in a new location.\n", "\n", "To get started with this approach, first take care of the necessary imports:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import os\n", "import shutil\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", "from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we'll specify the location of the manifest file. If you're creating a cache for the first time, this file won't exist yet, but it _must_ be placed in an existing data directory. Remember to choose a location that has plenty of free space available." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "data_directory = '/local1/ecephys_cache_dir' # must be updated to a valid directory in your filesystem\n", "\n", "manifest_path = os.path.join(data_directory, \"manifest.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can create the cache object, specifying both the local storage directory (the `manifest_path`) and the remote storage location (the Allen Institute data warehouse)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "cache = EcephysProjectCache.from_warehouse(manifest=manifest_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will prepare the cache to download four files:\n", "\n", "1. `sessions.csv` (7.8 kB)\n", "2. `probes.csv` (27.0 kB)\n", "3. `channels.csv` (6.6 MB)\n", "4. `units.csv` (51.4 MB)\n", "\n", "Each one contains a table of information related to its file name. If you're using the AllenSDK, you won't have to worry about how these files are formatted. Instead, you'll load the relevant data using specific accessor functions: `get_session_table()`, `get_probes()`, `get_channels()`, and `get_units()`. These functions return a [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html?highlight=dataframe) containing a row for each item and a column for each metric.\n", "\n", "If you are analyzing data without using the AllenSDK, you can load the data using your CSV file reader of choice. However, please be aware the columns in the original file do not necessarily match what's returned by the AllenSDK, which may combine information from multiple files to produce the final DataFrame.\n", "\n", "Let's take a closer look at what's in the `sessions.csv` file:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of sessions: 58\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
published_atspecimen_idsession_typeage_in_dayssexfull_genotypeunit_countchannel_countprobe_countecephys_structure_acronyms
id
7150937032019-10-03T00:00:00Z699733581brain_observatory_1.1118.0MSst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt88422196[CA1, VISrl, nan, PO, LP, LGd, CA3, DG, VISl, ...
7191615302019-10-03T00:00:00Z703279284brain_observatory_1.1122.0MSst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt75522146[TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan, N...
7211238222019-10-03T00:00:00Z707296982brain_observatory_1.1125.0MPvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt44422296[MB, SCig, PPT, NOT, DG, CA1, VISam, nan, LP, ...
7325921052019-10-03T00:00:00Z717038288brain_observatory_1.1100.0Mwt/wt82418475[nan, VISpm, VISp, VISl, VISal, VISrl]
7375810202019-10-03T00:00:00Z718643567brain_observatory_1.1108.0Mwt/wt56822186[grey, VISmma, nan, VISpm, VISp, VISl, VISrl]
\n", "
" ], "text/plain": [ " published_at specimen_id session_type \\\n", "id \n", "715093703 2019-10-03T00:00:00Z 699733581 brain_observatory_1.1 \n", "719161530 2019-10-03T00:00:00Z 703279284 brain_observatory_1.1 \n", "721123822 2019-10-03T00:00:00Z 707296982 brain_observatory_1.1 \n", "732592105 2019-10-03T00:00:00Z 717038288 brain_observatory_1.1 \n", "737581020 2019-10-03T00:00:00Z 718643567 brain_observatory_1.1 \n", "\n", " age_in_days sex full_genotype \\\n", "id \n", "715093703 118.0 M Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "719161530 122.0 M Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "721123822 125.0 M Pvalb-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "732592105 100.0 M wt/wt \n", "737581020 108.0 M wt/wt \n", "\n", " unit_count channel_count probe_count \\\n", "id \n", "715093703 884 2219 6 \n", "719161530 755 2214 6 \n", "721123822 444 2229 6 \n", "732592105 824 1847 5 \n", "737581020 568 2218 6 \n", "\n", " ecephys_structure_acronyms \n", "id \n", "715093703 [CA1, VISrl, nan, PO, LP, LGd, CA3, DG, VISl, ... \n", "719161530 [TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan, N... \n", "721123822 [MB, SCig, PPT, NOT, DG, CA1, VISam, nan, LP, ... \n", "732592105 [nan, VISpm, VISp, VISl, VISal, VISrl] \n", "737581020 [grey, VISmma, nan, VISpm, VISp, VISl, VISrl] " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "sessions = cache.get_session_table()\n", "\n", "print('Total number of sessions: ' + str(len(sessions)))\n", "\n", "sessions.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `sessions` DataFrame provides a high-level overview of the Neuropixels Visual Coding dataset. The index column is a unique ID, which serves as a key for accessing the physiology data for each session. The other columns contain information about:\n", "\n", "- the session type (i.e., which stimulus set was shown?)\n", "- the age, sex, and genotype of the mouse (in this dataset, there's only one session per mouse)\n", "- the number of probes, channels, and units for each session\n", "- the brain structures recorded (CCFv3 acronyms)\n", "\n", "If we want to find all of recordings from male Sst-Cre mice that viewed the Brain Observatory 1.1 stimulus and contain units from area LM, we can use the following query:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
published_atspecimen_idsession_typeage_in_dayssexfull_genotypeunit_countchannel_countprobe_countecephys_structure_acronyms
id
7150937032019-10-03T00:00:00Z699733581brain_observatory_1.1118.0MSst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt88422196[CA1, VISrl, nan, PO, LP, LGd, CA3, DG, VISl, ...
7191615302019-10-03T00:00:00Z703279284brain_observatory_1.1122.0MSst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt75522146[TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan, N...
7560299892019-10-03T00:00:00Z734865738brain_observatory_1.196.0MSst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt68422146[TH, DG, CA3, CA1, VISl, nan, PO, Eth, LP, VIS...
\n", "
" ], "text/plain": [ " published_at specimen_id session_type \\\n", "id \n", "715093703 2019-10-03T00:00:00Z 699733581 brain_observatory_1.1 \n", "719161530 2019-10-03T00:00:00Z 703279284 brain_observatory_1.1 \n", "756029989 2019-10-03T00:00:00Z 734865738 brain_observatory_1.1 \n", "\n", " age_in_days sex full_genotype \\\n", "id \n", "715093703 118.0 M Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "719161530 122.0 M Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "756029989 96.0 M Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt \n", "\n", " unit_count channel_count probe_count \\\n", "id \n", "715093703 884 2219 6 \n", "719161530 755 2214 6 \n", "756029989 684 2214 6 \n", "\n", " ecephys_structure_acronyms \n", "id \n", "715093703 [CA1, VISrl, nan, PO, LP, LGd, CA3, DG, VISl, ... \n", "719161530 [TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan, N... \n", "756029989 [TH, DG, CA3, CA1, VISl, nan, PO, Eth, LP, VIS... " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "filtered_sessions = sessions[(sessions.sex == 'M') & \\\n", " (sessions.full_genotype.str.find('Sst') > -1) & \\\n", " (sessions.session_type == 'brain_observatory_1.1') & \\\n", " (['VISl' in acronyms for acronyms in \n", " sessions.ecephys_structure_acronyms])]\n", "\n", "filtered_sessions.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `filtered_sessions` table contains the three sessions that meet these criteria.\n", "\n", "The code above uses standard syntax for filtering pandas DataFrames. If this is unfamiliar to you, we strongly recommend reading through the [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/). The AllenSDK makes heavy use of pandas objects, so we don't have to come up with our own functions for working with tabular data.\n", "\n", "Let's take a look at another DataFrame, extracted from the `probes.csv` file." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of probes: 332\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ecephys_session_idlfp_sampling_ratenamephasesampling_ratehas_lfp_dataunit_countchannel_countecephys_structure_acronyms
id
7294456487191615301249.998642probeA3a29999.967418True87374[APN, LP, MB, DG, CA1, VISam, nan]
7294456507191615301249.996620probeB3a29999.918880True202368[TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan]
7294456527191615301249.999897probeC3a29999.997521True207373[APN, NOT, MB, DG, SUB, VISp, nan]
7294456547191615301249.996707probeD3a29999.920963True93358[grey, VL, CA3, CA2, CA1, VISl, nan]
7294456567191615301249.999979probeE3a29999.999500True138370[PO, VPM, TH, LP, LGd, CA3, DG, CA1, VISal, nan]
\n", "
" ], "text/plain": [ " ecephys_session_id lfp_sampling_rate name phase sampling_rate \\\n", "id \n", "729445648 719161530 1249.998642 probeA 3a 29999.967418 \n", "729445650 719161530 1249.996620 probeB 3a 29999.918880 \n", "729445652 719161530 1249.999897 probeC 3a 29999.997521 \n", "729445654 719161530 1249.996707 probeD 3a 29999.920963 \n", "729445656 719161530 1249.999979 probeE 3a 29999.999500 \n", "\n", " has_lfp_data unit_count channel_count \\\n", "id \n", "729445648 True 87 374 \n", "729445650 True 202 368 \n", "729445652 True 207 373 \n", "729445654 True 93 358 \n", "729445656 True 138 370 \n", "\n", " ecephys_structure_acronyms \n", "id \n", "729445648 [APN, LP, MB, DG, CA1, VISam, nan] \n", "729445650 [TH, Eth, APN, POL, LP, DG, CA1, VISpm, nan] \n", "729445652 [APN, NOT, MB, DG, SUB, VISp, nan] \n", "729445654 [grey, VL, CA3, CA2, CA1, VISl, nan] \n", "729445656 [PO, VPM, TH, LP, LGd, CA3, DG, CA1, VISal, nan] " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "probes = cache.get_probes()\n", "\n", "print('Total number of probes: ' + str(len(probes)))\n", "\n", "probes.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `probes` DataFrame contains information about the Neuropixels probes used across all recordings. Each row represents one probe from one recording session, even though the physical probes may have been used in multiple sessions. Some of the important columns are:\n", "\n", "- `ecephys_session_id`: the index column of the `sessions` table\n", "- `sampling_rate`: the sampling rate (in Hz) for this probe's spike band; note that each probe has a unique sampling rate around 30 kHz. The small variations in sampling rate across probes can add up to large offsets over time, so it's critical to take these differences into account. However, all of the data you will interact with has been pre-aligned to a common clock, so this value is included only for reference purposes.\n", "- `lfp_sampling_rate`: the sampling rate (in Hz) for this probe's LFP band NWB files, after 2x downsampling from the original rate of 2.5 kHz\n", "- `name`: the probe name is assigned based on the location of the probe on the recording rig. This is useful to keep in mind because probes with the same name are always targeted to the same cortical region and enter the brain from the same angle (`probeA` = AM, `probeB` = PM, `probeC` = V1, `probeD` = LM, `probeE` = AL, `probeF` = RL). However, the targeting is not always accurate, so the actual recorded region may be different.\n", "- `phase`: the data may have been generated by one of two \"phases\" of Neuropixels probes. **3a** = prototype version; **PXI** = publicly available version (\"Neuropixels 1.0\"). The two phases should be equivalent from the perspective of data analysis, but there may be differences in the noise characteristics between the two acquisition systems.\n", "- `channel_count`: the number of channels with spikes or LFP data (maximum = 384)\n", "\n", "The `channels.csv` file contains information about each of these channels." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of channels: 123224\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ecephys_probe_idlocal_indexprobe_horizontal_positionprobe_vertical_positionanterior_posterior_ccf_coordinatedorsal_ventral_ccf_coordinateleft_right_ccf_coordinateecephys_structure_idecephys_structure_acronymecephys_session_idlfp_sampling_ratephasesampling_ratehas_lfp_dataunit_count
id
84970555879264550411120816533146862215.0APN7798394711250.0014793a30000.035489True0
84970556079264550425940816233076866215.0APN7798394711250.0014793a30000.035489True0
84970556279264550432740816033016871215.0APN7798394711250.0014793a30000.035489True0
84970556479264550444360815732956875215.0APN7798394711250.0014793a30000.035489True0
84970556679264550451160815532886879215.0APN7798394711250.0014793a30000.035489True0
\n", "
" ], "text/plain": [ " ecephys_probe_id local_index probe_horizontal_position \\\n", "id \n", "849705558 792645504 1 11 \n", "849705560 792645504 2 59 \n", "849705562 792645504 3 27 \n", "849705564 792645504 4 43 \n", "849705566 792645504 5 11 \n", "\n", " probe_vertical_position anterior_posterior_ccf_coordinate \\\n", "id \n", "849705558 20 8165 \n", "849705560 40 8162 \n", "849705562 40 8160 \n", "849705564 60 8157 \n", "849705566 60 8155 \n", "\n", " dorsal_ventral_ccf_coordinate left_right_ccf_coordinate \\\n", "id \n", "849705558 3314 6862 \n", "849705560 3307 6866 \n", "849705562 3301 6871 \n", "849705564 3295 6875 \n", "849705566 3288 6879 \n", "\n", " ecephys_structure_id ecephys_structure_acronym ecephys_session_id \\\n", "id \n", "849705558 215.0 APN 779839471 \n", "849705560 215.0 APN 779839471 \n", "849705562 215.0 APN 779839471 \n", "849705564 215.0 APN 779839471 \n", "849705566 215.0 APN 779839471 \n", "\n", " lfp_sampling_rate phase sampling_rate has_lfp_data unit_count \n", "id \n", "849705558 1250.001479 3a 30000.035489 True 0 \n", "849705560 1250.001479 3a 30000.035489 True 0 \n", "849705562 1250.001479 3a 30000.035489 True 0 \n", "849705564 1250.001479 3a 30000.035489 True 0 \n", "849705566 1250.001479 3a 30000.035489 True 0 " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "channels = cache.get_channels()\n", "\n", "print('Total number of channels: ' + str(len(channels)))\n", "\n", "channels.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The most important columns in the `channels` DataFrame concern each channel's location in physical space. Each channel is associated with a location along the probe shank (`probe_horizontal_position` and `probe_vertical_position`), and may be linked to a coordinate in the Allen Common Coordinate framework (if CCF registration is available for that probe).\n", "\n", "The information about channel location will be merged into the `units` DataFrame, which is loaded from `units.csv`:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of units: 40010\n" ] } ], "source": [ "units = cache.get_units()\n", "\n", "print('Total number of units: ' + str(len(units)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This DataFrame contains metadata about the available units across all sessions. By default, the AllenSDK applies some filters to this table and only returns units above a particular quality threshold.\n", "\n", "The default filter values are as follows:\n", "\n", "- `isi_violations` < 0.5\n", "- `amplitude_cutoff` < 0.1\n", "- `presence_ratio` > 0.9\n", "\n", "For more information about these quality metrics and how to interpret them, please refer to [this tutorial](./ecephys_quality_metrics.ipynb).\n", "\n", "If you want to see _all_ of the available units, it's straightfoward to disable the quality metrics filters when retrieving this table: " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of units: 99180\n" ] } ], "source": [ "units = cache.get_units(amplitude_cutoff_maximum = np.inf,\n", " presence_ratio_minimum = -np.inf,\n", " isi_violations_maximum = np.inf)\n", "\n", "print('Total number of units: ' + str(len(units)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the number of units has increased substantially, but some fraction of these units will be incomplete or highly contaminated. Understanding the meaning of these metrics is a critical part of analyzing the Neuropixels dataset, so we strongly recommend learning how to interpret them correctly.\n", "\n", "In addition to the quality metrics, there are a number of stimulus-specific metrics that are computed for each unit. These are not downloaded by default, but are accessed via a separate SDK function:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "21842 units in table 1\n", "18168 units in table 2\n" ] } ], "source": [ "analysis_metrics1 = cache.get_unit_analysis_metrics_by_session_type('brain_observatory_1.1')\n", "\n", "analysis_metrics2 = cache.get_unit_analysis_metrics_by_session_type('functional_connectivity')\n", "\n", "print(str(len(analysis_metrics1)) + ' units in table 1')\n", "print(str(len(analysis_metrics2)) + ' units in table 2')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will download two additional files, `brain_observatory_1.1_analysis_metrics.csv` and `functional_connectivity_analysis_metrics.csv`, and load them as pandas DataFrames. Note that the total length of these DataFrames is around 40k units, because the default quality metric thresholds have been applied.\n", "\n", "To load _all_ of the available units, and create one giant table of metrics, you can use the following code:" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "99180 units overall\n" ] } ], "source": [ "analysis_metrics1 = cache.get_unit_analysis_metrics_by_session_type('brain_observatory_1.1', amplitude_cutoff_maximum = np.inf,\n", " presence_ratio_minimum = -np.inf,\n", " isi_violations_maximum = np.inf)\n", "\n", "analysis_metrics2 = cache.get_unit_analysis_metrics_by_session_type('functional_connectivity', amplitude_cutoff_maximum = np.inf,\n", " presence_ratio_minimum = -np.inf,\n", " isi_violations_maximum = np.inf)\n", "\n", "all_metrics = pd.concat([analysis_metrics1, analysis_metrics2], sort=False)\n", "\n", "print(str(len(all_metrics)) + ' units overall')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The length of this DataFrame should match that of the `units` DataFrame we retrieved earlier. A few things to note about this DataFrame:\n", "\n", "- The unit analysis metrics DataFrame _also_ includes all quality metrics, so it's a superset of the `units` DataFrame\n", "- Since some of the stimuli in the `brain_observatory_1.1` session are not present in the `functional_connectivity` session, many of the data points in the unit analysis metrics DataFrame will be filled with `nan` values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accessing data for individual sessions\n", "\n", "Assuming you've found a session you're interested in analyzing in more detail, it's now time to download the data. This is as simple as calling `cache.get_session_data()`, with the `session_id` as input. This method will check the cache for an existing NWB file and, if it's not present, will automatically download it for you.\n", "\n", "Each NWB file can be upwards of 2 GB, so please be patient while it's downloading!\n", "\n", "As an example, let's look at one of the sessions we selected earlier, disabling the default unit quality metrics filters:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['DETAILED_STIMULUS_PARAMETERS', 'LazyProperty', 'age_in_days', 'api', 'channel_structure_intervals', 'channels', 'conditionwise_spike_statistics', 'ecephys_session_id', 'from_nwb_path', 'full_genotype', 'get_current_source_density', 'get_inter_presentation_intervals_for_stimulus', 'get_invalid_times', 'get_lfp', 'get_parameter_values_for_stimulus', 'get_pupil_data', 'get_stimulus_epochs', 'get_stimulus_parameter_values', 'get_stimulus_table', 'inter_presentation_intervals', 'invalid_times', 'mean_waveforms', 'metadata', 'num_channels', 'num_probes', 'num_stimulus_presentations', 'num_units', 'optogenetic_stimulation_epochs', 'presentationwise_spike_counts', 'presentationwise_spike_times', 'probes', 'rig_equipment_name', 'rig_geometry_data', 'running_speed', 'session_start_time', 'session_type', 'sex', 'specimen_name', 'spike_amplitudes', 'spike_times', 'stimulus_conditions', 'stimulus_names', 'stimulus_presentations', 'structure_acronyms', 'structurewise_unit_counts', 'units']\n" ] } ], "source": [ "session = cache.get_session_data(filtered_sessions.index.values[0],\n", " isi_violations_maximum = np.inf,\n", " amplitude_cutoff_maximum = np.inf,\n", " presence_ratio_minimum = -np.inf\n", " )\n", "\n", "print([attr_or_method for attr_or_method in dir(session) if attr_or_method[0] != '_'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the `session` object has a lot of attributes and methods that can be used to access the underlying data in the NWB file. Most of these will be touched on in other tutorials, but for now we will look at the only one that is capable of triggering additional data downloads, `get_lfp()`.\n", "\n", "In general, each NWB file is meant to be a self-contained repository of data for one recording session. However, for the Neuropixels data, we've broken with convention a bit in order to store LFP data in separate files. If we hadn't done this, analyzing one session would require an initial 15 GB file download. Now, the session is broken up in to ~2 GB chunks..\n", "\n", "Once you have created a `session` object, downloading the LFP data is simple (but may be slow):" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "probe_id = session.probes.index.values[0]\n", "\n", "lfp = session.get_lfp(probe_id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Tips for analyzing LFP data can be found in [this tutorial](./ecephys_lfp_analysis.ipynb)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Downloading the complete dataset\n", "\n", "Analyzing one session at a time is nice, but in many case you'll want to be able to query across the whole dataset. To fill your cache with all available data, you can use a `for` loop like the one below. Note that we've added some checks to ensure that the complete file is present, in case the download has been interrupted due to an unreliable connection.\n", "\n", "Before running this code, please make sure that you have enough space available in your cache directory. You'll need around 855 GB for the whole dataset, and 147 GB if you're not downloading the LFP data files." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for session_id, row in sessions.iterrows():\n", "\n", " truncated_file = True\n", " directory = os.path.join(data_directory + '/session_' + str(session_id))\n", " \n", " while truncated_file:\n", " session = cache.get_session_data(session_id)\n", " try:\n", " print(session.specimen_name)\n", " truncated_file = False\n", " except OSError:\n", " shutil.rmtree(directory)\n", " print(\" Truncated spikes file, re-downloading\")\n", "\n", " for probe_id, probe in session.probes.iterrows():\n", " \n", " print(' ' + probe.description)\n", " truncated_lfp = True\n", " \n", " while truncated_lfp:\n", " try:\n", " lfp = session.get_lfp(probe_id)\n", " truncated_lfp = False\n", " except OSError:\n", " fname = directory + '/probe_' + str(probe_id) + '_lfp.nwb'\n", " os.remove(fname)\n", " print(\" Truncated LFP file, re-downloading\")\n", " except ValueError:\n", " print(\" LFP file not found.\")\n", " truncated_lfp = False" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Accessing data on AWS" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to analyze the data without downloading anything to your local machine, you can use the AllenSDK on AWS.\n", "\n", "Follow [these instructions](https://github.com/AllenInstitute/AllenSDK/wiki/Use-the-Allen-Brain-Observatory-%E2%80%93-Visual-Coding-on-AWS) to launch a Jupyter notebook. Then, simply point to the existing manifest file in the Allen Institute's S3 bucket, and all of the data will be immediately available:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_root = \"/data/allen-brain-observatory/visual-coding-neuropixels/ecephys-cache/\"\n", "\n", "manifest_path = os.path.join(data_root, \"manifest.json\")\n", "\n", "cache = EcephysProjectCache(manifest=manifest_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once your cache is initialized, you can create the `sessions` table, load individual `session` objects, and access LFP data using the same commands described above.\n", "\n", "Additional tutorials specific to using AWS are coming soon." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Direct download via api.brain-map.org" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some people have reported issues downloading the files via the AllenSDK (the connection is extremely slow, or gets interrupted frequently). If this applies to you, you can try downloading the files via HTTP requests sent to **api.brain-map.org**. This approach is not recommended, because you will have to manually keep track of the file locations. But if you're doing analysis that doesn't depend on the AllenSDK (e.g., in Matlab), this may not matter to you.\n", "\n", "You can follow the steps below to retrieve the URLs for all of the NWB files in this dataset." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "from allensdk.brain_observatory.ecephys.ecephys_project_api.utilities import build_and_execute\n", "from allensdk.brain_observatory.ecephys.ecephys_project_api.rma_engine import RmaEngine\n", "from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache\n", "\n", "rma_engine = RmaEngine(scheme=\"http\", host=\"api.brain-map.org\")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "data_directory = '/mnt/nvme0/ecephys_cache_dir_10_31' # remember to change this to something that exists on your machine\n", "\n", "manifest_path = os.path.join(data_directory, \"manifest.json\")" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "cache = EcephysProjectCache.from_warehouse(manifest=manifest_path)\n", "\n", "sessions = cache.get_session_table()" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://api.brain-map.org//api/v2/well_known_file_download/953132529\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141250\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141227\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140952\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132170\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124669\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116170\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117445\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129455\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123586\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131767\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126543\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132367\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140775\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132593\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124638\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124493\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125810\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120513\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123875\n", "http://api.brain-map.org//api/v2/well_known_file_download/953113218\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132623\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131982\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119537\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153563\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166608\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124819\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132485\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125415\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131343\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153012\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125614\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126387\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125485\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152763\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128694\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125354\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147806\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140009\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141324\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132281\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132404\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132115\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132566\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117998\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120710\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132317\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131548\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132449\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132137\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123518\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123768\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123691\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116273\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124902\n", "http://api.brain-map.org//api/v2/well_known_file_download/953110284\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147846\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124557\n" ] } ], "source": [ "def retrieve_link(session_id):\n", " \n", " well_known_files = build_and_execute(\n", " (\n", " \"criteria=model::WellKnownFile\"\n", " \",rma::criteria,well_known_file_type[name$eq'EcephysNwb']\"\n", " \"[attachable_type$eq'EcephysSession']\"\n", " r\"[attachable_id$eq{{session_id}}]\"\n", " ),\n", " engine=rma_engine.get_rma_tabular, \n", " session_id=session_id\n", " )\n", " \n", " return 'http://api.brain-map.org/' + well_known_files['download_link'].iloc[0]\n", "\n", "download_links = [retrieve_link(session_id) for session_id in sessions.index.values]\n", "\n", "_ = [print(link) for link in download_links]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`download_links` is a list of 58 links that can be used to download the NWB files for all available sessions. Clicking on the links above should start the download automatically.\n", "\n", "Please keep in mind that you'll have to move these files to the appropriate sub-directory once the download is complete. The `EcephysProjectCache` object expects the following directory structure:\n", "\n", "```\n", "cache_dir/\n", "+-- manifest.json \n", "+-- session_/ \n", "¦ +-- session_.nwb\n", "+-- session_/\n", "¦ +-- session_.nwb\n", "+-- session_/\n", "¦ +-- session_.nwb\n", "\n", "```\n", "\n", "If you aren't interested in using the `EcephysProjectCache` object to keep track of what you've downloaded, you can create a `session` object just by passing a path to an NWB file:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from allensdk.brain_observatory.ecephys.ecephys_session import EcephysSession\n", "\n", "nwb_path = '/mnt/nvme0/ecephys_cache_dir_10_31/session_721123822/session_721123822.nwb'\n", "\n", "session = EcephysSession.from_nwb_path(nwb_path, api_kwargs={\n", " \"amplitude_cutoff_maximum\": np.inf,\n", " \"presence_ratio_minimum\": -np.inf,\n", " \"isi_violations_maximum\": np.inf\n", " })" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This will load the data for one session, without applying the default unit quality metric filters. Everything will be available except the LFP data, because the `get_lfp()` method can only find the associated LFP files if you're using the `EcephysProjectCache` object.\n", "\n", "To obtain similar links for the LFP files, you can use the following code:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "http://api.brain-map.org//api/v2/well_known_file_download/953141262\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141253\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141256\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141266\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141269\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141272\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140956\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140960\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140963\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140969\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140976\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140778\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140787\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140799\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140791\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140782\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140795\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129492\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129517\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129552\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129582\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129619\n", "http://api.brain-map.org//api/v2/well_known_file_download/953129650\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131775\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131772\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131778\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131781\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131787\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131784\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124672\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124688\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124685\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124681\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124678\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124675\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132374\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132380\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132383\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132386\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132377\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132370\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132187\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132196\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132202\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132199\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132190\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132180\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141236\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141233\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141230\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141239\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141246\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141243\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132599\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132605\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132608\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132602\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132596\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124512\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124509\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124506\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124500\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124503\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124496\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124657\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124654\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124645\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124641\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124648\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124651\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123596\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123600\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123591\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123605\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123615\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123610\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153570\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153573\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153566\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153576\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153579\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119593\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119571\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119548\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119582\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119560\n", "http://api.brain-map.org//api/v2/well_known_file_download/953119604\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166611\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166620\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166631\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166625\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166636\n", "http://api.brain-map.org//api/v2/well_known_file_download/953166614\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116227\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116237\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116219\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116183\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116207\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116194\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125828\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125825\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125832\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125820\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125816\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125813\n", "http://api.brain-map.org//api/v2/well_known_file_download/953113230\n", "http://api.brain-map.org//api/v2/well_known_file_download/953113227\n", "http://api.brain-map.org//api/v2/well_known_file_download/953113221\n", "http://api.brain-map.org//api/v2/well_known_file_download/953113224\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126578\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126567\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126606\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126596\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126616\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126555\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131991\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131988\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131985\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131995\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131999\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132003\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120546\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120577\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120561\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120534\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120523\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117451\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117455\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117448\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117464\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117461\n", "http://api.brain-map.org//api/v2/well_known_file_download/953117458\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123880\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123888\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123884\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123892\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125440\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125443\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125427\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125421\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125434\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125437\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124832\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124828\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124823\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124839\n", "file for probe 773463023 not found\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124846\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131346\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131352\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131359\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131355\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131349\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131362\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132495\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132492\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132488\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132498\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132501\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132505\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153019\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153022\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153025\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153028\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153031\n", "http://api.brain-map.org//api/v2/well_known_file_download/953153015\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125634\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125656\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125644\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125650\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125638\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125624\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141343\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141337\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141340\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141331\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141334\n", "http://api.brain-map.org//api/v2/well_known_file_download/953141328\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147809\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147813\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147822\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147819\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147825\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147816\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125489\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125495\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125499\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125503\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125508\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125492\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152766\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152772\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152769\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152775\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152781\n", "http://api.brain-map.org//api/v2/well_known_file_download/953152778\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140024\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140053\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140038\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140065\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140078\n", "http://api.brain-map.org//api/v2/well_known_file_download/953140090\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128714\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128733\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128724\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128707\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128744\n", "http://api.brain-map.org//api/v2/well_known_file_download/953128753\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125365\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125358\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125362\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125369\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125372\n", "http://api.brain-map.org//api/v2/well_known_file_download/953125376\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126412\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126444\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126429\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126457\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126470\n", "http://api.brain-map.org//api/v2/well_known_file_download/953126484\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120726\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120716\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120713\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120723\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120729\n", "http://api.brain-map.org//api/v2/well_known_file_download/953120719\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132284\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132296\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132290\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132287\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132299\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132293\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132417\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132414\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132423\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132408\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132411\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132420\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132132\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132129\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132126\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132119\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132136\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132141\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132320\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132327\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132324\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132331\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132334\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132337\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118009\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118001\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118005\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118019\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118022\n", "http://api.brain-map.org//api/v2/well_known_file_download/953118015\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132637\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132634\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132626\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132630\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132642\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132547\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132544\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132532\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132538\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132535\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132541\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132569\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132575\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132572\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132578\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132582\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132585\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132461\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132464\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132458\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132452\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132455\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123534\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123528\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123545\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123539\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123523\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123799\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123784\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123788\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123793\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123774\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123705\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123700\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123709\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123697\n", "http://api.brain-map.org//api/v2/well_known_file_download/953123714\n", "file for probe 832810584 not found\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131551\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131560\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131554\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131564\n", "http://api.brain-map.org//api/v2/well_known_file_download/953131567\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132154\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132145\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132165\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132159\n", "http://api.brain-map.org//api/v2/well_known_file_download/953132149\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116309\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116346\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116377\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116402\n", "http://api.brain-map.org//api/v2/well_known_file_download/953116443\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147855\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147859\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147852\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147849\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147862\n", "http://api.brain-map.org//api/v2/well_known_file_download/953147865\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124908\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124922\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124912\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124915\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124918\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124905\n", "file for probe 846401828 not found\n", "file for probe 846401830 not found\n", "file for probe 846401834 not found\n", "file for probe 846401838 not found\n", "file for probe 846401841 not found\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124578\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124572\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124575\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124567\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124560\n", "http://api.brain-map.org//api/v2/well_known_file_download/953124563\n" ] } ], "source": [ "def retrieve_lfp_link(probe_id):\n", "\n", " well_known_files = build_and_execute(\n", " (\n", " \"criteria=model::WellKnownFile\"\n", " \",rma::criteria,well_known_file_type[name$eq'EcephysLfpNwb']\"\n", " \"[attachable_type$eq'EcephysProbe']\"\n", " r\"[attachable_id$eq{{probe_id}}]\"\n", " ),\n", " engine=rma_engine.get_rma_tabular, \n", " probe_id=probe_id\n", " )\n", "\n", " if well_known_files.shape[0] != 1:\n", " return 'file for probe ' + str(probe_id) + ' not found'\n", " \n", " return 'http://api.brain-map.org/' + well_known_files.loc[0, \"download_link\"]\n", "\n", "probes = cache.get_probes()\n", "\n", "download_links = [retrieve_lfp_link(probe_id) for probe_id in probes.index.values]\n", "\n", "_ = [print(link) for link in download_links]" ] } ], "metadata": { "kernelspec": { "display_name": "Python (allensdk)", "language": "python", "name": "allensdk" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" } }, "nbformat": 4, "nbformat_minor": 2 }