Neural Data Server

A.I.-Powered Search Engine for A.I.:
Find Best Data to Pre-Train Your Deep Neural Models

What is Neural Data Server?

Neural Data Server (NDS) is a search engine for A.I. data. It uses Machine Learning to find the most relevant data to pre-train your deep neural models for your target application. Pre-training is known to significantly boost performance of a deep neural network particularly in cases where the amount of labeled data for the target application is scarce. However, there are tens of millions of labeled examples available across hundreds of existing datasets. Using all this data for pre-training is prohibitive: massive computational resources would be required particularly when the neural architecture for your application is still being developed and possibly many need to be tested. Finding a smaller, manageable amount of relevant data for pre-training is a crucial issue that NDS addresses.

Neural Data Server indexes several popular Machine Learning datasets in its quest to serve optimal data to ML users. In order to perform search, we provide a set of simple-to-use Machine Learning tools that probe your target application data (which is performed on the user's end) and send minimal information back to NDS in order to decide which data examples are relevant. Note that user's data is NOT visible to NDS, i.e., no user's data is at any point being copied to NDS. NDS finally generates a file with url links to relevant data samples. The user can then download the data (after agreeing to licenses of the datasets) and is ready to train a powerful neural network for their application!

Watch the short video below on how Neural Data Server works. For further technical information about NDS please see our paper.

We will be continuously growing our Dataset Registry, so please check back for updates. If you own a Machine Learning dataset and wish to add it to our Dataset Registry, please Contact Us.

2. Adapt Your Dataset

Fast Adapt Colab Notebook

Web Option: Open the Colab Notebook to download and adapt experts

Fast Adapt Local Script

Local Option: Download experts and adapt to your dataset locally

3. Fetch IDs of Most Relevant Data

Upload the transfer.pkl file generated by the fast adapt script and download the most relevant subset. Please specify the number of data examples you want in the "Budget" box.

If you find NDS useful, please consider citing our paper.