Jülich DATA
Goto Repository
Goto Training
Guides
Jülich DATA Guide
Log in to Jülich DATA
One-time registration
Gather permissions
Why Helmholtz AAI?
Dataverse Organization
Baseline
Campus Collection
Institutes Dataverses
Data Storage Linking
Contributors Guide to Data URLs
Linking to Data Locations
Curators Guide to Data URLs
Criteria for releasing a data publication
Formal criteria
Content-related criteria
External data sources
README File
Quality indicator for research data and research software publications
Glossary
Privacy Policy
Legal Notice
Copyright
Disclaimer
Terms of Use
Registration
Rights and Obligations of Users
Rights and Obligations of Uploader
Rights and Obligations of the Content User
Data Protection and Data Processing
Liability
Miscellaneous Terms
Mode of Access
Repository access
Data access
Upload access
Contact
User Guide
Account Creation + Management
Account Information
Remote Authentication
My Data
Notifications
API Token
Finding and Using Data
Finding Data
Using Data
Dataverse Management
Create a New Dataverse
Edit Dataverse
Dataset Linking
Dataverse Linking
Publish Your Dataverse
Dataset + File Management
Supported Metadata
Adding a New Dataset
File Upload
File Handling
Restricted Files
Edit Files
Replace Files
Terms
Roles & Permissions
Data Provenance
Thumbnails + Widgets
Publish Dataset
Submit for Review
Private URL to Review Unpublished Dataset
Dataset Versions
Dataset Metrics and Make Data Count
Cloud Storage + Computing
Dataset Deaccession
Tabular Data File Ingest
Supported File Formats
Tabular Data, Representation, Storage and Ingest
SPSS
Stata
R Data Format
Excel
CSV/TSV
Data Exploration Guide
TwoRavens: Tabular Data Exploration
WorldMap: Geospatial Data Exploration
Appendix
Metadata References
API Guide
Introduction
What is an API?
Types of Dataverse API Users
How This Guide is Organized
Getting Help
Getting Started with APIs
Servers You Can Test With
Getting an API Token
curl Examples and Environment Variables
Depositing Data
Finding and Downloading Data
Managing Permissions
Beyond “Getting Started” Tasks
Getting Help
API Tokens and Authentication
How to Get an API Token
How Your API Token Is Like a Password
Passing Your API Token as an HTTP Header (Preferred) or a Query Parameter
Resetting Your API Token
Search API
Parameters
Basic Search Example
Advanced Search Examples
Date Range Search Example
Iteration
Data Access API
Basic File Access
Multiple File (“bundle”) download
“All Formats” bundled download for Tabular Files.
Data Variable Metadata Access
Preprocessed Data
Authentication and Authorization
Access Requests and Processing
Native API
Dataverses
Datasets
Files
Users Token Management
Builtin Users
Roles
Explicit Groups
Shibboleth Groups
Info
Metadata Blocks
Notifications
Admin
Metrics API
Total
To-Month
Past Days
Dataverse Specific Metrics
Dataset Specific Metrics
Metric Query Parameters
SWORD API
About
Backward incompatible changes
New features as of v1.1
curl examples
Known issues
Bug fixes in v1.1
Client libraries
Client Libraries
Python
Javascript
R
Java
Building External Tools
Introduction
Examples of External Tools
How External Tools Are Presented to Users
Creating an External Tool Manifest
Testing Your External Tool
Spreading the Word About Your External Tool
Apps
Javascript
Python
Java
PHP
Frequently Asked Questions
What is an API?
What Are Common Use Cases for Dataverse APIs?
Where Can I Find Examples of Using Dataverse APIs?
When Should I Use the Native API vs. the SWORD API?
To Operate on a Dataset Should I Use Its DOI (or Handle) or Its Database ID?
Where is the Comprehensive List of All API Functionality?
Is There a Changelog of API Functionality That Has Been Added Over Time?
What Functionality is GUI Only and Not Available Via API
Why Are the Return Values (HTTP Status Codes) Not Documented?
What If My Question Is Not Answered Here?
Admin Guide
Dashboard
Harvesting
Metadata Export
Users
Move Data
External Tools
Inventory of External Tools
Managing External Tools
Testing External Tools
Writing Your Own External Tool
Managing Harvesting Clients
Your Dataverse as a Metadata Harvester
Managing Harvesting Clients
Managing Harvesting Server and Sets
Your Dataverse as an OAI server
How does it work?
OAI Sets
OAI Set updates
Metadata Customization
Introduction
About the metadata block TSV
Metadata Block Setup
Reloading a Metadata Block
Tips from the Dataverse Community
Footnotes
Metadata Export
Automatic Exports
Batch exports through the API
Export Failures
Downloading Metadata via GUI
Downloading Metadata via API
Dataverse Application Timers
Dedicated timer server in a Dataverse server cluster
Harvesting Timers
Metadata Export Timer
Known Issues
Make Data Count
Introduction
Architecture
Limitations for Dataverse Installations Using Handles Rather Than DOIs
Configuring Dataverse for Make Data Count Views and Downloads
Configuring Dataverse for Make Data Count Citations
Retrieving Make Data Count Metrics from the DataCite Hub
Retrieving Make Data Count Metrics from Dataverse
Integrations
Getting Data In
Embedding Data on Websites
Analysis and Computation
Discoverability
Research Data Preservation
Future Integrations
Geoconnect and WorldMap
Update “mapitlink”
Removing Dead Explore Links
User Administration
Manage Users
Merge User Accounts
Change User Identifier
Confirm Email
Deleting an API Token
Managing Datasets and Dataverses
Dataverses
Datasets
Solr Search Index
Full Reindex
Manual Reindexing
Manually Querying Solr
IP Groups
Listing IP Groups
Creating an IP Group
Listing an IP Group
Deleting an IP Group
Monitoring
Operating System Monitoring
HTTP Traffic
Database Connection Pool used by Glassfish
actionlogrecord
Edit Draft Versions Logging
Solr Indexing Failures Logging
EJB Timers
Reporting Tools and Common Queries
Maintenance
Backups
Troubleshooting
Using Dataverse APIs to Troubleshoot and Fix Problems
Glassfish
Deployment fails, “EJB Timer Service not available”
Timer Not Working
Constraint Violations Issues
Many Files with a File Type of “Unknown”, “Application”, or “Binary”
Getting Help
Installation Guide
Introduction
Quick Links
Intended Audience
Related Guides
Getting Help
Improving this Guide
Preparation
Choose Your Own Installation Adventure
Architecture and Components
System Requirements
Decisions to Make
Next Steps
Prerequisites
Linux
Java
Glassfish
PostgreSQL
Solr
jq
ImageMagick
R
Counter Processor
Next Steps
Installation
Running the Dataverse Installer
Logging In
Troubleshooting
Fresh Reinstall
Configuration
Securing Your Installation
Network Ports
Root Dataverse Permissions
Persistent Identifiers and Publishing Datasets
Auth Modes: Local vs. Remote vs. Both
File Storage: Using a Local Filesystem and/or Swift and/or S3 object stores
Branding Your Installation
Internationalization
Web Analytics Code
BagIt Export
Going Live: Launching Your Production Deployment
JVM Options
Database Settings
Upgrading
TwoRavens
0. Overview
1. Prerequisites
2. Install Extra R Packages
3. Install the TwoRavens Application
4. Appendix
Geoconnect
Shibboleth
Introduction
Installation
Configure Glassfish
Configure Apache
Configure Shibboleth
Disable or Reconfigure SELinux
Restart Apache and Shibboleth
Configure Apache and shibd to Start at Boot
Verify DiscoFeed and Metadata URLs
Add the Shibboleth Authentication Provider to Dataverse
Exchange Metadata with Your Identity Provider
Backup sp-cert.pem and sp-key.pem Files
Debugging
Converting Accounts
Institution-Wide Shibboleth Groups
OAuth Login Options
Introduction
Setup
Converting Local Users to OAuth
Converting OAuth Users to Local
OpenID Connect Login Options
Introduction
Other use cases and combinations
How to use
External Tools
Inventory of External Tools
Managing External Tools
Building External Tools
Advanced Installation
Multiple Glassfish Servers
Developer Guide
Introduction
Intended Audience
Getting Help
Core Technologies
Roadmap
Kanban Board
Issue Tracker
Related Guides
Related Projects
Development Environment
Quick Start
Set Up Dependencies
Run the Dataverse Installer Script
Configure Your Development Environment for Publishing
Next Steps
Windows Development
Running Dataverse in Vagrant
Running Dataverse in Minishift
Improving Windows Support
Tips
Iterating on Code and Redeploying
Netbeans Connector Chrome Extension
Database Schema Exploration
Deploying With
asadmin
Running the Dataverse
install
Script in Non-Interactive Mode
Preventing Glassfish from Phoning Home
Solr
Git
Sample Data
Troubleshooting
context-root in glassfish-web.xml Munged by Netbeans
Configuring / Troubleshooting Mail Host
Rebuilding Your Dev Environment
DataCite
Version Control
Where to Find the Dataverse Code
Branching Strategy
How to Make a Pull Request
How to Resolve Conflicts in Your Pull Request
Adding Commits to a Pull Request from a Fork
SQL Upgrade Scripts
Location of SQL Upgrade Scripts
How to Determine if You Need to Create a SQL Upgrade Script
How to Create a SQL Upgrade Script
Troubleshooting
Testing
The Health of a Codebase
Testing in Depth
Unit Tests
Integration Tests
Measuring Coverage of Integration Tests
Load/Performance Testing
Continuous Integration
Accessibility Testing
Future Work
Writing Documentation
Quick Fix
Other Changes (Sphinx)
Table of Contents
Images
Versions
Dependency Management
Terms
Direct dependencies
Transitive dependencies
Helpful tools
Repositories
Debugging
Logging
Coding Style
Java
Bash
Bike Shedding
Deployment
Deploying Dataverse to Amazon Web Services (AWS)
Docker, Kubernetes, and OpenShift
OpenShift
Docker
Making Releases
Create the release GitHub issue and branch
Merge “develop” into “master”
Write Release Notes
Make Artifacts Available for Download
Publish Release
Tools
Netbeans Connector Chrome Extension
pgAdmin
Maven
Vagrant
PlantUML
Eclipse Memory Analyzer Tool (MAT)
PageKite
MSV
FontCustom
SonarQube
Infer
lsof
jmap and jstat
Universal Numerical Fingerprint (UNF)
UNF Version 3
UNF Version 5
UNF Version 6
Make Data Count
Architecture
Dev Environment Setup for Make Data Count
Testing Make Data Count and Dataverse
Resources
Shibboleth and OAuth
Shibboleth and OAuth
Geospatial Data
Geoconnect
How Dataverse Ingests Shapefiles
WorldMap JoinTargets + API Endpoint
SELinux
Introduction
Development Environment
Recreating the shibboleth.te File
Big Data Support
S3 Direct Upload and Download
Data Capture Module (DCM)
Repository Storage Abstraction Layer (RSAL)
Workflows
Introduction
Style Guide
Foundations
Grid Layout
Typography
Color Palette
Icons
Logos
Patterns
Navbar
Breadcrumbs
Tables
Forms
Buttons
Pagination
Labels
Alerts
Images
Panels
Tabs
Modals
Jülich DATA Guide
User Guide
Account Creation + Management
Finding and Using Data
Dataverse Management
Dataset + File Management
Tabular Data File Ingest
Supported File Formats
Tabular Data, Representation, Storage and Ingest
SPSS
Stata
R Data Format
Excel
CSV/TSV
Data Exploration Guide
Appendix
API Guide
Admin Guide
Installation Guide
Developer Guide
Style Guide
Tabular Data File Ingest
¶
Contents:
Supported File Formats
Tabular Data, Representation, Storage and Ingest
What Happens During this “Ingest”?
Tabular Data and Metadata
Data vs. Metadata
Tabular Metadata in Dataverse
SPSS
Supported Versions
Limitations
SPSS Control Cards - not supported
Support for Language Encodings in SPSS
Stata
R Data Format
Overview.
Data Formatting Requirements.
Data Types, compared to other supported formats (Stat, SPSS)
Integers, Doubles, Character strings
R Factors
Boolean values
Limitations of R, as compared to SPSS and STATA.
Time values in R
Excel
CSV/TSV
Ingest of Comma-Separated Values and Tab-Separated Values files as tabular data.
Main formatting requirements
Limitations
Recognized data types and formatting