Supported File Formats

File input and output

AMUSE can read and write to/from several data formats. The main routines are write_set_to_file and read_set_from_file, explained below.

Introduction

The AMUSE framework provides a generic way of reading and writing sets of entities (these entities can be particles, gasclouds or gridpoints). AMUSE provides a function to write a set to a file and a function to read a set from a file. These functions need the name of the file and the format of the data in the file. We will describe these functions first in this chapter. The functions can throw 3 kinds of exceptions, these are described next.

Use

To write a data set to a space separated text file do:

>>> from amuse.io import write_set_to_file
>>> from amuse.datamodel.core import Particles
>>> from amuse.units import units
>>> x = Particles(100)
>>> x.mass = 10.0 | units.MSun
>>> x.radius = range(1.0,101.0) | units.RSun
>>> write_set_to_file(x, "test.csv","txt", attribute_types = [units.MSun, units.RSun])
amuse.io.write_set_to_file(set, filename, format='amuse', **format_specific_keyword_arguments)

Write a set to the given file in the given format.

Parameters:
  • filename – name of the file to write the data to

  • format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

All other keywords are set as attributes on the fileformat processor. To determine the supported options for a processor call get_options_for_format()

Registered file formats:

amuse,

Process an HDF5 file

amuse-txt,

Process text files with AMUSE header format

csv,

Process comma separated files

dyn,

Process a Starlab binary structured file

gadget,

Process a Gadget binary data file

hdf5,

Process an HDF5 file

nemo,

Process a NEMO binary structured file

starlab,

Process a Starlab binary structured file

tsf,

Process a NEMO binary structured file

txt,

Process a text file containing a table of values separated by a predefined

vts,

Process a text file containing a table of values separated by a predefined character

vtu,

Process a text file containing a table of values separated by a predefined character

amuse.io.read_set_from_file(filename, format='amuse', **format_specific_keyword_arguments)

Read a set from the given file in the given format.

Parameters:
  • filename – name of the file to read the data from

  • format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

All other keywords are set as attributes on the fileformat processor. To determine the supported options for a processor call get_options_for_format()

Registered file formats:

amuse,

Process an HDF5 file

amuse-txt,

Process text files with AMUSE header format

csv,

Process comma separated files

dyn,

Process a Starlab binary structured file

gadget,

Process a Gadget binary data file

hdf5,

Process an HDF5 file

nemo,

Process a NEMO binary structured file

starlab,

Process a Starlab binary structured file

tsf,

Process a NEMO binary structured file

txt,

Process a text file containing a table of values separated by a predefined

vts,

Process a text file containing a table of values separated by a predefined character

vtu,

Process a text file containing a table of values separated by a predefined character

amuse.io.get_options_for_format(format='amuse')

Retuns a list of tuples, each tuple contains the name of the option, a description of the option and the default values.

Parameters:

format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

Exceptions

class amuse.io.UnsupportedFormatException(*arguments)

Raised when the given format is not supported by AMUSE.

class amuse.io.CannotSaveException(*arguments)

Raised when the given format cannot save data (only reading of data is supported for the format)

class amuse.io.CannotLoadException(*arguments)

Raised when the given format cannot read data (only saving of data is supported for the format)

Amuse HDF5 files

AMUSE provides it’s own data format based on hdf5 files. The HDF5 file storage provides support for storing particle set and grids in the same file. It also retains the particle references and can store multiple versions of the same set.

>>> from amuse.support import io
>>> from amuse.ic.plummer import new_plummer_model
>>> particles = new_plummer_model(1000)
>>> io.write_set_to_file(
            particles,
           'plummer.hdf5',
           'hdf5',
    )
>>> io.read_set_from_file('plummer.hdf5', 'hdf5')
allow_writing

If set to True, data can be written to the file, equivalent of append_to_file even if read_set_from_file is used

append_to_file

If set to True, new data is appended to HDF5 files. If set to False, the existing file is removed and overwritten. Only relevant for write set to file. (default: False)

close_file

If set to True, the file is closed after reading, unless you set copy_history to True no previous savepoints will be returned

compression

If set to False, don’t use compression, otherwise use specified compression mode (defaults to False, True means ‘gzip’)

compression_opts

Compression level to use if using compression (0-9, defaults to None)

copy_history

If set to True, the savepoint history is read from file into memory.

extra_attributes

Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.

names

A list of names to save the data under. If filled this will load the sets or grids with the given names and return this as a list. When saving the names will be used to save each set, a list of sets is needed in the write_set_to_file function (default: [])

overwrite_file

If set to True, overwrite file if it exists, otherwise writing an existing file generates an exception

return_context

If set to True, will return a context manager instead of the loaded set(s). This context manager will take care of properly closing any connected resources. To access the set, use the with statement. .. code-block:: python with load_set_from_file(“example.h5”, “amuse”) as particles: print particles Usefull for cases where close_file == False. (default: False)

version

AMUSE storage version to use, needs to be >= ‘2.0’ if you want to store links between particles and grids (default: ‘2.0’)

Text files

AMUSE has support for reading and writing text (.txt, .csv) files. You can specify the txt format by entering “txt” (for space separated files) or “csv” (for comma separated files) as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file(
           'plummer.txt',
           'txt',
           attribute_types = (units.m, units.m, units.kms),
           attribute_names= ('x', 'y', 'vx')
    )
>>> io.write_set_to_file(particles, 'plummer.txt', 'txt')
attribute_dtypes

List of the data types of the attributes. If not given, the data types will be float64.

attribute_names

List of the names of the attributes to load or store

attribute_types

List of the units of the attributes to store. If not given the units will be derived from the units stored in the attribute set. When derived from the set, the units will always be converted to the base units.

column_separator

Separator between the columns

comments

Comments to add to the header

extra_attributes

Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.

float_format_string

format specification string to convert numbers to strings, see format_spec in python documentation

footer_prefix_string

Lines starting with this character will be handled as part of the footer

has_amuse_header

If true, store the decription of the attributes and units in the header, better format than must_store_units_in_header provides

header_prefix_string

Lines starting with this character will be handled as part of the header

is_precise

Output most precise text output for floats (9 digits) and doubles (17 digits)

key_in_column

Column for the key (default is -1, no key stored/loaded). Keys will be interleaved with the data if set other than 0. No attribute type or name can be given for the key.

maximum_number_of_lines_buffered

The maximum number of lines to convert and write out in one go, larger values may be a little faster but will use a lot more memory

must_store_units_in_header

Store line with parsable units in the header of the text file. This line can be used to restore the units on reading the file. When this line is added to the file you do not need to specify the attribute_types on reading the file

precision_of_number_output

The precision is a decimal number indicating how many digits should be displayed after the decimal point

quantities

List of vector quantities, each vector quantity is one column in the text file. By default this list will be derived from the particles set. When this option is given it will override the particle set data.

stream

“Set the stream to output to

By default, text files are stored with some unit information on a comment line in the header. Unfortunately, amuse cannot depend on this line to read the file. When reading a text file you always need to specify the units of every column in the file. For example, to read a file with particle positions where all positions are stored in parsec, do:

particles = io.read_set_from_file(
       'positions.txt',
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names= ('x', 'y', 'z')
)

When writing a text file, you do not need to specify the units. If you don’t specify the units the write_set_to_file function will default to the units of the quantities being saved. For reliability and reproducibility we suggest to always specify the units and the names of the attributes to save when saving a text file:

particles = io.write_set_from_file(
       particles,
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names= ('x', 'y', 'z')
)

The amuse text file routines are very generic and since version 6.0, you can also specify a list of vector quantities instead of a particle set:

particles = io.write_set_from_file(
       None,
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names = ('x', 'y', 'z'),
       quantities = [
            [0.1,0.2] | units.parsec,
            [0.3,0.4] | units.parsec,
            [0.5,0.6] | units.parsec,
       ]
)

Starlab

AMUSE has support for reading and writing starlab (.dyn) files. You can specify the starlab format by entering “dyn” or “starlab” as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab')
>>> io.write_set_to_file(particles, 'output.dyn', 'dyn')

The starlab format support sevaral options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab', must_scale = False, return_children = False)
dynamics_length_units

The length fields in the dynamics section of a starlab file can be in parsec or in scaled units , defaults to scaled units (nbody_system.length) When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.

dynamics_mass_units

The m field in the dynamics section of a starlab file can be in MSun or in scaled units , defaults to scaled units (nbody_system.mass). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off

dynamics_time_units

The time fields in the dynamics section of a starlab file can be in Myr or in scaled units , defaults to scaled units (nbody_system.time). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.

extra_attributes

Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.

must_scale

If True use the scaling values from the file, if False do not scale the stellar dynamics properties. Only used when no nbody to si converter has been set.

nbody_to_si_converter

Starlab datafiles store stellar dynamics properties in scaled nbody values, provide a converter to store si data (defaults to None). Value None means no converter, or use scaling values provided in the file

return_children

If True returns the children of the root node, if False returns the root node (defaults to True)

return_converter

If True also return the converter when reading a set using read_set_from_file

The units of the values in the star (stellar properties) section of a starlab file are always in derived S.I units (solar mass, million years, solar luminocity etc.).

The masses given in de the dynamics section of a starlab file are usually in nbody units. Some starlab tools will set the mass values in Solar mass units (for example the makemass tool will return the masses in solar mass units). To read these files you need to set the dynamics_mass_units.

> makeplummer -n 1000 > plummer1.dyn
> cat plummer1.dyn | makemass -f 1 -x -2.0 -l 0.1 -u 20 > plummer2.dyn
> cat plummer2.dyn | add_star -Q 0.5 -R 5 > plummer3.dyn
> cat plummer3.dyn | scale -s > plummer4.dyn
> cat plummer4.dyn | kira -S > plummer5.dyn

The plummer1.dyn, plummer4.dyn and plummer5.dyn files will provide masses (and all other dynamical properties) in scaled nbody units. The plummer2.dyn and plummer3.dyn files will have masses in solar masses. To read each file in AMUSE, and return the particles with S.I. units, you need to do:

>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab', nbody_to_si_converter = converter)
>>> particles2 = io.read_set_from_file('plummer2.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles3 = io.read_set_from_file('plummer3.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab')
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab')

Note

No nbody converter object is needed for the last files, as the scale factors given in the files will be used.

The plummer1.dyn, plummer4.dyn and plummer5.dyn can also be read in nbody units. In the following example the returned particles have dynamic attributes (mass, radius, velocity, acceleration) in nbody units:

>>> from amuse.support import io
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab')
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab', must_scale = False)
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab', must_scale = False)

NEMO

AMUSE has support for reading and writing nemo (.tsf) files. You can specify the starlab format by entering “nemo” or “tsf” as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.tsf','nemo')
>>> io.write_set_to_file(particles, 'output.tsf', 'tsf')

The nemo format support several options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example:

>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles = io.read_set_from_file('plummer.nemo','tsf', nbody_to_si_converter = converter)
extra_attributes

Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.

nbody_to_si_converter

NEMO datafiles store nbody data, provide a converter to store si data (None means no converter)

Gadget

AMUSE has support for reading and writing gadget 2 files. You can specify the gadget format by entering “gadget” as the format:

>>> from amuse.support import io
>>> gas, halo, disk, bulge, stars, bndry = io.read_set_from_file('plummer.dat','gadget')
>>> io.write_set_to_file(particles, 'output.dat', 'gadget')

The gadget file format reader will return a tuple of gas, halo, disk, bulge, stars and boundary particles. This is different form other readers which only return one set. The write_set_to_file can take a particles set (will be saved as halo particles) or a tuple of gas, halo, disk, bulge, stars and boundary particles.

The gadget format support several options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example (will read a gadget file with timestep information):

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.nemo','gadget', has_timestep = True)
convert_gadget_w_to_velocity

Set to True to convert the w=sqrt(a)*dx/dt to (comoving) velocity in comoving integrations

endianness

The endianness of the binary date stored in the file

equal_mass_array

If filled with an array with masses > 0.0 assume equal mass for the corresponding set

extra_attributes

Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.

has_acceleration

Set to true if the file has a acceleration block

has_potential_energy

Set to true if the file has a potential energy block

has_rate_of_entropy_production

Set to true if the file has a block with the rate of change of the entropic function of each gas particle

has_timestep

Set to true if the file has a block with the individual timestep for each particle.

header_struct_format

The format of the header structure of the gadget file.

ids_are_keys

Set to True if the file contains correct keys. Set to False to generate the keys in amuse and prove an id attribute for the id’s in the gadget file

ids_are_long

Set to true the ids will be written as longs in the gadget file

is_initial_conditions_format

Set to true if the file contains initial conditions. An initial conditions gadget file contains less data.

return_header

Set to True to return both the particles and the header from the gadget file

write_header_from

Pass a namedtuple to store non-default values in the header of the gadget file