Universeodon Social Media

1 post1 participant0 posts today

mgorny-nyan (he) :autism:🙀🚂🐧<a href="https://social.treehouse.systems/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> is doing great. So basically:1. Originally, upstream used autotools. The build system installed a h5cc wrapper which — besides being a compiler wrapper — had a few config-tool style options. 2. Then, upstream added <a href="https://social.treehouse.systems/tags/CMake" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#CMake</a> build system as an alternative. It installed a different h5cc wrapper that did not have the config-tool style options anymore. 3. Downstreams that tried CMake quickly discovered that the new wrapper broke a lot of packages, so they reverted to autotools and reported a bug. 4. Upstream closed the bug, handwaving it as "CMake h5cc changes have been noted in the Release.txt at the time of change - archived copy should exist in the history files." 5. Upstream announced the plans to remove autotools support.So, to summarize the current situation:1. Pretty much everyone (at least <a href="https://social.treehouse.systems/tags/Arch" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Arch</a>, <a href="https://social.treehouse.systems/tags/Conda" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Conda</a>-forge, <a href="https://social.treehouse.systems/tags/Debian" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Debian</a>, <a href="https://social.treehouse.systems/tags/Fedora" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Fedora</a>, <a href="https://social.treehouse.systems/tags/Gentoo" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Gentoo</a>) is building using autotools, because CMake builds cause too much breakage. 2. Downstreams originally judged this to be a HDF5 issue, so they didn't report bugs to affected packages. Not sure if they're even aware that HDF5 upstream rejected the report. 3. All packages remain "broken", and I'm guessing their authors may not even be aware of the problem, because, well, as I pointed out, everyone is still using autotools, and nobody reported the issues during initial CMake testing. 4. I'm not even sure if there is a good "fix" here. I honestly don't know the package, but it really sounds like the config-tool was removed with no replacement, so the only way forward might be for people to switch over to CMake (sigh) — which would of course break the packages almost everywhere, unless people also add fallbacks for compatibility with autotools builds. 5. The upstream's attitude suggests that HDF5 is pretty much a project unto itself, and doesn't care about its actual users.<a href="https://github.com/HDFGroup/hdf5/issues/1814" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://github.com/HDFGroup/hdf5/issues/1814</a>

Howard Chu @ SymasThree Ways of Storing and Accessing Lots of Images in <a href="https://mastodon.social/tags/Python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Python</a> <a href="https://realpython.com/storing-images-in-python/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://realpython.com/storing-images-in-python/</a>Using plain files, <a href="https://mastodon.social/tags/LMDB" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LMDB</a>, and <a href="https://mastodon.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>. It's too bad there's an explicit serialization step for the LMDB case. In C we'd just splat the memory in and out of the DB as-is, with no ser/deser overhead.Also they use two separate tables for image and metadata in HDF5, but only one table in LMDB (with metadata concat'd to image). I don't see why they didn't just use two tables there as well.

Roadskater, Ph.D.I could have sworn <a href="https://mastodon.social/tags/HDF" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF</a> was only up to version 5.something, but maybe I lost track.<a href="https://mastodon.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>

ing.gridNew software descriptor published on ing.grid! "h5RDMtoolbox - A Python Toolbox for FAIR Data Management around HDF5" by Matthias Probst and Balazs Pritz <a href="https://www.inggrid.org/article/id/4028/" rel="nofollow noopener noreferrer" target="_blank">https://www.inggrid.org/article/id/4028/</a> <a href="https://nfdi.social/tags/RDM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RDM</a> <a href="https://nfdi.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> <a href="https://nfdi.social/tags/python" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#python</a> <a href="https://nfdi.social/tags/FAIR" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FAIR</a> <a href="https://nfdi.social/tags/kit" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#kit</a>

DAPHNE 4 NFDINeXus ist ein in unseren ommunities weitverbreitetes <a href="https://nfdi.social/tags/Datenformat" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Datenformat</a>. Es bietet einen Standard, welche Parameter gespeichert und wie sie im <a href="https://nfdi.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>-Datei strukturiert werden sollen, um <a href="https://nfdi.social/tags/Metadaten" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Metadaten</a> zusammen mit den Daten in einer hochstrukturierten Weise zu integrieren.Darauf aufbauend gibt es eine maschinenlesbare <a href="https://nfdi.social/tags/Ontologie" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Ontologie</a>; diese definiert eindeutige Bezeichner und schafft ein kontrolliertes Vokabular für die Namen aller experimentellen Parameter und gemessenen Variablen im Experiment.2/3

DAPHNE 4 NFDIGestern haben Rolf und Heike beim <a href="https://nfdi.social/tags/NFDI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NFDI</a>-Netzwerktreffen Berlin-Brandenburg unser Daphne-Konsortium vorgestellt. Speziell sind sie auf die Terminologien und Datenformate eingegangen, welche für <a href="https://nfdi.social/tags/Synchrotron" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Synchrotron</a>- und <a href="https://nfdi.social/tags/Neutronen" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Neutronen</a>-Experimente genutzt werden.1/3Die Folien gibt's bei Zonodo: <a href="https://zenodo.org/records/12728050" rel="nofollow noopener noreferrer" target="_blank">https://zenodo.org/records/12728050</a><a href="https://nfdi.social/tags/RDM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#RDM</a> <a href="https://nfdi.social/tags/Ontologien" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Ontologien</a> <a href="https://nfdi.social/tags/NeXus" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NeXus</a> <a href="https://nfdi.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>

MaltimoreSTOP DOING HDF5- Years of HDF5 yet no real-world use-case found - Files were never meant to be hierarchical or "self explaining" - "Hello I would like a file system within a file please" ...statements dreamt up by the utterly deranged - Look at what the HDF5 group has been demanding our respect for all this time - wanted to have an explanation for the stuff in a folder? We had a tool for that: it was called a ReadmeThey have played us for absolute fools<a href="https://social.tchncs.de/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>

Howard Chu @ Symas<a href="https://mastodon.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> is still a popular file format in scientific computing, but <a href="https://mastodon.social/tags/LMDB" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#LMDB</a> is always superior, especially in machine learning. <a href="https://www.reddit.com/r/MachineLearning/comments/1ad6j2i/d_how_to_make_my_training_faster/" rel="nofollow noopener noreferrer" translate="no" target="_blank">https://www.reddit.com/r/MachineLearning/comments/1ad6j2i/d_how_to_make_my_training_faster/</a>

Markus OsterhoffHELPMI: Develop a user-driven <a href="https://academiccloud.social/tags/NeXus" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#NeXus</a> extension proposal for laser-plasma (<a href="https://academiccloud.social/tags/laser" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#laser</a> <a href="https://academiccloud.social/tags/plasma" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#plasma</a>) experiments, by <a href="https://academiccloud.social/tags/Helmholtz" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Helmholtz</a> <a href="https://academiccloud.social/tags/Metadata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Metadata</a> CollaborationBasically, having nice and standardised metadata (e.g. in <a href="https://academiccloud.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> files) for <a href="https://academiccloud.social/tags/pewpew" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#pewpew</a> experiments, making data <a href="https://academiccloud.social/tags/FAIR" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#FAIR</a> for others – and also more easy to access for yourself ;)

Markus OsterhoffNow we're learning about h5wasm, especially viewing compressed <a href="https://academiccloud.social/tags/hdf5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#hdf5</a> data files with web assembly <a href="https://academiccloud.social/tags/wasm" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#wasm</a>.

Markus OsterhoffMaintenance of <a href="https://academiccloud.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> filters, plugins:Who fixes problems? What happens when, filter maintainers disappear? When underyling libraries are no longer maintained? API changes?Who cares, who pays?

Markus OsterhoffThis week, I'm at <a href="https://helmholtz.social/@DESY" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@DESY</a> for the <a href="https://academiccloud.social/tags/HDF" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF</a> User Group <a href="https://academiccloud.social/tags/HUG" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HUG</a> summit on plugins and data compression.We will start with latest updates on plugins: “HDF5 and plugins – overview and roadmap“, Dana Robinson. Then, Elena Pourmal will hopefully have good news: “Expanding HDF5 capabilities to support multi-threading access and new types of storage“.Currently, reading big datasets can be quite a burden due to some … challenges of <a href="https://academiccloud.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> with multi-threading.

Markus OsterhoffAnd after … a couple of days, here the <a href="https://academiccloud.social/tags/bugfix" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#bugfix</a> for my <a href="https://academiccloud.social/tags/memoryLeak" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#memoryLeak</a> It turns out that using <a href="https://academiccloud.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> needs some cleanup routines – that's totally okay, but I have plenty of <a href="https://academiccloud.social/tags/Terabytes" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#Terabytes</a> that worked without 🤔

Markus OsterhoffThe <a href="https://academiccloud.social/tags/HDF" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF</a> User Group will hold their annual European meeting on 19-21 September at <a href="https://academiccloud.social/tags/DESY" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#DESY</a>, Hamburg, Germany.The special topic of this meeting will be <a href="https://academiccloud.social/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a> plugins and <a href="https://academiccloud.social/tags/data" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#data</a> <a href="https://academiccloud.social/tags/compression" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#compression</a>, and the meeting is hosted in collaboration with the LEAPS-INNOV EU project. The HDF Group will give an update on the latest developments within HDF5 and present the roadmap for the future.<a href="https://indico.desy.de/e/ehug2023" rel="nofollow noopener noreferrer" target="_blank">https://indico.desy.de/e/ehug2023</a>

Christopher J Burke<a href="https://mastodon.online/@ExoHugh" class="u-url mention">@ExoHugh</a> I really hate getting an undocumented or poorly documented <a href="https://universeodon.com/tags/HDF5" class="mention hashtag" rel="tag">#HDF5</a> file. It is very challenging to parse it's contents. Having the human readable header metadata at beginning of fits/asdf files is a godsend for simply cat/less a file to figure out what's in it.

Hugh OsbornI'm at ESTEC (ESA's European Space Research and Technology Centre) in the Netherlands this week for my first <a href="https://mastodon.online/tags/PLATO" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#PLATO</a> science working team meeting as community scientist. One interesting discussion has been on astronomical file formats (honestly)... Future NASA missions look like they might move from <a href="https://mastodon.online/tags/fits" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#fits</a> to <a href="https://mastodon.online/tags/ASDF" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#ASDF</a>, while other astronomical surveys have started using <a href="https://mastodon.online/tags/HDF5" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#HDF5</a>. But it's tough to try to predict exactly where the community will be in 5-10 years time. Input welcome!

Recent searches

Search options

Administered by:

Server stats:

#hdf5

Recent searches

Search options

Administered by:

Server stats:

HDF5Back

#hdf5