Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
qim3d
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Requirements
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Package registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
QIM
Tools
qim3d
Commits
52f025fe
Commit
52f025fe
authored
1 year ago
by
s184058
Committed by
fima
1 year ago
Browse files
Options
Downloads
Patches
Plain Diff
Load common formats
parent
19a2df31
No related branches found
No related tags found
1 merge request
!29
Load common formats
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
qim3d/io/load.py
+47
-14
47 additions, 14 deletions
qim3d/io/load.py
qim3d/utils/system.py
+3
-3
3 additions, 3 deletions
qim3d/utils/system.py
with
50 additions
and
17 deletions
qim3d/io/load.py
+
47
−
14
View file @
52f025fe
...
@@ -11,6 +11,7 @@ import qim3d
...
@@ -11,6 +11,7 @@ import qim3d
from
qim3d.io.logger
import
log
from
qim3d.io.logger
import
log
from
qim3d.utils.internal_tools
import
sizeof
,
stringify_path
from
qim3d.utils.internal_tools
import
sizeof
,
stringify_path
from
qim3d.utils.system
import
Memory
from
qim3d.utils.system
import
Memory
from
PIL
import
Image
,
UnidentifiedImageError
class
DataLoader
:
class
DataLoader
:
...
@@ -45,7 +46,7 @@ class DataLoader:
...
@@ -45,7 +46,7 @@ class DataLoader:
stack when loading files. Default is False.
stack when loading files. Default is False.
dataset_name (str, optional): Specifies the name of the dataset to be loaded
dataset_name (str, optional): Specifies the name of the dataset to be loaded
in case multiple dataset exist within the same file. Default is None (only for HDF5 files)
in case multiple dataset exist within the same file. Default is None (only for HDF5 files)
return_metadata (bool, optional): Specifies whether to return metadata or not. Default is False (only for HDF5
and
TXRM/TXM/XRM files)
return_metadata (bool, optional): Specifies whether to return metadata or not. Default is False (only for HDF5
,
TXRM/TXM/XRM
and NIfTI
files)
contains (str, optional): Specifies a part of the name that is common for the TIFF file stack to be loaded (only for TIFF stacks).
contains (str, optional): Specifies a part of the name that is common for the TIFF file stack to be loaded (only for TIFF stacks).
Default is None.
Default is None.
"""
"""
...
@@ -61,7 +62,8 @@ class DataLoader:
...
@@ -61,7 +62,8 @@ class DataLoader:
path (str): The path to the TIFF file.
path (str): The path to the TIFF file.
Returns:
Returns:
numpy.ndarray: The loaded volume as a NumPy array.
numpy.ndarray or numpy.memmap: The loaded volume.
If
'
self.virtual_stack
'
is True, returns a numpy.memmap object.
"""
"""
if
self
.
virtual_stack
:
if
self
.
virtual_stack
:
...
@@ -80,7 +82,8 @@ class DataLoader:
...
@@ -80,7 +82,8 @@ class DataLoader:
path (str): The path to the HDF5 file.
path (str): The path to the HDF5 file.
Returns:
Returns:
numpy.ndarray or tuple: The loaded volume as a NumPy array.
numpy.ndarray, h5py._hl.dataset.Dataset or tuple: The loaded volume.
If
'
self.virtual_stack
'
is True, returns a h5py._hl.dataset.Dataset object
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
Raises:
Raises:
...
@@ -161,7 +164,8 @@ class DataLoader:
...
@@ -161,7 +164,8 @@ class DataLoader:
path (str): The path to the stack of TIFF files.
path (str): The path to the stack of TIFF files.
Returns:
Returns:
numpy.ndarray: The loaded volume as a NumPy array.
numpy.ndarray or numpy.memmap: The loaded volume.
If
'
self.virtual_stack
'
is True, returns a numpy.memmap object.
Raises:
Raises:
ValueError: If the
'
contains
'
argument is not specified.
ValueError: If the
'
contains
'
argument is not specified.
...
@@ -214,7 +218,7 @@ class DataLoader:
...
@@ -214,7 +218,7 @@ class DataLoader:
path (str): The path to the HDF5 file.
path (str): The path to the HDF5 file.
Returns:
Returns:
numpy.ndarray or tuple: The loaded volume
as a NumPy array
.
numpy.ndarray or tuple: The loaded volume.
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
Raises:
Raises:
...
@@ -252,7 +256,8 @@ class DataLoader:
...
@@ -252,7 +256,8 @@ class DataLoader:
path (str): The path to the NIfTI file.
path (str): The path to the NIfTI file.
Returns:
Returns:
numpy.ndarray or tuple: The loaded volume as a NumPy array.
numpy.ndarray, nibabel.arrayproxy.ArrayProxy or tuple: The loaded volume.
If
'
self.virtual_stack
'
is True, returns a nibabel.arrayproxy.ArrayProxy object
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
If
'
self.return_metadata
'
is True, returns a tuple (volume, metadata).
"""
"""
...
@@ -272,6 +277,17 @@ class DataLoader:
...
@@ -272,6 +277,17 @@ class DataLoader:
return
vol
,
metadata
return
vol
,
metadata
else
:
else
:
return
vol
return
vol
def
load_pil
(
self
,
path
):
"""
Load a PIL image from the specified path
Args:
path (str): The path to the image supported by PIL.
Returns:
numpy.ndarray: The loaded image/volume.
"""
return
np
.
array
(
Image
.
open
(
path
))
def
load
(
self
,
path
):
def
load
(
self
,
path
):
"""
"""
...
@@ -281,8 +297,9 @@ class DataLoader:
...
@@ -281,8 +297,9 @@ class DataLoader:
path (str or os.PathLike): The path to the file or directory.
path (str or os.PathLike): The path to the file or directory.
Returns:
Returns:
numpy.ndarray: The loaded volume as a NumPy array.
numpy.ndarray, numpy.memmap, h5py._hl.dataset.Dataset, nibabel.arrayproxy.ArrayProxy or tuple: The loaded volume.
If
'
self.virtual_stack
'
is True, returns numpy.memmap, h5py._hl.dataset.Dataset or nibabel.arrayproxy.ArrayProxy depending on file format
If
'
self.return_metadata
'
is True and file format is either HDF5, NIfTI or TXRM/TXM/XRM, returns a tuple (volume, metadata).
Raises:
Raises:
ValueError: If the format is not supported
ValueError: If the format is not supported
ValueError: If the file or directory does not exist.
ValueError: If the file or directory does not exist.
...
@@ -292,6 +309,7 @@ class DataLoader:
...
@@ -292,6 +309,7 @@ class DataLoader:
data = loader.load(
"
image.tif
"
)
data = loader.load(
"
image.tif
"
)
"""
"""
# Stringify path in case it is not already a string
path
=
stringify_path
(
path
)
path
=
stringify_path
(
path
)
# Load a file
# Load a file
...
@@ -306,7 +324,10 @@ class DataLoader:
...
@@ -306,7 +324,10 @@ class DataLoader:
elif
path
.
endswith
((
"
.nii
"
,
"
.nii.gz
"
)):
elif
path
.
endswith
((
"
.nii
"
,
"
.nii.gz
"
)):
return
self
.
load_nifti
(
path
)
return
self
.
load_nifti
(
path
)
else
:
else
:
raise
ValueError
(
"
Unsupported file format
"
)
try
:
return
self
.
load_pil
(
path
)
except
UnidentifiedImageError
:
raise
ValueError
(
"
Unsupported file format
"
)
# Load a directory
# Load a directory
elif
os
.
path
.
isdir
(
path
):
elif
os
.
path
.
isdir
(
path
):
...
@@ -357,8 +378,9 @@ def load(
...
@@ -357,8 +378,9 @@ def load(
to the DataLoader constructor.
to the DataLoader constructor.
Returns:
Returns:
numpy.ndarray: The loaded volume as a NumPy array.
numpy.ndarray, numpy.memmap, h5py._hl.dataset.Dataset, nibabel.arrayproxy.ArrayProxy or tuple: The loaded volume.
If
'
return_metadata
'
is True and file format is either HDF5 or TXRM/TXM/XRM, returns a tuple (volume, metadata).
If
'
virtual_stack
'
is True, returns numpy.memmap, h5py._hl.dataset.Dataset or nibabel.arrayproxy.ArrayProxy depending on file format
If
'
return_metadata
'
is True and file format is either HDF5, NIfTI or TXRM/TXM/XRM, returns a tuple (volume, metadata).
Example:
Example:
data = qim3d.io.load(
"
image.tif
"
, virtual_stack=True)
data = qim3d.io.load(
"
image.tif
"
, virtual_stack=True)
...
@@ -374,16 +396,27 @@ def load(
...
@@ -374,16 +396,27 @@ def load(
data
=
loader
.
load
(
path
)
data
=
loader
.
load
(
path
)
if
not
virtual_stack
:
def
log_memory_info
(
data
):
mem
=
Memory
()
mem
=
Memory
()
log
.
info
(
log
.
info
(
"
Volume using %s of memory
\n
"
,
"
Volume using %s of memory
\n
"
,
sizeof
(
data
[
0
].
nbytes
if
return_metadata
else
data
.
nbytes
),
sizeof
(
data
[
0
].
nbytes
if
isinstance
(
data
,
tuple
)
else
data
.
nbytes
),
)
)
mem
.
report
()
mem
.
report
()
if
return_metadata
and
not
isinstance
(
data
,
tuple
):
log
.
warning
(
'
The file format does not contain metadata
'
)
if
not
virtual_stack
:
log_memory_info
(
data
)
else
:
else
:
log
.
info
(
"
Using virtual stack
"
)
# Only log if file type is not a np.ndarray, i.e., it is some kind of memmap object
if
not
isinstance
(
data
[
0
]
if
isinstance
(
data
,
tuple
)
else
data
,
np
.
ndarray
):
log
.
info
(
"
Using virtual stack
"
)
else
:
log
.
warning
(
'
Virtual stack is not supported for this file format
'
)
log_memory_info
(
data
)
return
data
return
data
...
...
This diff is collapsed.
Click to expand it.
qim3d/utils/system.py
+
3
−
3
View file @
52f025fe
...
@@ -19,7 +19,7 @@ class Memory:
...
@@ -19,7 +19,7 @@ class Memory:
self
.
total
=
mem
.
total
self
.
total
=
mem
.
total
self
.
free
=
mem
.
available
self
.
free
=
mem
.
available
self
.
free_pct
=
(
mem
.
fre
e
/
mem
.
total
)
*
100
self
.
free_pct
=
(
mem
.
availabl
e
/
mem
.
total
)
*
100
self
.
used
=
mem
.
total
-
mem
.
available
self
.
used
=
mem
.
total
-
mem
.
available
self
.
used_pct
=
mem
.
percent
self
.
used_pct
=
mem
.
percent
...
@@ -28,7 +28,7 @@ class Memory:
...
@@ -28,7 +28,7 @@ class Memory:
"
System memory:
\n
• Total.: %s
\n
• Used..: %s (%s%%)
\n
• Free..: %s (%s%%)
"
,
"
System memory:
\n
• Total.: %s
\n
• Used..: %s (%s%%)
\n
• Free..: %s (%s%%)
"
,
sizeof
(
self
.
total
),
sizeof
(
self
.
total
),
sizeof
(
self
.
used
),
sizeof
(
self
.
used
),
round
(
self
.
used_pct
,
2
),
round
(
self
.
used_pct
,
1
),
sizeof
(
self
.
free
),
sizeof
(
self
.
free
),
round
(
self
.
free_pct
,
2
),
round
(
self
.
free_pct
,
1
),
)
)
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment